Practical AEO, GEO, and SMO Optimization -- What an Answer Engine Can Read, Quote, and Cite

What this chapter is for

You already made your website rank on Google. That is SEO, and it is the previous chapter. This chapter is about the next question, the one that decides whether the newer wave of AI systems can use your page at all.

When someone asks ChatGPT, Perplexity, Gemini, or Google's AI Overview a question, those systems do not send the person a list of blue links. They read pages, pull out an answer, and often name the source they took it from. If your page is written so a machine can extract a clean answer from it, you get quoted. If the source looks trustworthy enough, you get named. If neither is true, the AI answers the question from someone else's page and your site is never mentioned, even when the answer came from your topic.

That is the whole subject of this chapter, and it splits into three jobs, each with its own name:

• AEO, Answer Engine Optimization. Making your page easy for a machine to read and lift a direct answer from.
• GEO, Generative Engine Optimization. Making your source trusted enough that a generative AI names it, rather than just using it silently.
• SMO, Social Media Optimization. Choosing which outside platforms actually feed those AI systems, and which are a waste of effort.

The stake this chapter is actually about

When an AI system answers a question directly instead of sending a click to your site, Google does not lose anything. Its revenue was never tied to that click. Your site does. Part 5 of this chapter verifies that claim against primary sources, not on trust, and shows what is actually still available to compete for once that specific click is gone. This chapter also covers the actual mechanism for controlling which crawlers reach your content in the first place, not just the polite request robots.txt makes that any crawler is free to ignore. What makes a crawler worth blocking covers the real gate.

One phrase gets used loosely throughout, so it is worth pinning down now. When this chapter says a machine "can see your page," it does not mean the page looks fine in a browser. It means the words exist in the raw HTML the server sends back, before any JavaScript runs. A page can look perfect to you and be completely blank to an AI crawler. The first illustration below shows exactly that, as a real file on disk, which is why the chapter opens with it rather than with definitions.

This chapter is a hands-on tutorial, not theory. Every mechanism is shown on one real, live platform, FolioVista Books (https://www.foliovistabooks.com, a Vite + React site), including the failures, the fixes, and the commands used to verify each one.

Table of contents

Front matter

• What this chapter is for
• Table of contents
• How to read the flags in this chapter
• Terms and acronyms used in this chapter

Illustration: the empty div

Part 1 -- The role of prerendering [FRAMEWORK-SPECIFIC: REACT/VITE, with a universal alternative]

• Why "my browser shows the content" is not the test that matters
• The build chain under the hood
• Verifying prerendering actually worked, not assuming it did
• A real instance of this exact failure
• Why keep the old URL alive: it protects the ranking and the crawl
• Alternative to Part 1 -- Universal version: frameworks with built-in rendering (Next.js, Nuxt, SvelteKit, Remix, Astro)

Part 2 -- AEO: making the content an answer engine can quote [UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

• SEO, AEO, GEO: three layers, one page
• Access before content
  • What makes a crawler worth blocking
    • The mechanism: blocking a crawler in code, not just in robots.txt
• What makes a sentence liftable for AI, not just readable
• FAQPage schema, and why the visible version still matters
• Two schema types worth adding, and one attribute worth reconsidering
• The /#faq mechanism -- why the fragment is not the target, the schema is
• One consistent description, everywhere
• Repetition is not the same as consistency

Part 3 -- GEO: earning the citation, not just the visit [UNIVERSAL]

• What generative engines actually pull from
• First-party content: the excerpt is not a compromise
• Content patterns that read as trustworthy to a citation-seeking system
• E-E-A-T, and the one letter that matters most here
• What a citation is actually worth, and why you cannot see it

Part 4 -- SMO: which platforms, and which to deliberately skip [UNIVERSAL]

• The uncomfortable finding this section has to answer

Part 5 -- Verify the source before you trust the claim [UNIVERSAL]

• The five-source verification diagram -- skip straight to the picture: what each source actually said, next to what got repeated
• Why this isn't the search engine's problem to fix
• Compete where the click is actually available
• Why well-resourced content keeps winning, whether or not it is the most accurate

Appendix -- Setting up the one free citation report, and what it found [UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

• Verifying the property at Bing Webmaster Tools
• The mistake worth documenting
• A canonical audit script failure at Google Search Console
• Making the check impossible to forget
  • Phase 1 -- static manual pages
  • Phase 2 -- generated React routes
  • Phase 3 -- sitemap checks for both
• The date field that has to be honest and to added for the Sitemap xml file for indexing
• The step no script can perform
• What the report actually shows
• The audit findings, and why one was fixed and one was not
• What Bing Webmaster Tools is and is not worth
• The complete file

How to read the flags in this chapter

Parts of this chapter apply to any website on any stack. Other parts show one specific mechanism that only exists on this project's React and Vite setup. So each Part, and some individual sections, carries one of four labels, telling a reader on a different stack exactly what to take as-is and what to translate.

[UNIVERSAL] -- applies to any project, any framework, unchanged. The AEO, GEO, and SMO ideas themselves are almost all of this kind: schema, answer-first writing, E-E-A-T, and platform choice do not depend on your stack.

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE] -- the underlying rule applies everywhere; the specific code, file, or command shown is just this project's particular way of doing it. A JSON-LD schema block is universal; the exact buildFaqSchema function shown is this project's example.

[FRAMEWORK-SPECIFIC: REACT/VITE] -- this mechanism itself is specific to this project's stack. A reader on a different framework needs a different mechanism for the same problem, not this one adapted. Part 1, prerendering, is the main example: the empty-div problem and the hand-written prerender.mjs only exist because this is a React and Vite single-page app.

[FOLIOVISTA-SPECIFIC] -- describes this project's own particular files, architecture, or a real incident on it. Not something to copy as-is, though the lesson drawn from it, stated wherever this tag appears, does carry over.

If you are not on React or Vite: Part 1 is the one part written around this project's specific rendering setup. The section right after it, "Alternative to Part 1 -- Universal version," is written for you. It covers how Next.js, Nuxt, and similar frameworks solve the same "a crawler needs real HTML, not an empty page" problem, usually automatically, so you can skip the hand-written prerender step entirely. Everything from Part 2 onward, the actual AEO, GEO, and SMO work, applies to your stack unchanged.

Terms and acronyms used in this chapter

This chapter does not assume prior AEO or GEO vocabulary. Everything below is defined again in context where it first matters; this list is the quick reference.

• SEO -- Search Engine Optimization. Making a page findable and indexable by a search engine, so it can rank and earn organic clicks. The subject of the previous chapter, and the layer everything here builds on.
• AEO -- Answer Engine Optimization. Structuring a page so a machine can extract a clean, direct answer from it. Measured by the page appearing inside AI answers and snippets. The subject of Part 2.
• GEO -- Generative Engine Optimization. Earning enough demonstrated trust that a generative AI names your source in its answer, not just uses it. Measured by being cited by name. The subject of Part 3.
• SMO -- Social Media Optimization. Choosing which external platforms actually feed AI systems and search, and which to skip. The subject of Part 4.
• Answer engine -- a system that responds to a question with a written answer instead of a list of links. ChatGPT, Perplexity, Gemini, and Google's AI Overview are all answer engines.
• Crawler / bot -- an automated program that fetches a page's raw HTML. GPTBot (OpenAI), ClaudeBot (Anthropic), and PerplexityBot are examples. A crawler reads the first raw response and does not run JavaScript.
• Prerendering -- generating a page's real HTML at build time, so a crawler gets actual content on its first request instead of an empty shell that only fills in after JavaScript runs. Covered in Part 1.
• Structured data / schema / JSON-LD -- a block of machine-readable data embedded in a page, following the shared vocabulary at schema.org, that states plainly what a page is (a book, an FAQ, an organization) instead of leaving a machine to infer it from prose. Covered in Part 2.
• FAQPage schema -- a specific schema type that marks a set of questions and answers on a page as machine-readable, so an answer engine can lift the pairs directly. Covered in Part 2.
• E-E-A-T -- Experience, Expertise, Authoritativeness, Trustworthiness. Google's framework for judging whether a source is worth trusting. The first E, Experience, is the one that matters most here. Covered in Part 3.
• Canonical (URL) -- the one address a page officially declares as its real version, when the same content could be reached at more than one URL. Referenced in Part 1 and the Appendix.
• Citation -- in this chapter, a generative engine naming your site as the source of an answer, which is the GEO goal, not a footnote in an academic sense.

Illustration: the empty div

"AI can't see your page" is not a warning. It is a file sitting on disk. Here it is, before any of the definitions.

A production React site has two files that share a name and nothing else. The first is index.html at the project root. This is the template Vite builds from:

<div id="root"></div>
<script type="module" src="/src/main.jsx"></script>

That is the entire body. No book titles, no FAQ answers, no navigation text. Just an empty container and a script tag pointing at unbuilt source code.

The second is dist/index.html. This is what a build step produces from the first one:

<div id="root">
  <div class="app">
    <header class="site-header">...</header>
    <section class="hero-section">
      <h1>Digital books, practical guides, manuals, and free sample
      chapters.</h1>
      ...

The two files share a filename and a starting point, but their content is completely different. The only thing that separates them is one build step. A browser cannot tell the difference. It runs the JavaScript either way and ends up looking at the same rendered page. A search engine or an AI answer engine reads the raw HTML directly, without executing anything, so it sees exactly one of these two files. Your build pipeline may skip the step that turns the first file into the second. If that happens, every AI system that reads your site sees an empty box and a script tag it will never run.

This is not a hypothetical. This is the literal, unedited content of a real file on a real production site, checked directly rather than assumed, the same day this chapter was written.

Why "my browser shows the content" is not the test that matters

[UNIVERSAL]

A team can ship a page, open it in Chrome, watch the headings and the text render correctly, and conclude everything is fine. That test measures the wrong thing.

Every browser downloads and runs JavaScript by default. Chrome does it. Safari does it. Firefox does it. So if a page's content only appears after JavaScript runs, the browser still shows that content correctly. This is why "I checked the page in my browser" tells you nothing about whether an AI crawler can read it.

GPTBot from OpenAI, ClaudeBot from Anthropic, and PerplexityBot all work the way classic search engine crawlers have always worked. They send one HTTP request. They read whatever HTML comes back in that single response. Then they stop. They do not run JavaScript. They do not wait for a framework to hydrate. They do not click anything. Whatever text exists in that first raw response is the entire universe of content that crawler will ever see for that page.

The only reliable way to check what those crawlers see is to fetch the page the same way they do. That means a plain HTTP request, not a browser. That is why every check in this chapter uses curl. A curl request returns the raw response and nothing else. It never runs JavaScript, so what it prints is what the crawler receives.

Before you run it, know what you are about to look at. The command has exactly three possible responses. The first response is the healthy one. The second and the third both look like failure. Only the second is a real failure. The third is a working page and a mistake in how you asked for it. Read all three before you judge your own result.

First Response. The healthy one.

curl -sL https://www.foliovistabooks.com | grep -o '<div id="root">.\{0,300\}'
# <div id="root"><div class="app"><header class="site-header"><a class="brand"
# aria-label="FolioVista Books home" href="/"><img class="brand-logo"
# src="/icons/logo-horizontal.svg" alt="FolioVista Books"/></a>
# <nav class="main-nav" aria-label="Main navigation">...

That output is real, taken from the live site. Header markup, a brand link, a navigation element. All of it exists in the response before any JavaScript runs, which is what tells you a text-only crawler can read the page.

Second Response. A real failure.

A page that was never prerendered returns the container and nothing inside it. This is the unbuilt template as it sits on disk, which is exactly what a broken build step would publish:

grep -o '<div id="root">.\{0,300\}' index.html
# <div id="root"></div>

The container arrived and the content did not. Here, no amount of "it looks fine in the browser" changes what a crawler actually receives.

Third Response. A working page and a wrong command.

A request to the wrong host returns nothing at all. No output, and grep exits with status 1:

curl -s https://foliovistabooks.com | grep -o '<div id="root">.\{0,300\}'
# (no output)

That one is not a broken site. It is a redirect that was never followed. The same URL returns real content once -L is added:

curl -sL https://foliovistabooks.com | grep -o '<div id="root">.\{0,120\}'
# <div id="root"><div class="app"><header class="site-header"><a class="brand"
# aria-label="FolioVista Books home" href="/"><img class="br

Two details in that command decide which of the three you get, and getting either one wrong produces the third response every time.

The -L flag tells curl to follow redirects. Without it, a request to a domain that redirects returns only the redirect header and an empty body. Then grep finds nothing. This site canonicalises to the www host. So curl -s against the bare domain returns a 308 status and no HTML at all.

The host has to match the canonical one. This site canonicalises to www.foliovistabooks.com. So requesting foliovistabooks.com produces that same empty result, for the same reason. Check which host your own site actually serves from before you conclude anything from a blank response.

The conclusion. An empty container means the page is broken. No output at all usually means the command is wrong. People routinely reach the opposite conclusion from both, because both arrive as silence. Run the command with -L and against the canonical host. Only then should you judge the page by what comes back.

Every output above was taken from the live site or from the real template file, on the day this chapter was written.

The build chain under the hood

[FRAMEWORK-SPECIFIC: REACT/VITE]

You never type these commands yourself. You type one command, and it launches the whole chain:

npm run build

That single command is what runs on your machine and on the deploy platform. For a Vite-based React site, package.json expands it into three commands that run in sequence, each doing a distinct job:

vite build
vite build --ssr src/entry-server.jsx --outDir .prerender
node scripts/prerender.mjs

This matters because the failure this chapter describes happens inside step three. If you only ever see npm run build succeed, you never see which of the three did what.

Step 1 bundles the client-side JavaScript and writes dist/index.html from the root template. At this point <div id="root"> is still empty. Only the asset paths have been swapped in.

A script tag pointing at /src/main.jsx becomes one pointing at a built, hashed filename like /assets/index-[hash].js. Content has not been generated yet.

Step 2 builds a completely separate bundle. This is a server-rendering version of the same React code, output to its own directory. The step does not touch dist/ at all. Its only job is producing something capable of turning a route into an HTML string when asked.

Step 3 is the one that actually matters for AEO and GEO. It runs that server-rendering bundle once per real route the site has.

That means /, /books, each individual book page, /contact, and so on. It writes the fully rendered HTML for each route directly into dist/.

This is the step that turns an empty <div id="root"> into one full of real, readable content. It happens automatically during the build.

The one part that does depend on a person is the route list. prerender.mjs only renders the routes it has been told about, and that list is written by hand:

const routes = [
  "/",
  "/books",
  "/books/operational-bug-bounty-fieldwork",
  ...
];

A route that is not in that list is never prerendered. The build still succeeds. Every other page still gets its content.

That missing route still renders fine in the browser. React runs and fills the div. The raw HTML response is still empty, so a crawler sees nothing. The missing one ships as the empty root index.html shown at the start of this chapter.

That holds regardless of how much schema markup, FAQ content, or careful copywriting exists in the source code of this missing route (its React component). None of it reaches the actual HTTP response a crawler receives for that one route.

Verifying prerendering actually worked, not assuming it did

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

Checking the source code for a prerender step proves the step exists.

It does not prove the step is running correctly, or that every route is covered, or that a recent change to routing didn't silently drop a page from the route list.

The only proof that counts is running a curl command against each live route path. That is the same way a crawler reads it, and it shows you the actual bytes:

for path in \
  "/" \
  "/books" \
  "/books/operational-bug-bounty-fieldwork" \
  "/books/practical-layered-security-for-small-platforms" \
  "/books/practical-seo-aeo-geo-smo-optimization" \
  "/contact" \
  "/privacy-and-terms"
do
  out=$(curl -sL "https://www.foliovistabooks.com$path" \
        | grep -o '<div id="root">.\{0,60\}')
  if [ -n "$out" ]; then
    printf "OK    %s\n" "$path"
  else
    printf "EMPTY %s   <-- investigate\n" "$path"
  fi
done
# OK    /
# OK    /books
# OK    /books/operational-bug-bounty-fieldwork
# OK    /books/practical-layered-security-for-small-platforms
# OK    /books/practical-seo-aeo-geo-smo-optimization
# OK    /contact
# OK    /privacy-and-terms

Run against a real site with prerendering correctly configured, every one of these returns real page content in the first 60 characters after the root div, not an empty shell. That is the actual evidence a prerender pipeline is working.

A code review is not that evidence. Neither is a local npm run dev check, which runs a JS-driven dev server and would show correct content in a browser regardless of whether prerendering works at all. Neither is a build log that says "success". A build can complete successfully while still shipping empty routes, if the route list inside the prerender script is stale.

A real instance of this exact failure

[FOLIOVISTA-SPECIFIC]

This is not a hypothetical extension of the argument above. It is the same failure mode, caught live, on this same site, while the previous chapter was being written.

If you read that chapter, you have met this incident before. It appears there as a case study about search indexing, and the question it answers there is whether Google ever received the page. The question here is different and more basic: whether any system that does not execute JavaScript could read the page at all. Same bug, same commands, two different failures downstream of it. It is repeated rather than cross-referenced because the mechanism is the foundation of everything in this chapter.

A book's URL was renamed to include a missing keyword. It went from /books/practical-aeo-geo-smo-optimization to /books/practical-seo-aeo-geo-smo-optimization. That was a one-line change to the route table in the main app component. The change was committed, and a deploy did go out. In a browser, the new URL worked. Clicking through from the homepage rendered the page correctly, full content, no visible problem.

curl told a different story:

The output below is what these commands returned during the incident. The bug has since been fixed, so running them today returns something different, and the verification after the fix is shown further down.

# During the incident. The new URL:
curl -sL -o /dev/null -w "%{http_code}\n" \
  https://www.foliovistabooks.com/books/practical-seo-aeo-geo-smo-optimization
# 404

# And the old one:
curl -sL -o /dev/null -w "%{http_code}\n" \
  https://www.foliovistabooks.com/books/practical-aeo-geo-smo-optimization
# 200 -- now check what that 200 response actually contains:

curl -sL https://www.foliovistabooks.com/books/practical-aeo-geo-smo-optimization \
  | grep -o '<div id="root">.\{0,300\}'
# <div id="root"><div class="app"><header>...</header><footer>...</footer></div></div>
# -- header and footer only. No book title. No content. Empty.

The new URL was a genuine 404. No static file for it existed anywhere on the server. The old URL returned 200, but the content was an empty shell. Header and footer, nothing else.

Neither was a rendering bug in the component code. The <Route> list in App.jsx, the app's root component, had been updated correctly and was live and working. That part of the deploy succeeded.

The one file that had not been updated was the prerender script's own hardcoded route list, which still named only the old URL.

The result was two symptoms. Prerendering the old URL now ran that path through the current app code, where it had become a client-side-only redirect element with nothing to render server-side. That produced the empty shell. The new URL had never been added to the list at all, so no file for it was ever generated. That produced the 404.

The exact diagnostic this chapter recommends is what found it. Not "does it look right in a browser", because it did, via client-side routing that never touches the server for that path. It was a plain curl against the live URL, reading the raw bytes a crawler would actually receive.

Fix: update the prerender script's route list to match the current route table in the App.jsx component, and add a real server-side redirect (not a client-side one) from the old URL to the new one, so a crawler hitting the old URL gets pointed at the new one directly instead of receiving an empty page. Rebuilt, redeployed, reverified with the same curl commands before considering it closed.

The difference between those two kinds of redirect is the whole point here.

A client-side redirect is a <Navigate> element in App.jsx. It only runs after JavaScript executes in a browser. The server still answers 200 with an empty shell, so a crawler receives nothing.

A server-side redirect is a redirects entry in vercel.json. The server answers 301 or 308 before any HTML is sent, so a crawler is pointed at the new URL without running anything.

prerender.mjs plays no part in either. It only writes static HTML files. It never redirects.

The redirect sits on the old URL and points at the new one. Old URL, then 308, then new URL.

Click image to enlarge

You have to update the route list in prerender.mjs to match the route table in App.jsx, then rebuild and redeploy, then recheck with curl -sL. Both lists have to name the same paths, and nothing in the build checks that for you.

Those same two commands, run against the live site today, return this instead:

curl -sL -o /dev/null -w "%{http_code}\n" \
  https://www.foliovistabooks.com/books/practical-seo-aeo-geo-smo-optimization
# 200

curl -sI https://www.foliovistabooks.com/books/practical-aeo-geo-smo-optimization \
  | grep -i "^location"
# location: /books/practical-seo-aeo-geo-smo-optimization

The new URL now serves real content, and the old one returns a permanent redirect rather than an empty page.

Note the second command uses -I for headers only and deliberately omits -L: the point is to see the redirect itself, not to follow it. The flag is case-sensitive. Capital -I sends a HEAD request and returns only the response headers, while lowercase -i sends a normal request and prints the headers together with the full body. Here you want capital -I.

Adding -L here would silently hide the very thing being checked. Without -L, curl stops at the old URL and shows its 308 status and its location header, which is the redirect you want to confirm. With -L, curl follows that redirect immediately and reports the final page instead, a 200 from the new URL. The 308 and the location line never appear, so you would see success and never learn whether the redirect was actually there. To test a redirect you have to stop on it, not follow it.

Click image to enlarge

Why keep the old URL alive: it protects the ranking and the crawl

[UNIVERSAL]

That redirect is permanent by design, not a temporary cleanup step. The old URL was live in production before the rename, so anything created during that window still points at it. That includes external links, bookmarks, and a search engine's earlier crawl. A permanent redirect also carries forward whatever ranking signal that URL accumulated. A bare 404 would discard it.

Alternative to Part 1 -- Universal version: frameworks with built-in rendering

[UNIVERSAL VERSION GUIDE EXAMPLE -- Next.js, Nuxt, SvelteKit, Remix, Astro, and any framework with built-in rendering]

These are guidelines, not tested code. Nothing below was run against a real Next.js build; it describes what the framework's own documentation states its APIs do. Verify each mechanism against your own project before relying on it.

Everything in Part 1 so far is this project's specific way of solving one problem: a crawler making a plain HTTP request needs real content in that first response, not an empty shell that only fills in after JavaScript runs. This project solves it with Vite, React Router, and a hand-written prerender.mjs. That is one way. For a new project today it is not even the most common way.

This section is for readers on a framework that solves the same problem automatically. Most commonly that means Next.js, App Router or Pages Router, but the same logic applies to Nuxt, SvelteKit, Remix, and Astro. The problem does not change. What changes is who is responsible for making the first response contain real content, and how much of it you build by hand. On this project, you build it. On Next.js, the framework guarantees it.

Why this matters for AEO specifically: the entire chapter rests on one fact, that an answer engine reads the raw HTML and never runs your JavaScript. A framework that guarantees real HTML in the first response has already solved the access half of AEO for you, before you write a single schema block. That is why a reader on Next.js can skip the hand-written prerender step here and go straight to Part 2, the content work.

Next.js App Router:

• No separate prerender script exists or is needed. Every route is server-rendered on each request, statically generated at build time, or incrementally regenerated, depending on how the route is configured. In all three cases Next.js itself guarantees the HTML response contains real content, not an empty shell. An answer engine crawling that URL gets the words on its first request.
• Metadata, which on this project is split between the <Seo> call and the prerender step, is declared in one place on Next.js: the built-in generateMetadata() function, or a static metadata export, in each route's page.jsx. Whatever title, description, and canonical you put in that export goes straight into the real server-rendered <head>. It is never rendered as JSX in the page body at all, so the empty-div problem this part opened with does not exist here in the first place.
• Structured data, the JSON-LD you will meet in Part 2, is added the same way it is on any stack conceptually: a <script type="application/ld+json"> tag whose contents are a JSON string. On the App Router that tag is rendered inside a server component, so it lands in the same real HTML response as everything else, with no separate step required to capture it. React requires a specific mechanism to emit a raw (non-escaped) JSON string as that script's contents; check the current React and Next.js documentation for the sanctioned way to do that, since the exact API has changed across versions.
• Verifying it worked: after next build, Next prints which routes are static and which are dynamic. But the real check is the same one this whole chapter insists on. Curl the live URL and read the body. "It renders in next dev" is not proof of what a crawler receives, exactly as "it renders in my browser" was not proof on this project.
• The failure mode from the case study, two separate route lists drifting apart, does not exist in Next.js the same way. There is one route table, the file system itself, so a route cannot be silently forgotten from a separate prerender list. The closer Next.js risk is a route's caching configuration silently serving stale or empty content.

How this happens: the App Router caches statically generated routes and fetch() calls by default. A route configured to revalidate on an interval serves the same cached HTML until that interval elapses. If the data behind it changes inside that window, every visitor and every crawler sees the stale version, with no error in any log. Worse, if the data fetch feeding a statically generated page fails or returns empty at build time, Next.js can bake that empty state into the cached page and keep serving it. That is the same "a successful build is not proof of a correct render" problem, triggered by a data fetch instead of a routing bug.

How to catch it: curl the live URL and read the body, not a browser tab. Two Next.js/Vercel-specific signals help: the x-vercel-cache response header (HIT/STALE/MISS tells you immediately whether the response was cached), and comparing any dynamic value on the page against what the data source currently holds.

How to fix it: set the revalidation interval to match how often the data actually changes, not a default copied from another route. Use on-demand revalidation, triggered by the real content-update event, rather than relying purely on a time-based window. For the empty-content case, fail the build explicitly when a required fetch returns empty, rather than shipping a blank page. Check the current Next.js documentation for the specific revalidation functions, since the App Router and Pages Router use different ones and both have changed across versions.

Next.js Pages Router (older, still common in existing projects):

• Metadata is set with a built-in head-management component inside each page, typically fed by a build-time or per-request data-fetching function. This is conceptually closer to a hand-authored metadata call than the App Router is, but it is still framework-managed: the built-in component is what actually reaches <head>, and the data-fetching function is what owns making sure the response has real content. There is no separate script to keep in sync with the route table.

Other frameworks (Nuxt, SvelteKit, Remix, Astro): the API names differ, but the single question this section answers applies to all of them. Does the framework guarantee real content in the raw HTTP response by default, which most modern meta-frameworks do, or is there a manual step responsible for making that true, which is closer to this project's situation? Answering that for your framework is the translation work. The verification method, curl the live URL and read the body, does not change.

What carries over completely unchanged: everything from Part 2 onward. The AEO content rules, the GEO trust work, the SMO platform choices, and the source-verification discipline in Part 5 are all about what a crawler receives and what an answer engine does with it, not about which framework produced the HTML. A Next.js reader skips only the hand-written prerender mechanics of this part, nothing else.

SEO, AEO, GEO: three layers, one page

[UNIVERSAL]

SEO, AEO and GEO are often written about as if they compete. They do not. They are three layers on the same page, each one depending on the one before it, and each one measured by something different.

Reading the table top to bottom is the order the work has to happen in. A page that cannot be crawled cannot be extracted from, and a page nothing can extract from will not be cited.

Part 1 of this chapter is the first row. The rest of this part is the second. Part 3 is the third.

Reading the last column is the more uncomfortable exercise, because those three metrics do not move together. Clicks can fall while citations rise.

That is not a contradiction, and the section on what a citation is actually worth, later in this chapter, deals with it directly.

Access before content

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

Before any content work matters, confirm nothing is blocking the crawlers in the first place. Three things to check, in order:

1. robots.txt. A bare User-agent: * with Allow: / and no Disallow rules blocks nothing. GPTBot, ClaudeBot, and PerplexityBot are not special-cased crawlers that need separate allow rules; they fall under the wildcard like anything else.

The real file this site serves is fully open. Four patterns cover every case, all four illustrative except the first, which is the live file.

User-agent: *
Allow: /

Sitemap: https://www.foliovistabooks.com/sitemap.xml

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Allow: /

User-agent: *
Disallow: /

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Allow: / is not proof of a citation, only permission for the fetch that could lead to one. A blocked agent can never cite the page, since it can never read it. An allowed agent might still read it and never cite it. robots.txt only ever decides the first half of that. The second half happens inside the AI provider's own systems, a different company's infrastructure this file has no reach into at all, and it is decided after the fetch, not by it.

robots.txt is not where citation is earned. The rest of this Part is: writing content in a shape an answer engine can actually lift and quote. This file only ever decides who gets to try.

Each blocked or allowed agent is one User-agent line plus one Disallow or Allow line. A rule only applies to the exact User-agent above it, so a misspelled name silently does nothing.

Two distinctions matter. Training crawlers and answer-time fetchers are different agents, so a site can be excluded from training while still being quoted in live answers, or the reverse. And Google-Extended is not Googlebot: blocking it removes the site from Gemini and Google's AI features while leaving ordinary search indexing untouched. Each operator publishes its own current list, and those lists change. Check them rather than copying this table in two years.

The table above sorts by operator. Sorting the same agents by what they actually do is more useful for deciding what to block, and it corrects a mix-up worth naming directly: a live fetch and a citation are not the same event. A -User agent fetching a page is a fact, visible in a server log. Whether that fetch turns into a named citation is a decision the AI system makes afterward, and it is invisible either way, the same point this chapter makes later in what a citation is actually worth and why you cannot see it. So "citation" is not its own category below. It is a possible outcome of a live fetch, never a guaranteed one, and never something a User-Agent name can confirm by itself.

Caution: six of these need more than the table above.

Diffbot's own documentation states it honors Disallow for a mass crawl, but a single URL a customer requests directly can still be processed even when that URL is blocked, and a partnership agreement can override robots.txt entirely.

Bytespider was measured, in a peer-reviewed study (IMC '25, arXiv:2411.15091), fetching robots.txt and not complying with it.

Meta-ExternalAgent sits in Training here, but Meta's own documentation describes it as used for training foundation models "or improving products by indexing content directly." There is no separate token that lets a site allow the indexing half while blocking the training half; blocking this one agent blocks both.

Meta-ExternalFetcher sits in Live fetch, and Meta's own documentation states this crawler may bypass robots.txt for a fetch a user directly triggered. Google documents the same design under the name "user-triggered fetcher," and names a direct analog: Google-Agent, described as being "used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request." Because the fetch was requested by a user, Google's own wording for this whole category is that these fetchers "generally ignore robots.txt rules," not that they always do. Same reasoning either way: the fetch is standing in for a person's own action, not an automated crawl.

What actually makes a fetch "user-triggered" is consistent across every fetcher Google names in that category. Feedfetcher "retrieves feeds at the request of services or apps installed by users." Google Read Aloud is "triggered by a user request," and its own page adds that "it doesn't follow links." Gemini Notebook fetches only the URLs "users have provided" as sources. Google Pinpoint fetches only the URLs "users specified." The wording repeats across every one of these pages because the underlying shape repeats. A person took one specific action inside a product. The fetch exists because of that action, and only because of it.

Google's own Googlebot page makes the contrast explicit, without ever mentioning the user-triggered category by name. Googlebot "discovers new URLs to crawl primarily from links embedded in previously crawled pages." Nobody asks Googlebot to fetch one specific page. It decides that for itself, by following what it already found on an earlier page, on its own schedule. That self-directed discovery is exactly the behavior robots.txt exists to govern. A user-triggered fetcher never discovers anything this way. It has no queue, no memory of what it fetched last, and no next page it plans to follow. It only ever requests the one URL a person handed it, at the moment they asked.

robots.txt being the wrong tool here is not just an assertion. It is what Google actually tells site owners to use instead, on the fetcher's own documentation page, every time. Feedfetcher's page states plainly that it "ignores robots.txt rules," for this exact reason, then names the real lever: serve a 404 or 410 status to the Feedfetcher-Google user agent, or block it at the hosting level. Google Read Aloud's page says almost the same sentence, "you can't opt out by using a robots.txt file," then names its own separate lever: a nopagereadaloud meta tag, plus the isAccessibleForFree structured data property for paywalled pages Read Aloud should not voice for free.

Same shape, every time. robots.txt governs a decision to crawl. A user-triggered fetch was never that decision, so Google builds a separate opt-out scoped to the actual feature that triggered it, a status code for Feedfetcher, a meta tag for Read Aloud, a verification token for Site Verifier. Never the crawl file, because the crawl file was never the thing making that decision in the first place.

Click image to enlarge

webzio-extended sits in Special, not Training, because Webz.io's own account of it describes a bot that does not crawl. It reads what a separate crawler, Webzio, already collected, and tags that data as usable or not usable for AI training. The name looks like a crawler; the documented behavior is closer to a flag.

ImagesiftBot sits in Ranking / index, not Training, because Hive's own page states its purpose as building "an index to enable search and retrieval of similar images," and never states a training purpose. It does respect robots.txt in the ordinary way, unlike the other five agents in this list.

The first four are about robots.txt itself not being a reliable stop.

The last two are a different problem:

The agent's own name suggests one function, and its documented behavior is another. Both are worth checking against the vendor directly rather than trusting a table, including this one, on sight.

Two more agents in the table above are worth linking directly, not because anything about them needed correcting, but because their vendor pages settle the question outright rather than leaving it to a secondhand summary.

Amazonbot is one bot with three names now, not one. Amazon's own page states Amazonbot itself "may be used to train Amazon AI models," while Amzn-SearchBot and Amzn-User each explicitly state they do "not crawl content for generative AI model training." Search, Alexa, and training used to sit on one token; they do not anymore, and the split is recent enough to date. Wayback Machine snapshots of that same page show a single, unified Amazonbot as late as October 2025, with training folded into that one token from around mid-2025 onward and no way to opt out of it separately from search. The three-token split, the exact structure documented today, was already live by February 2026. A distinction that took only months to appear can take only months to change again, or split further, the next time a new AI feature needs its own separate opt-out.

Applebot-Extended does not crawl at all, in Apple's own words: "Applebot-Extended does not crawl webpages. Applebot-Extended is only used to determine how to use the data crawled by the Applebot user agent." That page also notes that blocking it does not remove a page from Apple's search results, only from AI training use.

What actually makes a crawler worth blocking, and the mechanism that stops one that ignores this file, are covered next, right after this list.

2. Edge middleware and WAF-level rules. Check for any logic that inspects the User-Agent header and blocks or alters the response based on it. A content-security-policy nonce being swapped into <script> tags on every request is unrelated to crawler access. CSP is a browser-enforced policy, and a crawler that does not run JavaScript never evaluates it. It is still worth confirming directly rather than assuming. That logic sits in the same file as other request-time rules that could plausibly do something crawler-affecting. A plain curl request carries no browser User-Agent and runs no JavaScript, the same shape as a non-JS crawler's request, so its response answers the question directly, against the live site, not the dev server:

curl -sI https://www.foliovistabooks.com/ | head -1
# HTTP/2 200

A 200, not a 403 or a redirect, rules out anything in this file gating access by User-Agent. The body confirms the same thing a different way, real content coming back rather than an empty shell:

curl -s https://www.foliovistabooks.com/ | grep -oi "<title[^>]*>[^<]*</title>"
# <title data-rh="true">Digital Books, Practical Guides & Free Samples | FolioVista Books</title>

The real title tag, not a blank response, confirms the CSP nonce swap ran and nothing in the same file quietly blocked the request first.

3. noindex / nosnippet / max-snippet / data-nosnippet directives. These do the opposite of everything else in this chapter. They actively suppress a page from the exact AI snippets and citations this whole effort is trying to earn. A single stray noindex meta tag left over from a staging environment undoes everything else in this chapter for that page.

What makes a crawler worth blocking

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

robots.txt is a request, not an enforcement mechanism. Nothing forces a crawler to read it or obey it. A crawler that ignores it can only be stopped where the request actually arrives, which is a layer robots.txt has no access to at all.

Before reaching for that, it is worth naming what actually makes a crawler worth blocking, since "bad crawler" is not one thing.

Three separate questions decide it, and they do not move together.

Does it respect robots.txt at all? Most named agents do. A crawler that fetches robots.txt and then ignores the Disallow lines it finds is a different category of problem, because no amount of correct configuration in that file changes its behavior.

Does anything come back? A search indexer sends the site traffic. An answer-time fetcher can cite the site by name in a live answer, which is the entire goal of this chapter. A pure training crawler does neither. It reads the page once, folds it into a model, and the site gets nothing back, not a click, not a citation, not an acknowledgment. That is not automatically bad; it is simply a one-way transaction, and whether that trade is acceptable is a judgment call, not a technical fact.

Is it aggressive? A crawler making thousands of requests in a short window, ignoring a published crawl-delay, costs real bandwidth and server time regardless of what it does with the content afterward. This is the one category that can affect a site's own visitors directly, by slowing the site down or driving up hosting cost.

A crawler failing more than one of these at once, ignoring robots.txt, taking without returning anything, and doing it aggressively, is the actual case worth spending effort on. A well-behaved crawler that only fails the second question is a strategic choice about training data, not an incident.

This site's real robots.txt and its live middleware.js are both fully open right now, and that is deliberate, not an oversight. Nothing here is currently blocked. What follows is the mechanism that exists for later, if that decision ever changes, not a description of what this site currently does.

The mechanism: blocking a crawler in code, not just in robots.txt

[UNIVERSAL PRINCIPLE, with a FOLIOVISTA implementation and a Next.js implementation shown separately below]

The FolioVista implementation.

[FRAMEWORK-SPECIFIC: REACT/VITE, FOLIOVISTA EXAMPLE]

This project is a Vite app, not Next.js, so it uses Vercel's generic Edge Middleware convention directly, plain Request/Response, no next/server import. The diagram and code below are specific to that setup.

On a platform served from Vercel, the enforcement layer is edge middleware, the same file that would already be setting security headers on every request. Vercel Edge Middleware runs before a request reaches any page, so it can read the User-Agent header and refuse the request outright. This draft shows the idea, not a drop-in patch; folding it into a real middleware file correctly is more involved than this snippet alone:

const BLOCKED_AGENTS = ["Bytespider", "Diffbot"];

export default async function middleware(request) {
  const userAgent = request.headers.get("user-agent") || "";
  if (BLOCKED_AGENTS.some((agent) => userAgent.includes(agent))) {
    return new Response("Blocked", { status: 403 });
  }
  // the rest of the middleware's existing logic runs after this point
}

Two things about this draft matter more than the code itself.

The export has to match exactly what the platform expects. Vercel Edge Middleware requires a default export, not a plain named one, and the check has to sit inside that same function, ahead of whatever logic already runs there, not beside it.

And the check only runs for requests that actually reach this function in the first place. Most platforms let you configure which paths a piece of middleware applies to, often to skip static, prebuilt pages for performance. If that configuration excludes a path, a check placed inside the function never sees a request to that path, blocked list or not. Before relying on a check like this, confirm directly, the same discipline this whole chapter applies everywhere else, that the paths you actually want protected are the ones the configuration lets through to it.

This site's own config.matcher currently excludes /manuals/ entirely, the exact gap the diagram below names. That gap is not being closed right now, on purpose. Every manual currently published there is free content, meant to be public, so there is nothing behind that gap to protect yet. The fix, when it matters, arrives with the paid chapter, and it will not be a wider matcher or a longer BLOCKED_AGENTS list either. It will be the real auth session or token check this same section builds up to below, since that is the only mechanism that actually distinguishes a paying reader from anyone else, crawler or human.

The equivalent on a Cloudflare-fronted site does not have this gap, because a WAF rule sits in front of the origin entirely, before any per-path configuration decides what a middleware function even sees. A custom rule matching the User-Agent header against the same names, action Block, would apply to every path on the site uniformly.

Click image to enlarge

The same mechanism in Next.js.

[UNIVERSAL VERSION GUIDE EXAMPLE -- Next.js]

This is a guideline, not tested code. It describes what Next.js's own current documentation says, not a snippet run against a real build.

The first thing worth knowing is that the file's name changed recently. As of Next.js 16, this feature is called Proxy, and the file is proxy.ts (or .js), not middleware.ts. Next.js's own documentation explains the rename directly: the old name was too easily confused with Express-style middleware, so the file, and the exported function, were renamed to proxy to describe what it actually does, sit in front of the app on the network, before a request reaches any route. Next.js ships a codemod to migrate an existing project automatically. If a tutorial or an older codebase still says middleware.ts, that is the same feature under its previous name, not a different one.

Two things carry over from the FolioVista version unchanged, and one thing is actually more forgiving here.

The config.matcher option works the same way, with the same silent risk. A path left out of the matcher never reaches the function, on Next.js exactly as on the generic Vercel case above, and Next.js's own documentation states the default behavior plainly: with no matcher at all, the function "runs on every request," including static files and image optimization paths, which is its own separate footgun in the other direction, protection running somewhere it was never meant to and breaking something unrelated.

The actual blocking logic barely changes. Next.js's own documentation confirms a plain Response works for producing output directly, the same new Response("Blocked", { status: 403 }) line from the draft above needs no translation at all. NextResponse exists for the cases plain Response cannot do, redirecting, rewriting to a different route, continuing with NextResponse.next(), but it is not required for a flat block response.

The one place Next.js is more forgiving: the export shape. Where the generic Vercel case requires a default export specifically, Next.js's own documentation states the file "must export a single function, either as a default export or named proxy." Either shape works. The strict, easy-to-get-wrong gate from the FolioVista version does not have a Next.js equivalent in the same form, though getting the file's name wrong (an old middleware.ts on a version where that name has stopped being recognized) would fail the same way, silently, for every path.

Click image to enlarge

Every mechanism covered so far answers one question: which crawlers get to read this page. That covers robots.txt and the User-Agent blocklist in middleware alike. Neither one answers a different question: which humans get to read this page for free. That second question is the one that actually matters for anything meant to be paid, not given away.

A perfect User-Agent blocklist, one naming every crawler that exists today, would still let an ordinary browser through untouched. User-Agent blocking discriminates by software identity, not by permission. A human who never paid, browsing with an ordinary browser, is simply never on that list.

Auth sessions and tokens answer the question robots.txt and User-Agent blocking cannot. Instead of checking what software made the request, a session check asks whether this specific request carries a credential proving who made it, issued at login or at the moment of purchase. A crawler has no way to acquire one. Neither does an unpaid human. Both get the same response, no content served, regardless of User-Agent, and regardless of whether the crawler claims to respect robots.txt or says outright that it does not.

This is also why the user-triggered fetchers covered earlier in this chapter need no special defense of their own. Google's own documentation says plainly that robots.txt cannot stop Feedfetcher or Read Aloud, because the fetch is standing in for a person, not a crawl. A session check does not care what the fetch is standing in for. It only checks whether a valid credential came with the request, and none of these fetchers carry a paying user's session token, however they were triggered.

What makes a sentence liftable for AI, not just readable

[UNIVERSAL]

An AI system generating a response is not reading a page for atmosphere. It is looking for a sentence it can lift and attribute. Content that makes that easy shares a shape:

• A direct answer in the first sentence or two, not three paragraphs of throat-clearing before the actual point.
• Specific claims instead of vague ones. "Supports Linux VPS hardening, SSH timeout recovery, and browser rendering and domain safety" is citable. "Offers comprehensive security solutions" is not, because there is nothing in it another system could quote as a fact.
• Questions phrased the way a person actually asks an AI assistant. Not "Our Services" as a heading, but "What is the best book to learn bug bounty hunting fundamentals?" as an actual question, answered directly underneath it.

FAQPage schema, and why the visible version still matters

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

Structured data gives a machine-readable, unambiguous version of the same facts a page states in prose. That means FAQPage, Book, Organization and WebSite schema, implemented via JSON-LD (JavaScript Object Notation for Linked Data) <script> tags. It is not a replacement for the prose. It is read by different consumers. A JSON-LD block is invisible to a human visitor. A browser never renders <script type="application/ld+json"> as page content. Some AI systems and search features read structured data directly. Others primarily read the visible page text. Covering both is not redundant. It is covering two different reading paths with different content, buildFaqSchema in src/seo.jsx on this project:

export function buildFaqSchema(questions) {
  return {
    "@context": "https://schema.org",
    "@type": "FAQPage",
    mainEntity: questions.map((item) => ({
      "@type": "Question",
      name: item.question,
      acceptedAnswer: { "@type": "Answer", text: item.answer }
    }))
  };
}

The same applies to authorship and publish dates. A Book schema entry with author, datePublished, and headline fields gives a machine a precise, structured fact. A visible byline under the page's own heading, reading "Written by [name], published [date]", gives a human reader (and a text-only crawler reading the visible page, not just the schema) the same fact in plain text. Neither one substitutes for the other.

Two schema types worth adding, and one attribute worth reconsidering

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

FAQPage is the obvious type for a page answering questions, but it is not the only one that fits this kind of content.

HowTo describes a procedure as a sequence of discrete steps, each with its own name and text, optionally with tools, supplies and an estimated time. Any page that walks a reader through something in order is a candidate, and a technical guide is mostly made of such pages. The value is the same as with FAQPage: the prose already contains the steps, and the schema states where each one begins and ends rather than leaving a machine to infer it from heading levels and paragraph breaks.

The same caution applies as everywhere else in this section. Schema describes content that exists. Marking up steps a page does not actually contain is not optimisation, it is a misrepresentation, and the structured data will contradict the visible page for anyone who checks.

Image alt text is the attribute worth reconsidering, because it is usually filed under accessibility and left there. It is that, and it is also the only machine-readable description of what an image contains. A system reading a page without rendering it cannot see a diagram. It can read the alt text, and for a page whose central explanation is a diagram, that text may be the only description of the explanation that reaches a machine at all. A lazy alt="diagram" is not only a missed accessibility requirement, then. It is a missed AEO signal, in the exact same line of code.

Which changes how it should be written. "Diagram" or "screenshot" or "chart" satisfies a validator and communicates nothing. A description of what the image actually shows is a sentence a system can use. What is being compared, what the result was, which path is the failure case. It is also a better experience for a reader who cannot see the image. Both audiences want the same thing, which is the description rather than the label.

The /#faq mechanism -- why the fragment is not the target, the schema is

[UNIVERSAL PRINCIPLE, with a FOLIOVISTA implementation and a Next.js implementation shown separately below]

Two different things are easy to conflate here, and separating them correctly is the actual point of this section: whether a URL like /#faq reaches a crawler at all, and whether the FAQ content reaches an answer engine in a form it can cite. They have almost nothing to do with each other.

The fragment is not a page. #faq is a fragment identifier. The browser strips it before the request is even sent to the server. The server, and any non-JS crawler making a plain HTTP request, never sees it. /#faq and / are the identical request as far as a crawler is concerned. The fragment's only job is scrolling the browser to the element with id="faq" once the real page has already finished loading. That is browser-only, post-load behavior. It follows that /#faq does not need, and should never be given, its own sitemap entry, its own prerendered file, or its own indexing decision. None of those rules apply to anything after a #. They apply only to the URL in front of it.

So the real AEO question is not "does the FAQ get crawled." It already does. The FAQ content lives inside /, and / is already crawled. The real, separate question is whether that content reaches an answer engine as a structured, citable fact rather than just prose text mixed into the rest of the homepage. That is what a FAQPage JSON-LD schema block does: it turns the same content the fragment merely scrolls to into a discrete, machine-readable Question / acceptedAnswer pair, independent of the fragment entirely.

The FolioVista implementation.

[FOLIOVISTA-SPECIFIC]

Writing buildFaqSchema() and calling it from a page component is still not the same as confirming its output exists in what a crawler actually receives. The same distinction from Part 1 applies here directly: a <script type="application/ld+json"> block only counts if it survives into the prerendered HTML, not just the React source. The one-line proof, against the live site, not the dev server:

curl -sL https://www.foliovistabooks.com/ | grep -o "FAQPage"
# FAQPage

A hit confirms the schema survived prerendering and is present in the exact bytes a non-JS crawler receives for the homepage. That is the actual, intended, structured AEO source for that content. No hit means the schema only exists in the component source and never reaches a crawler at all, regardless of how correct the code looks, and regardless of whether /#faq "works" when clicked in a browser.

The same mechanism in Next.js.

[UNIVERSAL VERSION GUIDE EXAMPLE -- Next.js]

This is a guideline, not tested code. It describes what Next.js's documented rendering model makes possible, not a snippet run against a real build.

The fragment half of this section does not change on Next.js. #faq is stripped by the browser before the request, on every stack, so /#faq and / are still the same request to a crawler. Nothing to translate there.

The implementation half does change, and it gets simpler. On this project the schema has to survive a separate prerender step, which is why the curl ... grep FAQPage proof matters so much here. On Next.js, a server component can emit the JSON-LD script tag directly, and because that component is server-rendered, the tag lands in the first HTML response with no separate step required to capture it. The verification does not change at all. It is still the same one-line proof, run against the deployed URL rather than next dev:

curl -sL https://example.com/ | grep -o "FAQPage"
# FAQPage

A hit means the schema is in the bytes a crawler receives. On Next.js the reason it survives is that the framework server-rendered it, not that a prerender script captured it, but the thing you are checking, and the way you check it, are identical.

One consistent description, everywhere

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

A recurring theme across how AI systems decide what to recommend: consistency across every place a brand's identity is stated. If a website's schema-level description, its homepage's own visible text, its Google Business Profile listing, and its social profiles each describe the platform slightly differently, that is not four sources confirming the same fact. It reads as four different, unreconciled claims about what the same entity is. The fix is not clever copy. It is picking one accurate sentence and using the identical sentence everywhere that entity gets described, then actually checking, byte for byte, that every instance matches:

grep -rn "your unified description text here" src/

If that command returns fewer matches than the number of places the description should appear, something drifted and needs fixing before moving on to anything else.

Repetition is not the same as consistency

[UNIVERSAL]

Consistency means the same fact, stated accurately, everywhere it belongs. It does not mean the same sentence, copy-pasted onto every section of a single page. A homepage that repeats one descriptive phrase in its hidden accessibility text, its H1, its overview section, and its FAQ answer is not being consistent. It is repeating itself, and a crawler reading that one document sees the same claim four times with no new information on the second, third, or fourth occurrence. Worth checking directly, since a phrase that is invisible to a sighted visitor (screen-reader-only text, hidden via a standard clip-path accessibility pattern) is not invisible to a crawler reading raw HTML. CSS visibility rules mean nothing to a system that never renders CSS in the first place.

What generative engines actually pull from

[UNIVERSAL]

Generative engines cite from a narrower set of source types than "the whole web," and the pattern repeats across providers: reference data from high-authority sources (encyclopedic reference sites, established publications), aggregated community discussion (forums, Q&A sites), video platforms, and professional-network content tend to dominate what gets cited across nearly every category, for nearly every major generative AI provider. Concretely, for a security-adjacent publisher, that means the highest-leverage moves are not necessarily on the publisher's own site at all.

They are presence on the platforms an AI system is already disproportionately likely to cite from. That does not mean pursuing all of them indiscriminately.

A reference-encyclopedia entry requires clearing a notability bar that a small, pre-revenue publisher will not clear, and chasing it wastes effort that could go toward something achievable. A forum-community presence in a technical niche can be a real, high-value channel. These are the same communities where a target reader already asks their questions.

Those communities are also fast to detect and penalize anything that reads as self-promotion, and a clumsy attempt to seed a listing without an established, trusted presence risks real reputational damage, not just wasted time. The honest move is engaging a genuinely sensitive community the way an actual member would. That means answering questions, contributing real expertise, and disclosing affiliation transparently when relevant. It does not mean treating the community as a distribution channel to seed and walk away from.

First-party content: the excerpt is not a compromise

[UNIVERSAL]

The single most repeated, most concrete piece of guidance across this whole subject: content already owned can be repurposed into standalone, citable pieces without writing anything new. A chapter already sitting inside a published book is not locked to that book. Cut loose, retitled, and published as its own page, it becomes a standalone asset a generative engine can cite independently, while the parent book remains the deeper resource for a reader who wants more. This only applies to content actually owned outright, meaning an author's own book rather than someone else's. Within that boundary, it is close to free: no new research, no new writing, just a publishing decision.

Content patterns that read as trustworthy to a citation-seeking system

[UNIVERSAL]

A handful of concrete, structural patterns recur across content that generative engines actually cite from:

• Named authorship with a visible byline and date. Not just in schema. It belongs in the actual page text, where both a reader and a text crawler can see it.
• Specific numbers in tables, not vague ranges buried in paragraphs. A reader, human or machine, can quote a table row. A paragraph of hedged prose is harder to extract a fact from.
• Direct comparisons. "X versus Y" framed honestly, rather than as a hidden sales pitch, gives a system something concrete to reference when a user asks a comparative question.
• A bolded, one-sentence "key takeaway" immediately after a dense technical passage. That is exactly the kind of self-contained, quotable sentence a generative engine is built to extract and repeat.

None of these require new expertise to implement. They are structural choices about how existing expertise gets formatted.

E-E-A-T, and the one letter that matters most here

[UNIVERSAL]

Those patterns are not arbitrary. They are surface expressions of a framework Google documents directly, in its guidance on creating helpful, reliable, people-first content and in its Search Quality Rater Guidelines. The acronym is E-E-A-T:

Read Google's own documentation for this rather than a summary of it. The framework is widely paraphrased, and the paraphrases tend to add mechanical claims about how models filter sources that the documentation does not make.

The first E is the one worth dwelling on, because it was not in the original framework. For years this was E-A-T, three letters, and Experience was added later as a separate pillar from Expertise. The distinction it draws is precise: expertise is knowing a subject, experience is having done the thing.

That distinction is the entire argument for writing this kind of book at all. A model trained on the whole web can produce competent, accurate, well-organised text about recovering a locked SSH session. What it cannot produce is the account of the time you actually locked yourself out, what the terminal printed, which of your first three guesses were wrong, and what the fix turned out to be. Expertise can be synthesised from other people's writing. Experience cannot, because it is a record of events that either happened to you or did not.

Every case study in this book exists for that reason, and so does the decision to print a citation count of zero rather than omit it. The commands are real because they were run, the outputs are real because they were returned, and the failures are included because they happened. None of that is a writing style. It is the only pillar of the four that a competitor cannot generate.

The practical consequences are unglamorous. Put a real name and a real date on the page, in text a reader can see rather than only in schema. Say what you actually tested and what the result was, including when the result was that nothing happened. Keep dates on figures so a reader can judge whether they still hold. Correct things publicly rather than quietly. Each of those is a small act of showing your work, and showing your work is what the first E measures.

What a citation is actually worth, and why you cannot see it

[UNIVERSAL]

This section is titled around earning a citation rather than a visit, and that distinction has to be made concrete before any of the work above is worth doing, because the honest answer is uncomfortable.

A citation rarely becomes a click. In the Pew Research Center brief published 22 July 2025, users clicked a link inside an AI summary in about 1% of visits. Being named, quoted and linked in the answer itself produced a visit roughly one time in a hundred. Getting cited perfectly, by every engine, on every relevant question, does not restore the traffic that the summary absorbed. Citation and click are close to unrelated outcomes, and any plan that treats the first as a route to the second is built on a misunderstanding.

The measurement problem follows directly from that, and it is the part most likely to cause a false conclusion later. Analytics measures visits. Google Analytics, or any equivalent, records a session when a browser arrives at the site. A citation that produces no click generates no session, no referrer and no event. It is not that the tracking is configured badly. There is nothing arriving to track. The 99% of citations that never produce a click are structurally invisible to every analytics tool, by design, because analytics was built to measure traffic and a citation is not traffic.

The practical consequence: anyone assessing this work from an analytics dashboard will conclude it failed, because the dashboard cannot display the outcome it succeeded at. Setting up a custom channel group to isolate referrals from chat.openai.com, perplexity.ai and claude.ai is still worth doing, and it is worth being exact about what it captures. It measures the small minority who clicked. It says nothing about how often the platform was named to someone who did not.

What can be checked instead is direct, manual and unglamorous. Ask the engines the questions a real reader would ask, in the wording a real reader would use, and record whether the site is cited, how it is described, and which page was referenced. Repeat it on a schedule and keep the answers. It has to be built by hand, because no analytics product can report on a visit that never happened.

One partial exception is worth knowing about, and it sits outside analytics entirely. Bing Webmaster Tools carries a report labelled "AI Performance (Beta)," which shows citation visibility rather than traffic. It reports where a site was surfaced in AI-generated answers, broken down by intent, by topic, and by citation share. That is precisely the thing a session-based tool cannot see. Note the beta label, and note that it covers Bing's own AI-generated answers only, nothing from ChatGPT, Gemini, Perplexity or Claude. Google Search Console exposes no equivalent at the time of writing, so this is one window rather than a solved problem. Check whether that has changed by the time you read this, because it is the kind of gap a search platform has an obvious reason to close.

Setting it up is worth describing plainly, because the experience is not what the announcement suggests. Bing Webmaster Tools is, for almost all of its surface, a second Google Search Console. Sitemaps, index status, crawl errors, query performance: the same reports, the same shapes, the same workflow. Verification even offers to import the existing Search Console property directly, which is the fastest route if a Google property already exists, and the DNS TXT verification method works identically otherwise. Anyone who has done the Search Console setup has already done this one.

That duplication is the point worth being clear about. Nearly everything in the tool is a second copy of reporting that already exists elsewhere, and none of that duplication justifies maintaining another account. The AI reporting does, on its own, because there is currently nowhere else to get it. Set it up for that one report and treat the rest as a duplicate.

Expect it to be empty at first. The site has to be crawled before most reports populate, and the AI report additionally depends on pages actually being surfaced in AI answers rather than merely indexed. Days, not minutes. If a chapter, a product or any other page is about to be published, verify the property before that happens rather than after, because the value of this report is almost entirely in the comparison between two dates, and the earlier one cannot be reconstructed later.

The general shape of the problem holds regardless. Citation measurement lives in webmaster and platform tools where those exist, in manual logging where they do not, and never in an analytics dashboard built to count arrivals. Useful proxy signals do exist in analytics, and they are proxies rather than measurements: growth in branded search, direct and returning visits, and conversions that a visitor completed without an obvious referring click.

The uncomfortable finding this section has to answer

[UNIVERSAL]

Start with the evidence rather than the recommendation, because the evidence points somewhere this section does not entirely follow.

The Pew Research Center brief published 22 July 2025 identified the sources cited most often in AI summaries, and the same three lead both AI summaries and ordinary search results: Wikipedia, YouTube and Reddit. Not vendor sites, not company blogs. An encyclopedia, a video platform and a forum.

That is awkward for what follows, because this section deprioritises two of those three. It is worth being explicit about why, rather than quietly recommending against the platforms the data favours and hoping a reader does not notice.

The reason is not that the finding is wrong. It is that "most cited" and "achievable for a small operation" are different questions, and this chapter answers the second one.

An encyclopedia entry requires clearing a notability bar that a small, pre-revenue publisher will not clear, and attempting it wastes effort that could go somewhere achievable.

A forum with a pseudonymous, high-scrutiny culture is a genuinely valuable channel and also one where unmoderated self-promotion is reliably detected and penalised by the community itself. Without an established presence there, a clumsy attempt costs reputation rather than just time, and reputation is the one asset a small operation cannot rebuild quickly.

So the trade being made here is real and worth naming plainly: the larger share of citation opportunity is being deferred, deliberately, in exchange for not spending credibility that has not been earned yet. That is a sequencing decision rather than a claim that those platforms do not matter. They matter most. They are also the ones where arriving badly is worst.

The third platform, video, is a different case entirely and should not be counted as a strategic deprioritisation at all.

[FOLIOVISTA-SPECIFIC incident, universal lesson -- the specifics below are this project's own account and its termination, but the mistake that caused it is one anyone can repeat, with the same permanent result]

This project's channel was terminated permanently, which removed the option rather than declining it. That episode is documented in full below, because how it happened is more instructive than the fact that it did, and because the mistakes that caused it are easy to repeat.

A reader with an existing presence anywhere in that list should reverse this ordering without hesitation. The constraint being worked around is having no standing to spend, and it does not apply to someone who already has some.

Click image to enlarge

Video looks like the most achievable of these channels, and this project's own attempt at it ended in permanent removal. The account is worth giving in full, because the failure mode is one no guide on this subject mentions and it is not recoverable.

The plan was ordinary. A dedicated channel, a description matching the platform's identity elsewhere, and a short explainer video presented by an AI-generated voice and avatar, on the reasoning that imperfect production is not a blocker to having a first piece of real content somewhere the citation data favours. Alongside it, three comments were left on other creators' videos covering related technical ground, each mentioning a free chapter and linking to the site.

Three comments. Written by hand, not automated, on genuinely relevant videos. The channel was terminated for "spam, deceptive practices and scams."

The important part is why three was enough, because the intuition that small numbers are safe is what makes this trap work. Detection here is structural rather than volumetric. The three comments shared a template: a compliment, a mention of the free chapter, the same link. Same shape, same destination, different channel each time. That is the pattern the policy describes, and the policy's own example says "across hundreds of videos," which is exactly the number that makes a reader assume three is fine.

The comments were probably not the heavier half. Consider what the account looked like from outside, as a combination rather than as separate items: a new channel with almost no history, whose main content was presented by a synthetic voice and avatar with no disclosure, posting templated promotional comments that drove traffic to an external site. Nothing in that list is unusual on its own. Together they match two categories the same policy names explicitly, synthetic mass production and off-platform diversion. An automated system does not see an author doing modest manual outreach. It sees the shape of a promotional network.

Note which word in the policy title covers which behaviour. The comments are the "spam" half. An undisclosed synthetic presenter is the "deceptive practices" half, and that is the more serious of the two.

An appeal was filed. It admitted both issues directly rather than denying anything, acknowledged the comment pattern, acknowledged the missing disclosure, and committed to correcting both. It was rejected. The outcome was not a strike or a warning, and not limited to the channel:

We have permanently removed your channel from YouTube. Going forward, you won't be able to access, possess, or create any other YouTube channels.

That is account-level and permanent, and it skipped the escalation ladder the platform's own documentation describes, where warnings precede strikes and three strikes within ninety days precede termination.

Three things worth taking from this, none of which require any claim about what the platform intended.

Disclose synthetic content, every time, without waiting to be asked. This is the item most relevant to a book about optimising for AI systems: using AI tooling to produce public-facing content carries platform obligations, and an undisclosed AI presenter is treated as deception rather than as a production shortcut. The cost of the disclosure is one line. The cost of omitting it, in this case, was the account.

Assess promotional actions by their shape, not their volume. Three identical-looking comments and three hundred trip the same detector. Before posting anything promotional across multiple destinations, ask what it would look like to a system that cannot see intent.

Do not let a moderated platform hold anything you cannot afford to lose. Not the only copy of a video, not an audience you have no other route to, not a link that other pages depend on. The termination arrived by email, the appeal was answered in days, and there was no further route. Any platform in the citation-favoured list can do the same thing, and the guides recommending you build presence there do not mention that this outcome exists.

Professional-network presence is a second channel achievable without a large team. That means both a personal profile actively posting about the work, and a distinct page for the platform itself. It is also one that a founder can build credibly in their own voice, since it is genuinely their own expertise being represented, not a fabricated brand voice.

The format choice within that profile is not neutral. LinkedIn's long-form Articles get indexed and stay listed on a public profile indefinitely, so a reader who looks the founder up months later can still find one. A regular post sinks in the feed within days and is effectively unfindable again after that. An article also carries more perceived depth, since the format itself signals a fuller argument, which suits a technical walkthrough better than a short take does.

The trade-off runs the other way for reach. LinkedIn's own feed distribution favors short native posts, so an article gets less immediate visibility, and its extra length costs some click-through unless the topic is compelling on its own. The practical fix is not choosing one over the other. A short post teases the topic and links to the fuller article, earning the feed's immediate reach and the article's long-term discoverability from the same piece of writing.

Community forums with pseudonymous, high-scrutiny cultures are the one category worth deliberately deprioritizing rather than awkwardly forcing. The reasoning is not "these platforms are dangerous" as a blanket claim, which overstates it. The reasoning is narrower and more practical: unmoderated self-promotion in a technical community is reliably detected and penalized by the community itself, and a small operation without an existing trusted presence or professional reputation-management support is better served waiting until that support exists than attempting a DIY listing and risking the account, or the brand's standing in that community, over it.

The detection is not a person reading every post. Many of these communities gate posting behind a minimum account age and karma threshold specifically so a brand-new account cannot arrive and post a link on day one, and the ones that do not gate that way lean on automated spam filters that flag the same external domain showing up across multiple posts, regardless of how new or old the account behind each one is.

A founder's own account is new to any community it has not already spent real time in, by definition. The exact signal an age-and-karma gate exists to catch, a fresh account whose first visible action is linking an outside site, is indistinguishable from the outside from a founder posting about their own new platform, no matter how genuine the post actually is.

The cost is not graduated the way a warning system would be. A flagged post or account gets removed outright, and the same domain can end up filtered across other communities too. One clumsy attempt does not just fail once. It can quietly cost every later post linking that same domain, in communities never even tried yet.

This is a sequencing decision, not a permanent one.

A reference encyclopedia is worth naming explicitly as a channel to not chase yet. Its notability bar is real, well-documented, and not cleared by a small, pre-revenue operation.

Wikipedia's own guideline is specific about what clears that bar: significant coverage in reliable sources that are independent of the subject. Not a mention. Not a press release. Not the platform's own website, an interview it gave, or a routine business listing. Real, substantial coverage written by someone with no connection to the platform, published somewhere with editorial standards.

That requirement is circular for a small, new platform by design, not by accident. Independent coverage is usually a consequence of a company already being notable, a funding round covered by a tech outlet, a product written up by an established journalist, a milestone that made outside news on its own. A pre-revenue platform has not accumulated that coverage yet, which is the same thing as not clearing the bar, regardless of how good the product is.

The downside of trying anyway is not neutral. A page created before that coverage exists is a normal target for deletion under the same notability guideline, and the deletion discussion is public and stays indexed. That is a worse outcome than never having a page at all, a permanent, searchable record of "not notable" rather than simple absence.

Time spent trying is time not spent on the channels above that are actually achievable right now.

Every profile that does get built should link back to the same canonical properties everywhere it appears. That means the same website, the same one-sentence description used consistently across every other property, and cross-links between the profiles themselves where the platform allows it. This is the same consistency principle from Part 2, applied to the platforms outside the website itself instead of the pages inside it.

A closing note that belongs in this chapter specifically because it was learned directly, not read secondhand: a meaningful share of publicly available AEO/GEO guidance is delivered by people and companies selling AEO/GEO consulting or tooling.

That does not make the guidance wrong. Most of what a seller says can still be independently correct. It does change how it should be received.

A presenter who cannot answer a direct technical implementation question without deferring to "my technical team," or who has to search live, on screen, for the same tool recommendations being pitched moments earlier, is demonstrating something worth noticing: the information is downstream of a sales motive, not necessarily wrong, but not independently verified either.

The practical response is not cynicism about the whole topic. It is the same discipline this book applies everywhere else.

Treat a vendor's claim as a hypothesis, then check it directly: read the actual source code instead of trusting a description of what it does, curl the live page instead of trusting a screenshot, run the exact verification command instead of accepting "trust me, it works." Every concrete tactic in this chapter was confirmed this same way before it was written down here.

Not because the sources were assumed dishonest, but because a claim that has been personally verified is worth more than the same claim taken on trust, and the verification itself is usually a five-minute command, not a research project.

The same discipline applies to a second source type, not just paid AEO/GEO consultants: free SEO audit tools, which run against a URL and hand back a numeric score with a list of "issues." Two such tools, run against this same site, flagged 17-23 links for missing a title attribute and flagged repeated internal anchor text. Both showed upgrade prompts mid-audit ("test free for 14 days," "save 30%").

Checked against Google's own documentation and current accessibility guidance rather than implemented on the tool's word: the title attribute carries no ranking weight and is not reliably read by screen readers at all, and the repeated-anchor-text concern in real SEO practice is about an external backlink profile, not a site's own nav repeating "Home" in its header and footer.

A free tool with a score to show and an upgrade to sell is not a hostile source, but it is not a neutral one either. The same "verify before implementing" rule that applies to a paid consultant's slide deck applies just as much to a freemium tool's red X marks.

A third source type deserves the same treatment, and it is the one most likely to reach a reader searching this topic: the free vendor ebook.

"How to Optimize Content for GEO and AEO in an AI-Native World," published by Jasper, a company selling AI content-generation tooling, was read in full, and every statistic in it was traced back to the primary source it cited.

Five claims, five sources, checked one at a time against what those sources actually say. Every source named below is listed in Further reading, so the same check can be repeated rather than taken on trust here.

Not one figure was invented. Every number exists in the research it was attributed to. What did not survive the retelling was the context sitting immediately beside those numbers.

The guide reported that click-through, the share of people who see a search result and actually click it, often written click-through rate (CTR), falls to 8% on searches where an AI summary appears. It left out two other numbers from the same brief, measuring two different things:

• The same CTR metric, measured instead on searches with no AI summary at all: 15%, nearly double.
• A completely separate figure: how often a summary shows up in the first place, only around 18% of searches, meaning the other 82% never trigger one.

All three numbers come from the same Pew Research Center brief, published 22 July 2025. It reported a 357% year-over-year surge in AI referral traffic, while the same Similarweb article it drew from, published 29 July 2025 and comparing June 2024 with June 2025, stated that those referrals, the actual count of website visits arriving from AI systems across the web that month, totalled 1.13 billion against 191 billion from Google search in that same month. The comparison did not carry across.

Both numbers below come from the same Adobe survey too, fielded 29 May 2025 among 1,000 US respondents already screened as ChatGPT users:

• 77% of ChatGPT users "rely on it for search," which measured only that they had tried it.
• Only 24% go to it first, the more meaningful measure of that same survey.

One case is worth separating from the others, because the failure mode is different, more instructive, and because the source has since revised its own answer.

A widely repeated figure holds that AI Overviews reduced click-through by 34.5%. That number is not a measurement. In the Ahrefs study it comes from, published 17 April 2025 and comparing March 2024 with March 2025 across 300,000 keywords, the researchers modelled what click-through on AI Overview keywords should have fallen to if it had declined at the same rate as keywords with no AI Overview, and the 34.5% is the gap between that forecast and the observed result. It is a counterfactual, not a before-and-after on the same keywords.

Ahrefs then re-ran the study on December 2025 data, and the figure moved to 58%. Same 300,000-keyword sample, same method, a wider window: December 2023 against December 2025, because by December 2024 the rollout had already happened and no clean baseline remained. Position-one click-through for keywords with no AI Overview fell from 0.076 to 0.039, a decline of roughly 49%. For keywords that do trigger one, it fell from 0.073 to 0.016, roughly 78%. The modelled counterfactual is 0.037, and the 58% is the distance between that and the observed 0.016.

Two things are worth taking from that revision rather than one.

The first is that the widely quoted 34.5% has been superseded by the organisation that produced it. Note the sequence before drawing the wrong conclusion from it: the ebook was published on 10 September 2025, when 34.5% was the current figure, and the revision came afterwards. It was not quoting a stale number. The number went stale underneath it, which is what happens to every figure in a field moving this fast. What matters is that anyone repeating 34.5% today is passing on a figure its own source has already replaced, and the only way to know that is to open the source rather than the summary.

The second is fairer to Ahrefs than a first reading of their work suggests. The update addresses the decline on keywords with no AI Overview directly, in its own section, attributing it to a longer history of search features that resolve a query without a click. It adds a position-by-position breakdown showing the effect running from 58% at position one down to about 19% at position ten.

It cites independent work landing in a similar band, roughly 47% to 65%, from methods other than its own. That is a research team disclosing the confounder rather than burying it, and the underlying effect is real, large, and corroborated outside a single company's model.

What has not changed is the part worth teaching. The headline number is still a model's output rather than an observation, and it is still repeated elsewhere as though it were measured directly. The distortion does not happen in the research. It happens in the retelling, and it survives every revision of the underlying figure.

The pattern across all five is worth stating carefully, because the temptation is to overstate it. The data these companies publish is mostly sound. The framing around it leans in proportion to how much they have to sell. The most balanced numbers in the set came from a research organisation with nothing to sell at all. The single strongest study, published by Semrush on 11 August 2025 and built on 260 billion rows of opted-in clickstream data from January 2024 to June 2025 with a never-adopted control group, is also the one that most contradicts the panic: adopting ChatGPT did not reduce Google usage, and total search activity rose. It was published by a company that sells SEO tooling. The research survived the commercial incentive. It was the retelling that bent, and the retelling was done by the one company in the chain selling AI writing tools.

None of this establishes motive, and it does not need to. Whether a missing baseline was a deliberate choice or an ordinary editing decision cannot be determined from outside the company that published it. The observable result is enough on its own: five claims traced back to their sources, four materially altered, every alteration pointing the same way. That is the finding worth trusting when you run this check against a source of your own, and it is the one you can show to someone else.

One vendor is a case, not a pattern, so a second was checked the same way, and it turned out to fail differently. Lumar publishes a four-pillar GEO framework across five free pages, dated between 10 April and 14 May 2026, drawn from an 80-page gated guide. Lumar sells a technical crawler, and it shows: the first pillar is Technical GEO, and it names the crawler user agents individually, covers robots directives including nosnippet, and describes the JavaScript rendering blind spot accurately enough that a reader would understand the risk.

It also cites no statistics at all. Across all five pages there is not one percentage, not one measured figure. Where the first ebook bent numbers, this one uses none, resting instead on named expert quotes and links to primary documentation. On the terms of the check applied above, there is nothing to catch.

The commercial pressure has simply moved somewhere else. Every page carries the same two calls to action, the gated download and a sales demo, repeated two or three times each, and on the technical page the demo is pitched three times against eight mentions of the guide. More telling than the frequency is where the pitch lands. The verbs throughout are check, ensure, confirm, audit and review, and not one of them is followed by a method. At the exact point where a reader needs to know how to see what an AI crawler actually receives, the answer offered is the publisher's own reporting product. There is no independent way to run the check. The word "prerender" does not appear anywhere across the five pages, so the mechanism that actually solves the rendering problem the guide correctly identifies is never named.

Two failure modes, then, and the second is the harder one to notice. The first bends a number, and a careful reader catches it by opening the source. The second states an accurate problem, withholds the method, and supplies a product in the method's place. Nothing in it is false. A reader simply cannot act on any of it alone, which is a quieter way of reaching the same commercial destination. When a guide tells you to verify something and the only verification on offer requires an account, that is the pattern, and it is worth recognising as clearly as a missing baseline.

The five-source verification diagram

Click image to enlarge

One more source deserves the same treatment, and it produced a different kind of error entirely. SocioApt's 2026 statistics roundup states that "a 2026 study analyzing over 300,000 keywords found that the presence of an AI Overview reduces the CTR of the #1 organic result by 58%," attributed to GrowthSRC. The number is real. The attribution is not. A 300,000-keyword sample and a 58% position-one decline is Ahrefs's own December 2025 update, already checked above.

GrowthSRC ran a real study of its own: 200,000-plus keywords across 30 sites, comparing 2024 against 2025, finding position-one CTR down 32%, from 28% to 19%. A different sample, a different year range, a different number, from a real study that exists and was never the one being quoted.

Neither figure was invented. What happened sits between the two studies rather than inside either one: somewhere in the retelling, Ahrefs's number kept its digits and picked up GrowthSRC's name. That is a distinct failure from a vendor bending a number toward a sales narrative or withholding a method behind a product. Nothing here points to a commercial motive on either source's part, and both underlying studies check out on their own terms. It is the plainest version of the lesson this whole section keeps returning to: a number checking out is only half the check. Confirming a figure is real and confirming which study it actually came from are two separate steps, and skipping the second one is exactly how a fully accurate statistic ends up attached to the wrong name.

Why this isn't the search engine's problem to fix

The same restraint applies when the subject is a company large enough to be an easy target. Around 18% of Google searches triggered an AI summary at the time of measurement, and summaries appeared far more often on longer, question-phrased queries. A tempting reading of that limited share is that Google is holding AI back to protect its own search business. The reading does not survive contact with how Google actually earns money.

Organic results are unpaid. A visitor who clicks through to a publisher's site generates no direct revenue for Google. What generates revenue is the advertising slot, and Google has been extending advertising into its AI surfaces rather than leaving them unmonetised, while also shipping a dedicated AI mode in search. The click an AI summary absorbs was never Google's revenue. It was the publisher's traffic. None of this is unique to Google. Any search engine whose revenue comes from advertising rather than the organic click carries the same incentive, on Bing and any other ad-funded search engine alike. Google is simply the one with verified numbers behind it here, because Google is what Pew, Ahrefs, and Semrush actually measured.

That correction matters more than it first appears, because it identifies who actually carries the cost of this shift. It is not the search engine. It is every site that used to receive those clicks, which includes yours.

Worth separating clearly here, since this whole section has been full of percentages: a click and a citation are not the same event, and every figure checked so far, Pew's 8% and 15%, Ahrefs's 34.5% and 58%, measures only the first one. A click is a visit, countable in a site's own analytics, whether or not it came from a link inside an AI summary. A citation is a site being named or quoted inside an AI system's answer, and that can happen with no click attached at all, because the person already has what they came for. Every number in this section is measuring the same direction of loss: fewer clicks. Whether citations are rising enough to offset that is a separate question, covered earlier in this chapter under What a citation is actually worth, and why you cannot see it, and the honest answer there is that a citation is real but far harder to verify or rely on than a click ever was. The two are not a trade a site can count on balancing.

Click image to enlarge

Compete where the click is actually available

None of what follows restores the click the diagram above just showed disappearing. It is not a fix for that loss. It is where the effort actually goes once waiting for the platform to reverse it stops being a plan.

An AI system can fully answer a single-fact question in one paragraph, and once it does, there was never a click coming for that page, no matter how well the page was written. It cannot substitute for a tool someone has to actually use, an account someone has to log into, a purchase, a download, a personalized result, or a decision that requires comparing several live options at once. Splitting a site's own pages along that line, which ones are a single fact and which ones require the actual product, changes what is worth measuring by click volume and what is not.

Traffic that never routes through a search query in the first place never reaches the point where an AI system can intercept it. A direct visit from someone who already knows the brand, an email subscriber, a community member: none of that passes through a "does the AI answer this instead" gate, because there is no query in the middle to answer. This is the same reason Part 4 of this chapter, on which platforms, and which to deliberately skip, matters more as this shift continues, not less.

A citation without a click still carries some value, just not a click's value. What a citation is actually worth, and why you cannot see it, covered earlier in this chapter, makes the case that being the name an AI system consistently attaches to a topic builds recognition on the visits that never happen, the same way being quoted in print without a link still builds a reputation. Real, but not a substitute for the traffic that is gone, and not something a founder can point to in an analytics dashboard.

None of this is a fix. It is a redirection of where the effort goes, once the click on that specific interaction is no longer available to compete for.

Waiting for the platform to restore the traffic assumes the platform lost something, and it did not. Whatever explanation exists for the current rollout, and no confident one is available from outside the company, it is not that Google is defending the clicks on your behalf.

One caveat belongs with that 18% figure, and with every other number in this section. These are snapshots, measured on specific dates, mostly on US-only panels, and in one case desktop only. "Most searches do not show an AI summary" is accurate as measured and is not a stable property of the web. Any figure quoted here should carry the date it was measured, so a reader in two years knows whether it still describes anything real.

That caveat is worth taking seriously rather than treating as a formality, because a longer-running dataset shows exactly why it matters. SparkToro's zero-click research, tracking US Google searches since 2016 across three different clickstream panels, two of which no longer exist in their original form, measured 45% of searches ending without a click in 2016, 50% in 2019, 60% in 2024, and 68% by early 2026. The panels are not directly comparable, and the research says so openly rather than smoothing it over.

A public exchange on the same post shows that discipline in practice: a reader pointed out the decline looked steeper than a competing report, and the disclosed answer was that the competing report measured desktop only, while this dataset adds a mobile panel, which is where more of the decline concentrates. That is the same "check which panel, which device split" caution this section has been making, demonstrated in public rather than only described. It is also the source that states most plainly what follows from all of this: getting better at SEO does not reverse a trend this size, and the traffic is not coming back. What is still worth doing is exactly what the next part of this chapter turns to.

Why well-resourced content keeps winning, whether or not it is the most accurate

Search the exact question this chapter answers, "what is AEO," and the page that shows up first is very often built by a company that sells AEO tooling. That is not a broken system rewarding the wrong content, and it is not proof that the winning page is inaccurate. It is a more specific problem than either of those, worth naming precisely rather than waved at.

Ranking and citation systems reward structural signals: an answer-first opening, schema markup, recent update dates, a dense trail of citations, consistent formatting across many pages. None of that is a measure of whether the underlying claims are correct. It is a measure of whether the content was built the way this chapter, and every source examined in it, describes building it.

That claim is not asserted on faith here. A peer-reviewed study, "GEO: Generative Engine Optimization", presented at KDD '24 by researchers from IIT Delhi, Princeton, and two independent researchers, tested nine content-modification strategies against a 10,000-query benchmark built for the paper and measured how much of a generative engine's own response text each strategy earned a source. The three strategies that worked best, adding citations, adding quotations, and adding statistics, each improved that measure by roughly 25 to 27% over an unmodified baseline, up to 40% on some queries, while classic SEO keyword stuffing measurably hurt it. That is direct evidence for the structural signals named above.

What it is not evidence for is a click. The paper's own limitations section states plainly that the study never measured how these changes affect search rankings or real-world traffic, only how much of a generative engine's own text a source earned in a benchmark built to measure exactly that. A 40% visibility gain and a click are not the same unit, and treating them as interchangeable would be the same mistake this whole section has spent its length checking for.

Those signals are expensive to produce at volume and keep fresh. A company whose entire revenue depends on ranking for its own core topic has both the direct incentive and the resources to produce them continuously, at a level an individual writing from real experience usually cannot match, regardless of which one of them actually has the better answer.

This is the same distinction this whole chapter has been making, stated once more because it is easy to forget the moment a founder starts competing against better-funded content instead of just reading it. Ranking well and being accurate are different things, checked by different means. A competitor's higher ranking is not proof their content is more trustworthy, the same way this section's five sources proved that a widely repeated figure is not proof it was quoted honestly. It usually means they have produced more of the structural signal, which is a real, addressable gap, and a separate problem from whether what a small platform has to say is actually worth saying.

Verifying the property at Bing Webmaster Tools

<meta name="msvalidate.01" content="REDACTED_TOKEN" />

That line goes in index.html, not in a component. On a prerendered site the distinction matters: index.html is the template every route is built from, so the tag is baked into the raw HTML of all of them at build time. A verification tag rendered by JavaScript would be invisible to the service checking for it, for exactly the reason Part 1 describes.

The token is not a secret. It ships in public HTML by design and proves ownership by where it appears rather than by being hidden. It also does nothing for ranking, indexing or citation. It is a key to a dashboard.

One instruction from the panel is worth obeying: the tag must stay after verification succeeds. It is a standing claim that gets re-checked, not a one-time handshake.

Confirm the tag is actually live before clicking Verify, because a premature attempt can be rate-limited:

curl -sL https://www.foliovistabooks.com/ | grep msvalidate

If that returns nothing, the deploy has not finished. Wait rather than retrying the verification.

The mistake worth documenting

[FOLIOVISTA-SPECIFIC]

The XML file method was tried first, by downloading the file with curl rather than from the browser:

curl -o public/BingSiteAuth.xml \
  "https://www.bing.com/webmaster/BingSiteAuth.xml?url=https%3a%2f%2fwww.foliovistabooks.com%2f"

curl reported success. It transferred 165 bytes and exited zero. The file was moved into the project and staged for commit before anyone opened it. Its actual contents:

{"type":"...","title":"Unauthorized","status":401,"traceId":"..."}

curl carries no session cookie from a browser login, so Bing refused the request and returned an error document. curl saved that error as the file, exactly as instructed, and reported success, because from its point of view the transfer succeeded. Committing it would have published a JSON error at a public URL, and verification would have failed with no obvious cause.

The general lesson is the same one from the case study where a 200 status concealed an empty page: a status code and an exit code describe the transport, not the content. A downloaded file has to be opened and read before it is trusted, and the check costs one command.

A canonical audit script failure at Google Search Console

[FOLIOVISTA-SPECIFIC]

This audit ran two days after the pages it checked went live, and checks canonical tags, which matter for how Google Search Console decides which URL of a duplicate set gets indexed.

An audit of the static pages on this site asked a simple question: does every page that is meant to be indexed declare a canonical URL? The command looked reasonable:

curl -sL "https://www.foliovistabooks.com/manuals/$page.html" \
  | grep -oE '<link rel="canonical"[^>]*>'

Run against one of the two pages that came back wrong, against the live site, not a guess at what it would do:

curl -sL "https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms.html" \
  | grep -oE '<link rel="canonical"[^>]*>'
# (no output, exit code 1)

Empty. Nothing matched, and the exit code confirms it, 1, not 0. The command did not fail to run. It ran cleanly and found nothing, and a clean exit with an empty result looks identical to a page that genuinely has no canonical tag, which is exactly what makes this kind of false negative dangerous.

It reported four pages with no canonical tag. Two of those four were wrong. The tag was present on both, and had been all along.

The cause is that grep reads one line at a time and cannot match across a line break. The pages that reported as broken write the tag like this:

<link
  rel="canonical"
  href="https://www.foliovistabooks.com/manuals/..."
/>

The pattern required <link rel="canonical" as consecutive characters on a single line. Those characters exist, but a newline sits between them, so the match never happened. Joining the document into one line first fixes it:

curl -sL "$url" \
  | tr '\n' ' ' \
  | grep -oE '<link[^>]*rel="canonical"[^>]*>'

Same page, same live URL, real output this time:

url="https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms.html"
curl -sL "$url" | tr '\n' ' ' | grep -oE '<link[^>]*rel="canonical"[^>]*>'
# <link       rel="canonical"       href="https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms.html"     />

tr translates one character into another, and tr '\n' ' ' replaces every newline with a space. The document becomes a single line, and the tag matches regardless of how it was formatted.

Two pages genuinely did lack a canonical, so the audit was not wasted. But for two others it produced a false alarm, and the false alarm was indistinguishable from the true finding. Both printed nothing. Silence from a check means one of two things, and the check does not tell you which: the thing is absent, or the question was asked wrongly.

This is the same shape as an empty curl response on a redirect, an empty grep result from a stale route, and a 200 status on an empty page. In each case the tool did exactly what it was told and the answer looked like information.

The habit that catches it costs one command. Before trusting a negative result, run the same pattern against something you know contains the thing. If it also returns nothing, the pattern is wrong, not the site.

Making the check impossible to forget

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

The canonical tags were missing for an ordinary reason. These static pages have no shared template, so every tag is typed by hand into each file, and a page is only correct if whoever created it remembered everything. Nothing failed. Something was not typed, and nothing existed to notice.

That is worth fixing at the level of the mechanism rather than the symptom, because the next page will be created the same way. What follows is the real file, scripts/check-manual-metadata.mjs, quoted in full across this section and the next rather than summarised. A guard is only worth reading if you can see everything it does and, more importantly, everything it does not.

The check runs in three phases: static manual pages, generated React routes, then the sitemap comparison that covers both.

Phase 1 -- static manual pages

It opens with a header comment, and that is not decoration. Six months from now the useful question about any guard is not what it checks but why anyone thought it was needed, because that is the part that decides whether deleting it is safe:

// Build-time guard for the hand-maintained static manual pages, and for
// whether every route -- manual or React -- is remembered in the sitemap.
//
// Why this exists: public/manuals/*.html files carry hand-typed metadata.
// Unlike the React routes, which generate their tags from one Seo component,
// each manual page is only correct if whoever created it remembered every
// tag. Nothing enforced that, and on 2026-07-27 three published pages were
// found with no canonical at all, including the largest chapter on the site.
//
// A missing canonical produces no error and no warning. Google simply
// chooses one itself, and if it chooses a different URL the page is treated
// as a duplicate and never indexed independently. The failure is silent in
// exactly the way this project keeps running into.
//
// This script makes it loud, at the one moment someone is paying attention.
//
// The Seo component only protects a React route's own tags once it renders.
// It does nothing for whether that route was ever added to
// public/sitemap.xml -- that file is hand-maintained for React routes
// exactly the same way it is for manual pages. This script reads the real
// <Route> declarations out of src/App.jsx and applies the same two-
// directional sitemap check used for manuals to every one of them too.

import { readdir, readFile } from "node:fs/promises";
import { join } from "node:path";

const MANUALS_DIR = "public/manuals";
const SITEMAP_PATH = "public/sitemap.xml";
const APP_JSX_PATH = "src/App.jsx";
const SITE_ORIGIN = "https://www.foliovistabooks.com";

// Only the <head> matters. A manuscript that happens to discuss canonical
// tags in its body text must not count as declaring one, which is a real
// case here: practical-seo-optimization.html mentions them repeatedly.
function extractHead(html) {
  const end = html.indexOf("</head>");
  return end === -1 ? html : html.slice(0, end);
}

// Tags in these files are sometimes written across several lines, so collapse
// whitespace before matching. Searching line by line is how an earlier check
// reported a present tag as missing -- the manual curl | grep audit hit the
// same bug and uses the same fix: flatten to one line before matching.
function flatten(text) {
  return text.replace(/\s+/g, " ");
}

function findCanonical(head) {
  const match = flatten(head).match(
    /<link[^>]*rel="canonical"[^>]*href="([^"]+)"[^>]*>/i
  );
  return match ? match[1] : null;
}

function findRobots(head) {
  const match = flatten(head).match(
    /<meta[^>]*name="robots"[^>]*content="([^"]+)"[^>]*>/i
  );
  return match ? match[1] : null;
}

Two collections separate the things that stop a build from the things that are merely worth saying out loud. Reading the directory is itself a check, because a guard pointed at nothing passes silently:

const problems = [];
const notes = [];

let files;
try {
  files = (await readdir(MANUALS_DIR)).filter((name) => name.endsWith(".html"));
} catch (error) {
  console.error(`Cannot read ${MANUALS_DIR}: ${error.message}`);
  process.exit(1);
}

if (files.length === 0) {
  console.error(`No HTML files found in ${MANUALS_DIR}. Expected at least one.`);
  process.exit(1);
}

node scripts/check-manual-metadata.mjs
# No HTML files found in public/manuals. Expected at least one.
# (exit code 1)

"build": "node scripts/check-manual-metadata.mjs && vite build && vite build --ssr src/entry-server.jsx --outDir .prerender && node scripts/prerender.mjs"

Then the loop, which is the check itself:

for (const file of files.sort()) {
  const html = await readFile(join(MANUALS_DIR, file), "utf8");
  const head = extractHead(html);
  const robots = findRobots(head);
  const canonical = findCanonical(head);

  // Every manual page must state its indexing intent explicitly. Google
  // treats a missing robots tag as "index, follow", so silence is a decision
  // made by default rather than on purpose. Left unchecked, an accidentally
  // indexable page can surface in Search Console as a duplicate against
  // whichever similar page Google prefers instead.
  if (!robots) {
    problems.push(`${file}: no <meta name="robots"> in <head>`);
    continue;
  }

  const isNoindex = /noindex/i.test(robots);

  // A noindex page is deliberately excluded from search, so a canonical adds
  // nothing. Skip it rather than demanding a pointless tag.
  if (isNoindex) {
    notes.push(`${file}: noindex, canonical not required`);
    continue;
  }

  if (!canonical) {
    problems.push(
      `${file}: indexable but no <link rel="canonical">. ` +
        `Add: <link rel="canonical" href="${SITE_ORIGIN}/manuals/${file}" />`
    );
    continue;
  }

  if (!canonical.startsWith(SITE_ORIGIN)) {
    problems.push(
      `${file}: canonical points off-site or is relative -> ${canonical}`
    );
    continue;
  }

And the ending, which is what turns a report into a guard:

if (problems.length > 0) {
  console.error("\nBuild stopped. Manual page metadata problems:\n");
  for (const problem of problems) console.error(`  ✗ ${problem}`);
  console.error(
    "\nThese pages are served as static HTML with no shared template, so " +
      "nothing else will catch this.\n"
  );
  process.exit(1);
}

The process.exit(1) is the entire difference between a linter and a gate. A script that prints a warning and exits zero is advice, and advice is ignored on the day it matters most.

A linter and a gate can run the exact same checks, print the exact same messages, and still be two different things, because what decides which one a script is happens in its very last line, not in any of the logic before it. process.exit(0) tells the shell the command succeeded, regardless of what it printed on the way there. The && chain wiring this script into npm run build only reads that one number. It does not read the terminal output. A clear warning followed by exit code zero looks, to that chain, identical to no warning at all, and the build continues either way.

process.exit(1) is the only line in this whole script an automated pipeline can actually act on. Every readable message, every specific filename, every explanation of what went wrong exists for the person reading the terminal afterward. That one line exists for the pipeline itself. Delete just that line, and leave every check and every message exactly as they are, and the script turns back into a linter without a single human-visible change to the code. Nothing about how it reads would tell you it stopped being a gate.

Three details in that are worth more than the rest of it.

It reads only the <head>, not the whole file. One of the pages being checked is a manuscript that discusses canonical tags in its body text eleven times. A check scanning the whole document would count those as declarations and pass a page that declares nothing.

It flattens whitespace before matching, for the reason above. A guard built on the broken pattern would enforce nothing while appearing to work.

It requires a robots tag on every page, and that is not for Google's benefit, since a missing robots tag already means "index, follow". It is so the script can tell the two cases apart. The rule being enforced is "an indexable page must declare a canonical", and applying it requires knowing whether a page is meant to be indexed at all. The robots tag is the only machine-readable statement of that intent. A default that is never written down cannot be verified, and an unverifiable default changes without anyone noticing.

The script runs first in the build chain rather than last. This is the whole scripts change in package.json, with nothing trimmed:

"build": "node scripts/check-manual-metadata.mjs && vite build && vite build --ssr src/entry-server.jsx --outDir .prerender && node scripts/prerender.mjs",
"check:manuals": "node scripts/check-manual-metadata.mjs",

The second line matters as much as the first. It gives the check a name that can be run on its own, without a full build, which is what makes it usable while a new page is still being written:

npm run check:manuals

Placed after the bundler, a failing check still lets the build complete, and the error scrolls past in output nobody reads. Placed first, the build stops in two seconds with one line naming the file and the exact tag to add.

It was then tested by deliberately deleting a canonical from a page that had one, and confirming the build failed with a non-zero exit code. A check that has never failed is not known to work. Most of them are written, observed to pass, and never tested against the case they exist for.

The real output from that test, canonical removed from a copy of practical-layered-security-for-small-platforms.html, run against the actual script:

node scripts/check-manual-metadata.mjs

Manual page metadata notes:
  - chapter_04_code_appendix_publication.html: noindex, canonical not required
  - practical-layered-security-for-small-platforms-chapter-2.html: canonical points elsewhere -> https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms.html

Build stopped. Manual page metadata problems:

  ✗ practical-layered-security-for-small-platforms.html: indexable but no <link rel="canonical">. Add: <link rel="canonical" href="https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms.html" />

These pages are served as static HTML with no shared template, so nothing else will catch this.

(exit code 1)

The notes above the failure are not noise. They print on every run, pass or fail, which is what makes them worth reading here: the two deliberate exceptions from earlier in this section, the noindex code appendix and Chapter 2's canonical pointing elsewhere, both show up correctly as notes, not problems, confirming the script still understood them even while it was failing on something else entirely.

The other check fails just as plainly. Same setup, this time with the <meta name="robots"> tag removed instead of the canonical:

Manual page metadata notes:
  - chapter_04_code_appendix_publication.html: noindex, canonical not required
  - practical-layered-security-for-small-platforms-chapter-2.html: canonical points elsewhere -> https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms.html

Build stopped. Manual page metadata problems:

  ✗ practical-seo-optimization.html: no <meta name="robots"> in <head>

These pages are served as static HTML with no shared template, so nothing else will catch this.

(exit code 1)

One line, naming the exact file and the exact missing tag, in both cases. That is the entire promise this script makes, and both runs above are it actually keeping that promise, not a description of what it would do.

Phase 2 -- generated React routes

Static manual pages are files on disk. React routes are not; they are <Route> declarations inside App.jsx's <Routes> block, rendered through one shared Seo component rather than hand-typed per page. Reading them requires a different source than readdir:

// Reads the real <Route> declarations out of App.jsx's <Routes> block,
// rather than trusting a separate hand-kept list of them, for the same
// reason this check reads real files instead of a manifest: a list that
// has to be updated in two places is a list that will eventually disagree
// with itself. Scoped to the <Routes> block specifically so an unrelated
// dynamic `path={...}` prop elsewhere in the file, and there are several,
// can never be mistaken for a route declaration.
function findIndexableRoutes(jsx) {
  const routesBlock = jsx.match(/<Routes>([\s\S]*?)<\/Routes>/);
  if (!routesBlock) return [];
  const flat = flatten(routesBlock[1]);
  const paths = [];
  for (const match of flat.matchAll(
    /<Route\s+path="([^"]+)"\s+element={<(\w+)/g
  )) {
    const [, path, component] = match;
    // Navigate is a redirect, not a real page, so it does not belong in
    // the sitemap -- the same reason a manual page pointing its canonical
    // elsewhere is kept out of the sitemap too. "*" is the catch-all 404
    // page and was never meant to be indexed either.
    if (component === "Navigate" || path === "*") continue;
    paths.push(path);
  }
  return paths;
}

Navigate elements are filtered out, since a redirect is not a real destination: /books/practical-aeo-geo-smo-optimization is a <Navigate> alias to the correctly spelled URL, one real case this excludes correctly. The catch-all * route is filtered out the same way. What survives filtering is the list Phase 3 checks against the sitemap.

Click image to enlarge

Phase 3 -- sitemap checks for both

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

Metadata alone is not enough. A page can carry a correct robots tag and a correct canonical and still never reach a crawler if nobody added it to the sitemap.

That gap stopped being theoretical almost immediately. The largest chapter on the site had published two days earlier with a correct robots tag, a correct canonical, and no sitemap entry at all. Nothing was broken. Something was not typed into a hand-maintained XML file, and no check existed to notice. It sat that way for two days on the single biggest page in the catalog.

The check reads the sitemap once, before any per-page loop runs, then verifies every page and route against it in three directions. Two ways of reading it count as failure on their own, an unreadable file and a readable-but-empty one, since either one left unflagged would let a check further down mistake "nothing to compare against" for "nothing wrong."

Click image to enlarge

The complete logic for all three, canonical, robots, and the sitemap, in one place:

// The sitemap is hand-maintained, the same as the metadata above.
// On 2026-07-25 the SEO chapter sat for two days during development with
// correct metadata and no sitemap entry, caught before anything went live.
// Read it once, up front, so every page can be checked against it below.
let sitemapUrls = new Set();
let sitemapReadable = true;
try {
  const xml = await readFile(SITEMAP_PATH, "utf8");
  // Whitespace inside a <loc> element is legal and would otherwise break an
  // exact string comparison, so trim each one.
  for (const match of xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)) {
    sitemapUrls.add(match[1]);
  }
} catch (error) {
  sitemapReadable = false;
  problems.push(`${SITEMAP_PATH}: cannot be read -> ${error.message}`);
}

if (sitemapReadable && sitemapUrls.size === 0) {
  problems.push(`${SITEMAP_PATH}: no <loc> entries found. Expected several.`);
}

let indexableRoutePaths;
try {
  const jsx = await readFile(APP_JSX_PATH, "utf8");
  indexableRoutePaths = findIndexableRoutes(jsx);
} catch (error) {
  console.error(`Cannot read ${APP_JSX_PATH}: ${error.message}`);
  process.exit(1);
}

if (indexableRoutePaths.length === 0) {
  console.error(`No indexable <Route> entries found in ${APP_JSX_PATH}. Expected several.`);
  process.exit(1);
}

const indexableRouteUrls = new Set(
  indexableRoutePaths.map((path) =>
    path === "/" ? `${SITE_ORIGIN}/` : `${SITE_ORIGIN}${path}`
  )
);

// Collected so the reminder at the end can name what was checked.
const indexablePages = [];

for (const file of files.sort()) {
  const html = await readFile(join(MANUALS_DIR, file), "utf8");
  const head = extractHead(html);
  const robots = findRobots(head);
  const canonical = findCanonical(head);

  if (!robots) {
    problems.push(`${file}: no <meta name="robots"> in <head>`);
    continue;
  }

  const isNoindex = /noindex/i.test(robots);

  // A noindex page is deliberately excluded from search, so a canonical adds
  // nothing. Skip it rather than demanding a pointless tag.
  if (isNoindex) {
    notes.push(`${file}: noindex, canonical not required`);
    continue;
  }

  if (!canonical) {
    problems.push(
      `${file}: indexable but no <link rel="canonical">. ` +
        `Add: <link rel="canonical" href="${SITE_ORIGIN}/manuals/${file}" />`
    );
    continue;
  }

  if (!canonical.startsWith(SITE_ORIGIN)) {
    problems.push(
      `${file}: canonical points off-site or is relative -> ${canonical}`
    );
    continue;
  }


  // A canonical pointing somewhere other than the page itself is legitimate,
  // but only ever on purpose. Chapter 2 does this deliberately, pointing at
  // the main sample page it is embedded in. Report it so an accidental one
  // cannot hide among the intentional ones.
  const expected = `${SITE_ORIGIN}/manuals/${file}`;
  if (canonical !== expected) {
    notes.push(`${file}: canonical points elsewhere -> ${canonical}`);
    // A page whose canonical is another URL should NOT be in the sitemap.
    // A sitemap lists canonical URLs only. Chapter 2 is the real case.
    if (sitemapReadable && sitemapUrls.has(expected)) {
      problems.push(
        `${file}: canonical points at ${canonical} but the page is listed in ` +
          `${SITEMAP_PATH}. A sitemap should list canonical URLs only. ` +
          `Remove this entry.`
      );
    }
    continue;
  }

  // Self-canonical and indexable, so it must be in the sitemap.
  indexablePages.push(expected);
  if (sitemapReadable && !sitemapUrls.has(expected)) {
    problems.push(
      `${file}: indexable and self-canonical but MISSING from ${SITEMAP_PATH}. ` +
        `Add:\n      <url>\n        <loc>${expected}</loc>\n      </url>`
    );
  }
}

// The reverse direction. A sitemap entry pointing at a page that no longer
// exists, or was renamed, sends crawlers to a 404 and wastes the crawl
// budget a small site has very little of. A manuals URL is verified
// against the files on disk; everything else is verified against the
// real <Route> paths read out of App.jsx above.
if (sitemapReadable) {
  for (const url of sitemapUrls) {
    if (url.startsWith(`${SITE_ORIGIN}/manuals/`)) {
      const name = url.slice(`${SITE_ORIGIN}/manuals/`.length);
      if (!files.includes(name)) {
        problems.push(
          `${SITEMAP_PATH}: lists ${url} but public/manuals/${name} does ` +
            `not exist. Remove the entry or restore the file.`
        );
      }
      continue;
    }

    // Not a manual page, so it must match one of the real React routes.
    if (!indexableRouteUrls.has(url)) {
      problems.push(
        `${SITEMAP_PATH}: lists ${url} but no matching <Route> was found ` +
          `in ${APP_JSX_PATH}. Remove the entry or restore the route.`
      );
    }
  }
}

// The forward direction for React routes, the same shape as the manuals
// forward loop above: every real, indexable route declared in App.jsx must
// have a matching sitemap entry.
for (const path of indexableRoutePaths) {
  const expected = path === "/" ? `${SITE_ORIGIN}/` : `${SITE_ORIGIN}${path}`;
  if (sitemapReadable && !sitemapUrls.has(expected)) {
    problems.push(
      `${APP_JSX_PATH}: route "${path}" has no matching entry in ` +
        `${SITEMAP_PATH}. Add:\n      <url>\n        <loc>${expected}</loc>\n      </url>`
    );
  }
}

The reverse loop's if (url.startsWith(...)) line is easy to misread. The condition is a positive match, not a negation: a URL starting with the manuals origin is checked against files on disk; everything else branches into the React-route check below instead of being skipped.

Tested the same way as the first check: a real sitemap entry was deleted, and the build stopped immediately, naming the exact file and the exact block to paste back in:

✗ practical-seo-optimization.html: indexable and self-canonical but
  MISSING from public/sitemap.xml. Add:
      <url>
        <loc>https://.../manuals/practical-seo-optimization.html</loc>
      </url>

The message prints the exact block to paste. A check that only reports a problem still leaves the reader to work out the fix, and that is where the next mistake gets made.

The same check covers React routes the same way. It reads the real <Route> declarations straight out of App.jsx's <Routes> block, rather than keeping a second, hand-typed list that would only drift from the first, and checks each one against the sitemap in both directions, exactly like manual pages. Navigate elements are filtered out, since a redirect is not a real destination: /books/practical-aeo-geo-smo-optimization is a <Navigate> alias to the correctly spelled URL, one real case this excludes correctly. The catch-all * route is filtered out the same way.

Tested the same way as every other check in this appendix: by deliberately breaking something and watching it fail. A temporary route, /partnership, was added to App.jsx, pointed at the existing Contact component so no new page had to be built just for the test, and deliberately left out of the sitemap:

✗ src/App.jsx: route "/partnership" has no matching entry in public/sitemap.xml. Add:
      <url>
        <loc>https://www.foliovistabooks.com/partnership</loc>
      </url>

Manual pages have no shared template and React routes have no automated
sitemap membership check outside this script, so nothing else will catch
this.

(exit code 1)

The route rendered correctly in a live dev server the whole time the test was running, confirming the only thing actually wrong was the missing sitemap entry, which is exactly the failure this check exists to catch, not a rendering problem masquerading as one. The test route was then removed, and the script run again clean, this time reporting on both categories it now covers:

Manual page metadata OK (8 files, 6 indexable, all present in sitemap.xml).
React route sitemap coverage OK (7 routes in src/App.jsx, all present in sitemap.xml).

(exit code 0)

This check verifies sitemap membership for React routes. A route's own canonical and robots tags are generated by the shared Seo component at render time, checked separately from this build step.

The date field that has to be honest and to added for the Sitemap xml file for indexing

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

The same sitemap listed thirteen URLs with no dates on any of them. Google's documentation describes lastmod as a signal it uses when deciding what to recrawl, and it was the last markup-level signal the site was not sending.

Adding it is one line per entry. Getting it right is the part that matters. Here is the entire file, comment included, because the comment is doing as much work as the markup:

<?xml version="1.0" encoding="UTF-8"?>
<!--
  Hand-maintained. There is no generator.

  When adding a page:
    1. add its <url> block here
    2. set <lastmod> to the date the CONTENT changed, not the date you
       edited this file. A lastmod that moves without the page changing
       is a signal Google learns to ignore.
    3. run: npm run check:manuals   (fails the build if a page is missing)

  Only canonical URLs belong here. Chapter 2 is deliberately absent
  because its canonical points at the main layered-security sample page.
-->
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://www.foliovistabooks.com/</loc>
    <lastmod>2026-07-26</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/books</loc>
    <lastmod>2026-07-26</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/books/operational-bug-bounty-fieldwork</loc>
    <lastmod>2026-07-26</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/books/practical-layered-security-for-small-platforms</loc>
    <lastmod>2026-07-26</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/books/practical-seo-aeo-geo-smo-optimization</loc>
    <lastmod>2026-07-26</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/contact</loc>
    <lastmod>2026-07-26</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/privacy-and-terms</loc>
    <lastmod>2026-07-26</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/manuals/operational-bug-bounty-fieldwork.html</loc>
    <lastmod>2026-07-03</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms.html</loc>
    <lastmod>2026-07-03</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms-chapter-3.html</loc>
    <lastmod>2026-07-07</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/manuals/chapter_04_publication.html</loc>
    <lastmod>2026-07-27</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/manuals/chapter_05_publication.html</loc>
    <lastmod>2026-07-27</lastmod>
  </url>
  <url>
    <loc>https://www.foliovistabooks.com/manuals/practical-seo-optimization.html</loc>
    <lastmod>2026-07-27</lastmod>
  </url>
</urlset>

Notice that the dates are not all the same. Seven routes carry one date, two sample pages carry a date more than three weeks earlier, and three pages carry the day the canonical fix landed. That spread is the evidence the field is real.

The dates were taken from version control, one query per file, rather than set to the day the field was added:

git log -1 --format=%cs -- public/manuals/chapter_04_publication.html

Stamping every page with today's date would have been faster and would have looked identical in the file. It would also have been a lie, and specifically the kind of lie the receiving system is built to detect. Google's own guidance says it ignores lastmod when the value proves unreliable. A date that moves on every deploy while the content sits unchanged is exactly that. The field is worth having only while it is true, and a signal that has been learned to be worthless cannot be repaired by sending it more often.

The step no script can perform

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

Both checks run at build time, which means they finish before anything reaches the internet. Everything they verify is a claim about files on disk.

The last step is not. After the deploy succeeds, someone still has to confirm the change is live, and then request indexing in Search Console. No build script can reach either of those.

So the check prints them, on every successful run. This is the tail of the same file, and it is nothing more sophisticated than a series of console.log calls:

if (notes.length > 0) {
  console.log("Manual page metadata notes:");
  for (const note of notes) console.log(`  - ${note}`);
}

console.log(
  `Manual page metadata OK (${files.length} files, ` +
    `${indexablePages.length} indexable, all present in sitemap.xml).`
);

// The third step cannot be automated from here, because it happens in
// Search Console after a deploy. Print it every time so it is never the
// forgotten one. Nothing below affects the exit code.
console.log("");
console.log("  ─────────────────────────────────────────────────────────");
console.log("  AFTER DEPLOYING, do the step no script can do for you:");
console.log("");
console.log("  1. Confirm the change is actually live:");
console.log("       curl -sL https://www.foliovistabooks.com/sitemap.xml \\");
console.log("         | grep -c '<new-page>.html'");
console.log("");
console.log("  2. Request indexing in Google Search Console:");
console.log("       URL Inspection -> paste the URL -> Request Indexing");
console.log("       Once per URL. Resubmitting does not move it up the");
console.log("       queue and the daily quota is small.");
console.log("");
console.log("  A sitemap entry is an invitation, not a request. Google");
console.log("  reads it on its own schedule, which is slow for a site");
console.log("  with few backlinks.");
console.log("  ─────────────────────────────────────────────────────────");

Which produces this, on a run where everything passed:

Manual page metadata notes:
  - chapter_04_code_appendix_publication.html: noindex, canonical not required
  - practical-layered-security-for-small-platforms-chapter-2.html: canonical points elsewhere -> https://www.foliovistabooks.com/manuals/practical-layered-security-for-small-platforms.html
Manual page metadata OK (8 files, 6 indexable, all present in sitemap.xml).

  ─────────────────────────────────────────────────────────
  AFTER DEPLOYING, do the step no script can do for you:

  1. Confirm the change is actually live:
       curl -sL https://www.foliovistabooks.com/sitemap.xml \
         | grep -c '<new-page>.html'

  2. Request indexing in Google Search Console:
       URL Inspection -> paste the URL -> Request Indexing
       Once per URL. Resubmitting does not move it up the
       queue and the daily quota is small.

  A sitemap entry is an invitation, not a request. Google
  reads it on its own schedule, which is slow for a site
  with few backlinks.
  ─────────────────────────────────────────────────────────

The two notes at the top are the deliberate exceptions being stated out loud on every run, rather than sitting silently in someone's memory. The code appendix is noindex on purpose. Chapter 2 points its canonical at the sample page it is embedded in, also on purpose. Printing both means that if either one ever becomes accidental, the line that used to make sense is still on screen, now reading as nonsense.

That is also why eight files produce six indexable pages. The two numbers differing is the check confirming it understood the exceptions, rather than quietly counting them as passes.

It prints on success rather than only on failure, and that choice is the whole point. A reminder that appears when something breaks is absent exactly when it is needed, because the forgotten step is always the one that follows a clean run. Nothing went wrong, the output looked good, and the work was treated as finished while the last action was never taken.

The text is written to the terminal after the pass line and does not affect the exit code. It changes nothing about whether the build succeeds. It exists to occupy the moment when a person is most likely to stop paying attention.

What the report actually shows

[FOLIOVISTA-SPECIFIC]

Once verified, the sitemap already submitted to Google was submitted to Bing Webmaster Tools unchanged, and reported twelve URLs discovered, no errors, no warnings.

The report that has no equivalent elsewhere is labelled "AI Performance (Beta)". It reports citation visibility rather than traffic: where the site was surfaced in AI-generated answers, broken down by intent, by topic, and by citation share.

On the day it was enabled, for this site, it read:

That is the honest result, and it is worth printing rather than hiding. A small, recently published catalog is not being cited by AI systems yet, and no amount of structural work changes that within days. What the zero provides is a dated baseline taken deliberately before a publication event, which is the only way a later number means anything. A baseline captured after the thing you want to measure is not a baseline.

The audit findings, and why one was fixed and one was not

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

Bing Webmaster Tools also offers an on-demand technical audit. Run against nineteen pages it returned no errors, no notices, and thirteen warnings, all of one type: title too long, meaning over seventy characters.

Checked directly rather than accepted:

curl -sL https://www.foliovistabooks.com/books \
  | sed -n 's/.*<title[^>]*>\([^<]*\)<\/title>.*/\1/p'

The finding is accurate. The homepage title measured sixty-nine characters and was correctly not flagged; the flagged pages measured eighty-six to ninety. The rule is real and consistently applied.

It was still not implemented, for three reasons.

Title length is not a ranking factor. The warning describes truncation in a results listing, not position within it.

The existing title pattern already handles truncation. Every title reads "most specific | book | brand", and truncation removes from the end, so what disappears is the brand, then the book, while the specific part always survives. Shortening the titles to satisfy the rule would delete information that currently survives anyway.

The two engines disagree about the number. Google truncates by pixel width, roughly 580 pixels, which lands near sixty characters for typical text. Rewriting to Bing's seventy would still be cut by Google. When two measurement systems give different thresholds for the same page, the threshold is not the thing that matters.

One honest counterpoint belongs here. If the brand is truncated away from every listing, it never accumulates recognition in search results, and brand recognition is one of the signals that feeds citation. That is a judgment call rather than a defect, and for a catalog nobody has heard of yet, leading with the specific content is still the better trade.

Three of the thirteen flagged pages were search-parameter URLs, which should never be indexed separately. Those were checked as well: each returns 200 and carries a canonical pointing at the parameterless page, which is correct. A crawler visiting them is normal, because crawling is not indexing.

This is the second time the same discipline produced the same result on a different tool. A previous chapter documented two commercial audit tools flagging findings that were real and not worth acting on. This one is free, has no upgrade prompt, and no commercial motive at all, and the outcome was identical. The pattern is not that audit tools are trying to sell something. It is that a threshold is easy to measure and an outcome is not, so tools report the threshold.

Not every finding from this same tool ended the same way, though, and the difference is worth showing rather than just asserting. On 2026-08-04, inspecting /books directly flagged a second, different warning: meta description too long. Checked the same way as the title:

# Before the fix:
curl -sL https://www.foliovistabooks.com/books \
  | grep -oE '<meta[^>]*name="description"[^>]*>'
# <meta data-rh="true" name="description" content="Browse the FolioVista
# Books catalog for operational bug bounty books, practical layered
# security guides, HTML sample chapters, PDF previews, and future
# digital release updates."/>

The description measured 176 characters, well past the roughly 155 to 160 characters search engines typically display before cutting a snippet off.

This one was implemented, and the title finding above explains why by contrast rather than by coincidence. The title survives truncation on purpose: "most specific | book | brand" loses the brand first, then the book label, while the specific content stays intact no matter where the cut lands. The description had no equivalent structure. It was one flowing sentence, and truncating it anywhere past 160 characters would cut it mid-clause, most likely mid-word, which reads as broken rather than merely shorter in an actual results snippet. That is a different failure shape from the title's, and it is the one worth fixing rather than explaining away.

The fix dropped the least essential clause, "and future digital release updates," which was also the exact clause a hard truncation would have cut off anyway, and kept the two real book categories intact:

"Browse the FolioVista Books catalog for operational bug bounty and practical layered security guides, with free HTML and PDF sample chapters."

141 characters, comfortably inside the truncation point, with nothing lost that a search snippet would have shown regardless. The same command, run against the live site after the fix:

curl -sL https://www.foliovistabooks.com/books \
  | grep -oE '<meta[^>]*name="description"[^>]*>'
# <meta data-rh="true" name="description" content="Browse the FolioVista
# Books catalog for operational bug bounty and practical layered
# security guides, with free HTML and PDF sample chapters."/>

What Bing Webmaster Tools is and is not worth

[UNIVERSAL PRINCIPLE, FOLIOVISTA EXAMPLE]

Most of Bing Webmaster Tools duplicates a Google Search Console property: sitemaps, index status, crawl errors, query performance, URL inspection, backlinks. Maintaining a second account for those is not worth the effort.

Three things have no equivalent:

• The AI citation report
• The on-demand site audit
• Organic keyword volume data

The first is the only free citation measurement available anywhere at the time of writing, and it alone justifies the setup.

It also covers exactly one engine. Nothing here reports on ChatGPT, Claude, or Perplexity. The manual logging described in Part 3 still covers everything else, and this changes the scale of that work rather than removing it.

One manual check costs nothing and needs no dedicated tool: ask an AI system directly, by brand name. On 2026-08-03, asking Gemini what FolioVista Books is returned an accurate description sourced directly from foliovistabooks.com, the real author, the real book titles, and an accurate summary of what the SEO/AEO/GEO/SMO chapter actually covers, not a hallucinated guess.

That does not mean the platform is being discovered. A separate, non-branded query in the same session, asking for the best books on VPS Linux hardening, returned no result for the platform at all, and the same was true for a general SEO-books query. A brand-name query only ever confirms that whoever already knows the name will find something accurate when they look. It says nothing about whether anyone finds the name in the first place, which is the harder problem this whole chapter has been about.

What it does verify, narrowly but for free, is that indexing and structured data actually took effect somewhere beyond Bing: the description an AI system gives when asked directly is only accurate if the underlying page was crawled, parsed and understood correctly. Any reader can run the same check on their own site, for free, any time, by asking the same question about their own name.

The complete file

The three phases above walk through why each part exists, one at a time. What follows is the same file, unfragmented, for anyone who wants to copy it whole rather than reassemble it from the pieces:

// Build-time guard for the hand-maintained static manual pages, and for
// whether every route -- manual or React -- is remembered in the sitemap.
//
// Why this exists: public/manuals/*.html files carry hand-typed metadata.
// Unlike the React routes, which generate their tags from one Seo component,
// each manual page is only correct if whoever created it remembered every
// tag. Nothing enforced that, and on 2026-07-27 three published pages were
// found with no canonical at all, including the largest chapter on the site.
//
// A missing canonical produces no error and no warning. Google simply
// chooses one itself, and if it chooses a different URL the page is treated
// as a duplicate and never indexed independently. The failure is silent in
// exactly the way this project keeps running into.
//
// This script makes it loud, at the one moment someone is paying attention.
//
// The Seo component only protects a React route's own tags once it renders.
// It does nothing for whether that route was ever added to
// public/sitemap.xml -- that file is hand-maintained for React routes
// exactly the same way it is for manual pages. This script reads the real
// <Route> declarations out of src/App.jsx and applies the same two-
// directional sitemap check used for manuals to every one of them too.

import { readdir, readFile } from "node:fs/promises";
import { join } from "node:path";

const MANUALS_DIR = "public/manuals";
const SITEMAP_PATH = "public/sitemap.xml";
const APP_JSX_PATH = "src/App.jsx";
const SITE_ORIGIN = "https://www.foliovistabooks.com";

// Only the <head> matters. A manuscript that happens to discuss canonical
// tags in its body text must not count as declaring one, which is a real
// case here: practical-seo-optimization.html mentions them repeatedly.
function extractHead(html) {
  const end = html.indexOf("</head>");
  return end === -1 ? html : html.slice(0, end);
}

// Tags in these files are sometimes written across several lines, so collapse
// whitespace before matching. Searching line by line is how an earlier check
// reported a present tag as missing -- the manual curl | grep audit hit the
// same bug and uses the same fix: flatten to one line before matching.
function flatten(text) {
  return text.replace(/\s+/g, " ");
}

function findCanonical(head) {
  const match = flatten(head).match(
    /<link[^>]*rel="canonical"[^>]*href="([^"]+)"[^>]*>/i
  );
  return match ? match[1] : null;
}

function findRobots(head) {
  const match = flatten(head).match(
    /<meta[^>]*name="robots"[^>]*content="([^"]+)"[^>]*>/i
  );
  return match ? match[1] : null;
}

// Reads the real <Route> declarations out of App.jsx's <Routes> block,
// rather than trusting a separate hand-kept list of them, for the same
// reason the manuals check reads real files instead of a manifest: a list
// that has to be updated in two places is a list that will eventually
// disagree with itself. Scoped to the <Routes> block specifically so an
// unrelated dynamic `path={...}` prop elsewhere in the file (there are
// several) can never be mistaken for a route declaration.
function findIndexableRoutes(jsx) {
  const routesBlock = jsx.match(/<Routes>([\s\S]*?)<\/Routes>/);
  if (!routesBlock) return [];
  const flat = flatten(routesBlock[1]);
  const paths = [];
  for (const match of flat.matchAll(
    /<Route\s+path="([^"]+)"\s+element={<(\w+)/g
  )) {
    const [, path, component] = match;
    // Navigate is a redirect, not a real page, so it does not belong in
    // the sitemap -- the same reason a manual page pointing its canonical
    // elsewhere is kept out of the sitemap too. "*" is the catch-all 404
    // page and was never meant to be indexed either.
    if (component === "Navigate" || path === "*") continue;
    paths.push(path);
  }
  return paths;
}

const problems = [];
const notes = [];

let files;
try {
  files = (await readdir(MANUALS_DIR)).filter((name) => name.endsWith(".html"));
} catch (error) {
  console.error(`Cannot read ${MANUALS_DIR}: ${error.message}`);
  process.exit(1);
}

if (files.length === 0) {
  console.error(`No HTML files found in ${MANUALS_DIR}. Expected at least one.`);
  process.exit(1);
}

// The sitemap is hand-maintained, the same as the metadata above.
// On 2026-07-25 the SEO chapter sat for two days during development with
// correct metadata and no sitemap entry, caught before anything went live.
// Read it once, up front, so every page can be checked against it below.
let sitemapUrls = new Set();
let sitemapReadable = true;
try {
  const xml = await readFile(SITEMAP_PATH, "utf8");
  // Whitespace inside a <loc> element is legal and would otherwise break an
  // exact string comparison, so trim each one.
  for (const match of xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/g)) {
    sitemapUrls.add(match[1]);
  }
} catch (error) {
  sitemapReadable = false;
  problems.push(`${SITEMAP_PATH}: cannot be read -> ${error.message}`);
}

if (sitemapReadable && sitemapUrls.size === 0) {
  problems.push(`${SITEMAP_PATH}: no <loc> entries found. Expected several.`);
}

let indexableRoutePaths;
try {
  const jsx = await readFile(APP_JSX_PATH, "utf8");
  indexableRoutePaths = findIndexableRoutes(jsx);
} catch (error) {
  console.error(`Cannot read ${APP_JSX_PATH}: ${error.message}`);
  process.exit(1);
}

if (indexableRoutePaths.length === 0) {
  console.error(`No indexable <Route> entries found in ${APP_JSX_PATH}. Expected several.`);
  process.exit(1);
}

const indexableRouteUrls = new Set(
  indexableRoutePaths.map((path) =>
    path === "/" ? `${SITE_ORIGIN}/` : `${SITE_ORIGIN}${path}`
  )
);

// Collected so the reminder at the end can name what was checked.
const indexablePages = [];

for (const file of files.sort()) {
  const html = await readFile(join(MANUALS_DIR, file), "utf8");
  const head = extractHead(html);
  const robots = findRobots(head);
  const canonical = findCanonical(head);

  // Every manual page must state its indexing intent explicitly. Google
  // treats a missing robots tag as "index, follow", so silence is a decision
  // made by default rather than on purpose. Left unchecked, an accidentally
  // indexable page can surface in Search Console as a duplicate against
  // whichever similar page Google prefers instead.
  if (!robots) {
    problems.push(`${file}: no <meta name="robots"> in <head>`);
    continue;
  }

  const isNoindex = /noindex/i.test(robots);

  // A noindex page is deliberately excluded from search, so a canonical adds
  // nothing. Skip it rather than demanding a pointless tag.
  if (isNoindex) {
    notes.push(`${file}: noindex, canonical not required`);
    continue;
  }

  if (!canonical) {
    problems.push(
      `${file}: indexable but no <link rel="canonical">. ` +
        `Add: <link rel="canonical" href="${SITE_ORIGIN}/manuals/${file}" />`
    );
    continue;
  }

  if (!canonical.startsWith(SITE_ORIGIN)) {
    problems.push(
      `${file}: canonical points off-site or is relative -> ${canonical}`
    );
    continue;
  }

  // A canonical pointing somewhere other than the page itself is legitimate,
  // but only ever on purpose. Chapter 2 does this deliberately, pointing at
  // the main sample page it is embedded in. Report it so an accidental one
  // cannot hide among the intentional ones.
  const expected = `${SITE_ORIGIN}/manuals/${file}`;
  if (canonical !== expected) {
    notes.push(`${file}: canonical points elsewhere -> ${canonical}`);
    // A page whose canonical is another URL should NOT be in the sitemap.
    // A sitemap lists canonical URLs only. Chapter 2 is the real case.
    if (sitemapReadable && sitemapUrls.has(expected)) {
      problems.push(
        `${file}: canonical points at ${canonical} but the page is listed in ` +
          `${SITEMAP_PATH}. A sitemap should list canonical URLs only. ` +
          `Remove this entry.`
      );
    }
    continue;
  }

  // Self-canonical and indexable, so it must be in the sitemap.
  indexablePages.push(expected);
  if (sitemapReadable && !sitemapUrls.has(expected)) {
    problems.push(
      `${file}: indexable and self-canonical but MISSING from ${SITEMAP_PATH}. ` +
        `Add:\n      <url>\n        <loc>${expected}</loc>\n      </url>`
    );
  }
}

// The reverse direction. A sitemap entry pointing at a page that no longer
// exists, or was renamed, sends crawlers to a 404 and wastes the crawl
// budget a small site has very little of. A manuals URL is verified
// against the files on disk; everything else is verified against the
// real <Route> paths read out of App.jsx above.
if (sitemapReadable) {
  for (const url of sitemapUrls) {
    if (url.startsWith(`${SITE_ORIGIN}/manuals/`)) {
      const name = url.slice(`${SITE_ORIGIN}/manuals/`.length);
      if (!files.includes(name)) {
        problems.push(
          `${SITEMAP_PATH}: lists ${url} but public/manuals/${name} does ` +
            `not exist. Remove the entry or restore the file.`
        );
      }
      continue;
    }

    // Not a manual page, so it must match one of the real React routes.
    if (!indexableRouteUrls.has(url)) {
      problems.push(
        `${SITEMAP_PATH}: lists ${url} but no matching <Route> was found ` +
          `in ${APP_JSX_PATH}. Remove the entry or restore the route.`
      );
    }
  }
}

// The forward direction for React routes, the same shape as the manuals
// forward loop above: every real, indexable route declared in App.jsx must
// have a matching sitemap entry.
for (const path of indexableRoutePaths) {
  const expected = path === "/" ? `${SITE_ORIGIN}/` : `${SITE_ORIGIN}${path}`;
  if (sitemapReadable && !sitemapUrls.has(expected)) {
    problems.push(
      `${APP_JSX_PATH}: route "${path}" has no matching entry in ` +
        `${SITEMAP_PATH}. Add:\n      <url>\n        <loc>${expected}</loc>\n      </url>`
    );
  }
}

if (notes.length > 0) {
  console.log("Manual page metadata notes:");
  for (const note of notes) console.log(`  - ${note}`);
}

if (problems.length > 0) {
  console.error("\nBuild stopped. Manual page metadata problems:\n");
  for (const problem of problems) console.error(`  ✗ ${problem}`);
  console.error(
    "\nManual pages have no shared template and React routes have no " +
      "automated sitemap membership check outside this script, so " +
      "nothing else will catch this.\n"
  );
  process.exit(1);
}

console.log(
  `Manual page metadata OK (${files.length} files, ` +
    `${indexablePages.length} indexable, all present in sitemap.xml).`
);
console.log(
  `React route sitemap coverage OK (${indexableRoutePaths.length} routes ` +
    `in ${APP_JSX_PATH}, all present in sitemap.xml).`
);

// The third step cannot be automated from here, because it happens in
// Search Console after a deploy. Print it every time so it is never the
// forgotten one. Nothing below affects the exit code.
console.log("");
console.log("  ─────────────────────────────────────────────────────────");
console.log("  AFTER DEPLOYING, do the step no script can do for you:");
console.log("");
console.log("  1. Confirm the change is actually live:");
console.log("       curl -sL https://www.foliovistabooks.com/sitemap.xml \\");
console.log("         | grep -c '<new-page>.html'");
console.log("");
console.log("  2. Request indexing in Google Search Console:");
console.log("       URL Inspection -> paste the URL -> Request Indexing");
console.log("       Once per URL. Resubmitting does not move it up the");
console.log("       queue and the daily quota is small.");
console.log("");
console.log("  A sitemap entry is an invitation, not a request. Google");
console.log("  reads it on its own schedule, which is slow for a site");
console.log("  with few backlinks.");
console.log("  ─────────────────────────────────────────────────────────");

• OpenAI -- web crawlers overview (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, each a separate user-agent requiring its own robots.txt entry): developers.openai.com/api/docs/bots
• Anthropic -- does Anthropic crawl data from the web, and how can site owners block the crawler (ClaudeBot, Claude-User, Claude-SearchBot): support.claude.com
• Perplexity -- Perplexity crawlers resource guide (PerplexityBot, Perplexity-User, robots.txt behavior): docs.perplexity.ai/docs/resources/perplexity-crawlers
• Google Search Central -- Consolidate duplicate URLs (canonical URLs, including the explicit guidance against fragments as canonical -- relevant here since a #fragment and a real page URL are not the same thing to a crawler either way): developers.google.com/search/docs/crawling-indexing/consolidate-duplicate-urls
• Google Search Central -- Robots meta tag, data-nosnippet, and X-Robots-Tag (noindex, nosnippet, max-snippet, and related directives): developers.google.com/search/docs/crawling-indexing/robots-meta-tag
• schema.org -- FAQPage: schema.org/FAQPage
• schema.org -- Book: schema.org/Book
• MDN Web Docs -- URL fragment / hash (what a fragment identifier is and is not sent to the server): developer.mozilla.org/en-US/docs/Web/URI/Fragment

Sources checked in Part 5

Every figure discussed in Part 5 is listed here with the source that originally published it, so the same check can be repeated directly rather than taken on trust. Read the primary source, not a summary of it, including this one.

The document under examination:

• Jasper -- "How to Optimize Content for GEO and AEO in an AI-Native World" (23-page ebook; five of its thirteen embedded links point to the publisher's own product pages): jasper.ai

The primary sources it cited:

• Pew Research Center -- Google users are less likely to click on links when an AI summary appears in the results (published 22 July 2025; source of the 8% and 15% click figures, the ~18% of searches that trigger a summary, the ~1% click rate on links inside a summary, the 67-word median summary length, and the multi-source citation counts): pewresearch.org
• Ahrefs -- AI Overviews reduce clicks (published 17 April 2025; 300,000 keywords, March 2024 against March 2025; the original source of the widely quoted 34.5% figure): ahrefs.com/blog/ai-overviews-reduce-clicks
• Ahrefs -- AI Overviews reduce clicks, updated study (re-run on December 2025 data, December 2023 used as the pre-rollout baseline; revises the headline figure to 58%, adds a position-by-position breakdown from -58% at position one to -19.4% at position ten, and cites independent corroboration from Seer Interactive, Kevin Indig and Authoritas in a roughly 47% to 65% band. Read this one before quoting the earlier figure): ahrefs.com/blog/ai-overviews-reduce-clicks-update
• Similarweb -- AI referral traffic winners (published 29 July 2025; source of the 357% year-over-year figure and of the 1.13 billion against 191 billion comparison in the same article): similarweb.com
• Adobe -- ChatGPT as a search engine (survey fielded 29 May 2025, 1,000 US respondents pre-screened as ChatGPT users; source of the 77% have-used figure and the 24% go-to-it-first figure): adobe.com
• Semrush -- AI search and SEO traffic study (published 21 July 2025; source of the 4.4x visitor value claim, and of the finding that LLMs frequently cite pages ranked below position 20): semrush.com/blog/ai-search-seo-traffic-study
• Semrush -- Google usage after ChatGPT adoption (published 11 August 2025; 260 billion rows of opted-in clickstream data, January 2024 to June 2025, US and desktop only, with a never-adopted control group; the strongest methodology of the five and the finding that ChatGPT adoption does not reduce Google usage): semrush.com/blog/google-usage-after-chatgpt-adoption
• Google Search Central -- Creating helpful, reliable, people-first content (the primary documentation behind E-E-A-T, cited by the ebook and worth reading directly): developers.google.com/search/docs/fundamentals/creating-helpful-content

The second vendor guide examined:

• Lumar -- The 4-Pillar GEO Strategy Framework (overview, 10 April 2026), and its four pillar pages: Technical GEO (11 May 2026), Content GEO (14 May 2026), Entity GEO (20 April 2026), Brand Authority GEO (13 May 2026). Cites no statistics across any of the five pages:
  4-pillar-geo-strategy-framework-for-ai-search-visibility
  technical-geo-aeo-guide-for-ai-search-optimization
  how-to-optimize-your-content-for-ai-search-visibility-geo-aeo
  entity-building-for-ai-brand-visibility-geo-aeo-explainer
  brand-authority-geo-aeo-ai-search-tips