Learning Path
Navigate the AXO curriculum
Websites
Website Deep Dives
How answer engines actually read your page
What happens between a fetcher requesting your URL and a sentence from it turning up in someone else’s answer.
The short version
An answer engine does not read your page. It fetches the HTML, strips it to something close to plain text, cuts that text into passages, and retrieves individual passages against a query — often a query you never targeted. The unit of citation is the passage, not the page. Almost every practical rule on this site follows from that one fact.
A correction to earlier guidance
This page used to say agents process content “systematically and sequentially”, and that you should size sections to fit a limited context window. Both were wrong, and the advice that came out of them was slightly off as a result.
Retrieval systems do not read top to bottom. They embed passages and fetch the ones that match, which means a section three-quarters of the way down your page competes on equal terms with your opening paragraph. And context windows stopped being the binding constraint some time ago — frontier models now hold well over a million tokens. What limits you is not how much the model can hold. It is which passage the retriever picked, and whether that passage still made sense once it was pulled out of your page.
The pipeline
Implementations differ, and the details are mostly undocumented. The broad shape is consistent enough to design against.
- 1Fetch. A crawler or a user-triggered fetcher requests the URL. Most do not execute JavaScript. If your content only exists after hydration, the pipeline ends here and nothing else on this page matters.
- 2Convert. The HTML is reduced to text, usually something close to markdown. Structure that lives in your CSS does not survive this step. Structure that lives in your HTML elements does.
- 3Chunk. The text is cut into passages. You do not control where the cuts land, and they will not always respect your section boundaries.
- 4Embed and retrieve. Passages are matched against a query. This is where your page competes — passage by passage, not as a whole.
- 5Rerank. Candidate passages are reordered, often by a separate model weighing relevance, recency and source signals.
- 6Synthesize. The answer gets written from the surviving passages. Some are cited; most are not.
Retrieval is not citation. Research published in July 2026 found ChatGPT cites only around 15% of the pages it retrieves, drawing on roughly four sources per answer — so being fetched, and even being used, still leaves most of the distance to being credited (Search Engine Land).
What survives conversion
This is the most useful thing to internalise, because it is entirely within your control and most sites get part of it wrong.
Survives
- Real heading elements, and their nesting
- Lists, tables, and their row and column relationships
- Paragraph boundaries
- Link text
- Emphasis, sometimes
- JSON-LD, parsed separately
Does not
- Anything rendered only by JavaScript
- Text inside images, including screenshotted tables
- Meaning carried by colour or position alone
- A
divstyled to look like a heading - Content behind a click — tabs, accordions, modals
- Visual grouping with no structural equivalent
A useful test costs nothing: disable JavaScript and CSS, then read the page. What is left is roughly what the retriever sees. If the result is an unstructured wall of text, or if half the content has vanished, you have found your problem without needing any tooling.
You are answering questions you never targeted
Google’s AI Mode has been documented decomposing a single prompt into roughly nine to eleven parallel sub-queries, retrieving for each, then writing one answer from the combined results. ChatGPT fans out less aggressively but still does it.
So your page is rarely competing for the question the person typed. It is competing for a decomposed fragment of it, generated on the fly, that neither of you saw. A page ranking eleventh for the headline query might be first for the sub-query that actually got retrieved.
This is the strongest available argument for depth over breadth, and for sub-headings that name real questions instead of gesturing at topics. “How long does verification take?” is a retrieval target. “Verification” is not.
Writing passages that survive extraction
A passage gets lifted away from everything around it. If it needed the three paragraphs above to make sense, it arrives in the answer either wrong or unintelligible — and a reranker that notices the ambiguity will drop it in favour of a source that stands alone.
Survives on its own
Acme Platform encrypts data at rest and in transit using AES-256. Role-based permissions are audited and logged, and administrators can require multi-factor authentication for all accounts.
Falls apart
As mentioned above, we’ve got you covered. This layer builds on the previous one and adds even more protection, making it the strongest option we discussed earlier.
The right-hand version fails for a mundane reason: every noun in it points somewhere else. “This layer”, “the previous one”, “we discussed earlier”. Extract it and it refers to nothing.
The habit worth building is small: name the subject again rather than pronouning back to it, and put the qualifier inside the sentence it qualifies. If a claim is only true for the enterprise tier, or only in the EU, or only since version 4, say so in that sentence and not in the paragraph above it.
What this does not mean
Do not pre-chunk your content. Rewriting a page into artificial fragments to match how you imagine the retriever cuts it is not supported by evidence, and Google has said explicitly that it is not required. Aggressive rewrites in this direction tend to strip out the connective material that made a passage meaningful, which makes extraction worse rather than better.
Write well-structured prose for people. Self-contained sections are a property of good technical writing anyway, which is convenient: the thing that helps retrieval is the thing you should be doing regardless.
Checks worth running
- Load the page with JavaScript disabled. Is the content still there?
- Disable CSS. Does the structure still read as a hierarchy?
- Pick three sections at random, read each without the rest of the page, and check they still say something true and complete.
- Look at your sub-headings. Do they name questions someone would ask, or topics?
- Search the page for “as mentioned above”, “this approach”, “the former”. Each one is a passage that will not survive being lifted.
Related
- What actually gets you cited — the evidence behind each tactic, ranked by confidence
- The CLEAR framework — the writing checklist that comes out of all this
- Technical implementation — rendering, markup and delivery
Last reviewed . Retrieval pipelines are largely undocumented and change without notice; treat the sequence above as the working model it is.
Stay Updated
Analysis of AI search, crawler policy and agent standards — sent when there is something worth reading, roughly twice a month. Unsubscribe anytime.