Learning Path

The web ran for two decades on an implicit trade: let crawlers take your content, receive visitors in return. AI systems kept the first half and largely dropped the second, and the ratios are not close. The 2026 response has been a set of competing attempts to reprice access — pay-per-answer, publisher marketplaces, revenue pools, and licensing standards — none of which has settled. In the meantime, the only decision you actually control is what you want from agent traffic: attribution, conversion, payment, or nothing. Access policy should follow from that answer rather than from a default.

What the numbers show

Cloudflare publishes crawl-to-refer ratios — how many times a company’s crawlers fetch pages for each visitor it sends back. The spread is the whole argument:

Anthropic: roughly 1,800 crawls per referral.
OpenAI: roughly 850–1,300 crawls per referral.
Perplexity: roughly 111–186 crawls per referral.
Google: roughly 5 crawls per referral.

Source: Cloudflare, AI crawler traffic by purpose and industry. The same analysis reports that bots now account for 57.5% of HTML requests observed by Cloudflare Radar in June 2026 — automated traffic is the majority of the document web, not an overlay on it.

Google’s ratio is the useful control. It is not that search is generous; it is that search was built on a bargain where the crawl existed to produce the click. When the answer is synthesized in the interface, the click is no longer the product, and the ratio detaches. Note also that Google’s figure blends conventional search crawling with AI use, so it understates how much the answer surface has changed the trade even there.

Two cautions before you build strategy on these numbers. First, ratios vary enormously by industry and by site — a reference site and a retailer are in different businesses with the same crawlers. Second, a referral is not the only form of value: appearing in an answer that a customer acts on later has real worth that no crawl-to-refer ratio captures. The ratios prove the old bargain stopped clearing. They do not by themselves prove the new arrangement is worthless.

The attempts to reprice it

Four serious efforts ran through 2026, each with a different theory of what should be paid for and who should collect.

Cloudflare: Pay Per Crawl becomes Pay Per Use

On July 1, 2026, Cloudflare replaced Pay Per Crawl with Pay Per Use. The change in unit is the substance: instead of charging for the fetch, the model compensates publishers when their content appears in an answer. That aligns payment with the moment value is actually extracted, and it removes the perverse incentive of a per-fetch price — which rewards a crawler for reading less and reusing more.

Cloudflare: Content Independence Day and AI options

Microsoft: Publisher Content Marketplace

Live since February 2026, with Copilot as its first buyer. The theory here is procurement: a platform-run marketplace where publishers list content and AI products license it through a single commercial channel rather than negotiating bilaterally. It is the most conventional of the four models — and its viability depends entirely on whether buyers beyond the operator’s own assistant show up.

Perplexity: Comet Plus publisher pool

A revenue pool funded from a consumer subscription and distributed to participating publishers. This is the media-bundle theory applied to answers: users pay for the assistant, and a share flows to the sources behind what it says. Pools are simple to join and hard to reason about — the payout to any one publisher depends on the size of the pool and the distribution rule, neither of which the publisher controls.

RSL: a licensing standard without signatures

Really Simple Licensing takes the standards route: express machine-readable license terms alongside your content, so that access carries stated conditions rather than an implied one. Roughly 1,500 publishers had backed it, but as of August 2026 no AI company had signed a payment deal under it. That gap is the honest state of the field — broad publisher-side agreement on what the terms should be, and no counterparty yet obliged to accept them.

None of these is settled, and they are not obviously compatible with one another. A per-answer clearing model, a licensing marketplace, a subscription pool, and a terms standard imply different infrastructure and different winners. Plan on ambiguity persisting; avoid architecture that only works if one of them prevails.

Decide what you actually want

Most access policy is inherited rather than chosen: a WAF default, a robots.txt copied from a blog post, a rate limit set for a slower era. The alternative is to name the return you want and design access to produce it. There are four honest answers, and different parts of one service can give different ones.

Attribution

You want to be named in the answer, because being the cited source is itself worth something. Then make content maximally readable and quotable: server-rendered, clearly sourced, unambiguous about who is speaking. Accept that many reads will never become visits, and measure presence in answers rather than sessions.

Conversion

You want the agent to complete a transaction or bring its user to you to complete one. Then the work is on the acting path, not the reading path: current inventory and pricing available to be read, and an unambiguous route from “the agent decided” to “the order exists.” Blocking here is self-harm.

Payment

Your content is the product and use of it should be licensed. Then you need enforceable gating — not just a policy file — plus a route through which someone can actually pay you. Given the state of the models above, this is the answer with the most operational work and the least certain revenue.

Nothing

Some content has no upside in being ingested and real cost in being served. Refusing is a legitimate choice — but make it deliberately and narrowly, and keep it separate from the question of whether your customers’ own agents can use your product.

Designing access from the answer

Separate crawl policy from act policy. Bulk ingestion for training and a customer’s assistant checking their own account are different relationships with different economics. One robots.txt-shaped decision cannot express both.
Differentiate by purpose, not just by name. The same operator may run a training crawler, a search index crawler, and a user-triggered fetcher. Your answer can reasonably differ across the three, and the major operators increasingly publish distinct identities for them.
Price the expensive parts, open the cheap ones. Static reference content costs you almost nothing to serve and may earn attribution. Deep queries against live data cost real money. Uniform policy over both is uniformly wrong.
Instrument before you legislate. You cannot choose a return you cannot measure. Being able to segment agent traffic by operator and purpose in your own logs is the prerequisite for every decision on this page.

The one durable conclusion: the traffic-for-content trade is not coming back in its old form, and no replacement has stabilized. Services that treat this as a policy question they own — rather than an infrastructure default they inherited — will be better placed whichever model wins. The principles page covers the observability that makes any of it measurable.

Principles Back to Applications

Stay Updated

Analysis of AI search, crawler policy and agent standards — sent when there is something worth reading, roughly twice a month. Unsubscribe anytime.