Cloudflare's September 15 Deadline: What Changes for Your Site

If your site sits behind Cloudflare — and a large share of the web does — the defaults governing who may read your content are about to change. This post covers what is changing, who it affects, and the decision you should make deliberately rather than inherit.

What Cloudflare announced

On July 1, 2026, in an announcement it branded "Content Independence Day 2," Cloudflare made three changes that matter for anyone thinking about AI visibility.

Pay Per Crawl became Pay Per Use. The original 2025 model charged AI companies for fetching a page, using HTTP 402 responses to signal that payment was required. The economics never quite worked: crawling is cheap and indiscriminate, and charging per fetch penalized the crawl rather than the benefit. The replacement ties compensation to content appearing in an AI answer. Cloudflare has named Ceramic.ai and You.com as early partners, with broader availability promised later in 2026.

Access control split into three categories. Rather than a single allow-or-deny decision per crawler, sites can now express permissions separately for Search, Agent, and Training use — available on all plans including free. This mirrors the Content-Signal syntax Cloudflare introduced in robots.txt in late 2025, and the newer use parameter with immediate, reference and full values.

New ad-monetized domains get a restrictive default. From September 15, 2026, domains newly onboarded to Cloudflare that carry advertising default to allowing Search while blocking Training and Agent access.

The part that will surprise people

Read that last change carefully, because there is a trap in it.

Several major crawlers are mixed-purpose. Googlebot crawls for search, but Google also uses web content for training under the separate Google-Extended token. Applebot and Bingbot have similar dual roles. Cloudflare's tri-category enforcement means that if you block Training, mixed-purpose crawlers can be caught by that rule too — including crawlers you certainly want to keep.

Why the economics shifted

The underlying problem is that the traditional bargain of the open web broke. Search engines crawled your content and sent you visitors in exchange. That exchange is no longer symmetric.

Cloudflare's own measurements put the crawl-to-refer ratio — how many times a company crawls your site for each visitor it sends back — at roughly 1,800:1 for Anthropic and somewhere between 850:1 and 1,300:1 for OpenAI, against about 5:1 for Google Search. Perplexity sits between them at roughly 111–186:1.

Those numbers are the whole argument. A crawler taking a thousand pages for every visitor returned is not participating in the old bargain; it is substituting for it. Meanwhile bots crossed a threshold in 2026: they now account for 57.5% of HTML requests, with AI crawlers making up roughly a fifth of verified bot traffic.

What you should actually do

If you are on Cloudflare, open AI Crawl Control and set the three categories deliberately. Do not assume the default matches your interests in either direction — an educational site that wants maximum reach and a subscription publisher protecting paid archives should end up with very different settings.

Decide Search and Training separately, because they are different questions. Allowing OAI-SearchBot (which powers citation in ChatGPT search) while blocking GPTBot (which gathers training data) is a coherent position. So is the reverse. Conflating them is what produces accidental invisibility — we have seen robots.txt templates in the wild that block GPTBot while recommending the site "optimize for ChatGPT," which cannot work.

Remember that robots.txt is advisory. User-triggered fetchers — ChatGPT-User, Claude-User, Perplexity-User, and Google's Google-Agent token introduced in March 2026 — fetch pages because a person asked for something in the moment. Providers generally treat these as the user acting rather than as crawling, and they may not honor crawler directives. If you need enforcement rather than a polite request, it belongs at the edge: WAF rules, rate limits, and bot verification.

Watch Web Bot Auth. The durable fix for "is this bot who it claims to be" is cryptographic rather than string-matching. The IETF working group has not published an RFC, but the mechanism — HTTP Message Signatures with a published key directory — is already deployed in production by Cloudflare, Vercel, Shopify, and AWS. Verifying signed agents is where access control is heading.

The bigger picture

Cloudflare is not the only party trying to rebuild content economics. Microsoft launched its Publisher Content Marketplace in February 2026 with Copilot as the first buyer. Perplexity's Comet Plus pool pays participating publishers. The RSL standard has roughly 1,500 media organizations behind it — though, tellingly, no AI company had signed an RSL payment agreement as of August 2026.

What all of these have in common is an attempt to price something that used to be free. Whether any of them works is genuinely unresolved. What is already settled is that "let everything crawl and hope for referrals" stopped being a strategy some time ago.

References

• Cloudflare, "Content Independence Day: what's next", July 1, 2026
• Cloudflare Radar, "AI crawler traffic by purpose and industry", 2026
• TechCrunch, "Cloudflare's new policy pushes AI companies to pay for publishers' content", July 1, 2026
• Content Signals Policy, contentsignals.org