Cloudflare's September 15 Deadline: What Changes for Your Site
Update, November 15, 2026: Cloudflare changed its plans for September 15 before they took effect, and Pay Per Use launched in beta on September 30. This post has been revised to match. Our first version also said Pay Per Use replaced Pay Per Crawl. It didn't.
The short version
Since September 15, 2026, Cloudflare has offered new domains preset AI settings. For sites with ads, the preset allows search crawlers (the bots that visit pages to build a search index), asks AI companies not to train on the content, and blocks AI agents on pages with ads. On September 30 Cloudflare launched Pay Per Use in beta alongside Pay Per Crawl. It pays publishers when a buyer reports using their content.
If your site uses Cloudflare (a service that sits between a website and its visitors to speed it up and block attacks), the settings that decide which AI bots may read your content changed on September 15. This post covers what changed, who it affects, and the one setting that can take you out of search results if you pick it without thinking.
What Cloudflare changed
On July 1, 2026, Cloudflare announced new AI traffic options for all its customers. More details followed on 15 and September 30.
Pay Per Use joined Pay Per Crawl. Pay Per Crawl, launched in 2025, charges AI companies each time they fetch a page, using an HTTP 402 response (a standard web reply that means "payment required") to ask for the money. Cloudflare added Pay Per Use, which launched in beta on September 30, 2026. In Cloudflare's own words, "Pay Per Crawl, which we launched in 2025, charges for access. Pay Per Use pays for what happens next. Publishers can choose the model that suits them." The AI company buying the content decides what counts as a use and sets a price. Pay Per Crawl still exists. When Pay Per Use was first announced in July, TechCrunch reported that Cloudflare was starting with two partners, Ceramic.ai and You.com.
Access control now has three categories. Instead of one allow-or-block
choice for each crawler (a bot that visits pages automatically), sites can set
separate permissions for Search, Agent, and Training use, on every
plan including the free one. This sits alongside the Content-Signal line
Cloudflare added to robots.txt (the file that tells bots which pages they may
visit) in late 2025, which lets a site say in that file which kinds of use it
allows.
New ad-supported domains are offered a stricter setup. From September 15, 2026, Cloudflare offers new ad-supported domains a recommended setup: Search allowed, Disallow AI Training, and agents blocked on pages with ads. The owner can change it. Sites without ads are offered a setup that allows all three. Cloudflare's documentation, updated on November 2, 2026, describes the default for new domains as allowing Search, Agents, and Training, so check which setup your own domain ended up with.
Block and Disallow AI Training do different things
Several of the biggest crawlers do more than one job. Googlebot, Applebot, and Bingbot crawl for Google's, Apple's, and Microsoft's search, and their owners also use web content for AI training. For Training, Cloudflare now gives you two choices that sound alike and aren't.
Since September 15, the Block setting for Training also stops Googlebot, Applebot, and Bingbot from visiting, so you can drop out of their search results. "Block on pages with ads" does the same on any page that shows ads. Disallow AI Training keeps those crawlers indexing your pages while asking them not to train on your content. Cloudflare's September 15 post says the older Block AI setting, and earlier choices of Block for Training, were moved over to Disallow AI Training, so sites that set those before September 15 stay in search.
There's one gap. The same post says Bing doesn't yet support a "no training" request in robots.txt, so Disallow AI Training can't pass that wish on to Bing until Microsoft adds it, which it's aiming to do in early 2027.
Why the economics shifted
Search engines crawled your content and sent visitors back. AI companies crawl far more and send far fewer people.
Cloudflare Radar measured a crawl-to-refer ratio (how many times a company crawls your site for each visitor it sends back). In the first week of August 2025, Anthropic's ratio was nearly 50,000:1, OpenAI's was 887:1, and Perplexity's was 118:1.
Estimates for 2026 from other sources, which we haven't been able to match to a Cloudflare publication, put Anthropic much lower, at roughly 1,800:1, OpenAI between 850:1 and 1,300:1, Perplexity between 111:1 and 186:1, and Google Search at about 5:1. We don't know whether Anthropic's drop is real or a different way of counting. Even the lowest AI figure is more than 20 times Google's.
Bots now make more of the web's traffic than people do, too. In June 2026, Cloudflare's CEO Matthew Prince said that bots made 57.5% of web requests, against 42.5% from people. AI crawlers make up roughly a fifth of the bot traffic that can be traced to a known company.
What you should actually do
If you're on Cloudflare, open Security > Settings > AI Crawler and Agent access policies and set all three categories yourself. Don't assume the preset suits you, in either direction. An educational site that wants as much reach as possible and a subscription publisher protecting paid archives should end up with very different settings.
Decide Search and Training separately, because they're different
questions. Allowing OAI-SearchBot (which powers citations in ChatGPT
search) while blocking GPTBot (which gathers training data) is a sensible
position. So is the reverse. Mixing the two up is how sites make themselves
invisible by accident. A robots.txt that blocks OAI-SearchBot while the site
tries to show up in ChatGPT can't work, because that's the bot ChatGPT search
uses.
Remember that robots.txt is only a request. Some bots fetch a page
because a person asked for something just now. They include ChatGPT-User,
Claude-User, Perplexity-User, and Google's Google-Agent. AI companies
generally treat these visits as the user acting rather than as crawling, and
they may ignore crawler rules. If you need rules that are enforced and not
just politely asked for, they belong at the edge (the network layer, such as
Cloudflare, that sits in front of your site): firewall rules, rate limits
(caps on how many requests a visitor can make), and bot verification.
Watch Web Bot Auth. Web Bot Auth lets a bot prove who it is with a digital signature, instead of a name anyone could copy. The standards group writing it hasn't finished yet. We think checking signed agents like this is where access control is heading.
The bigger picture
Cloudflare isn't alone. RSL (Really Simple Licensing), a standard for telling AI companies your licensing terms, is another attempt to put a price on content that used to be free. RSL's own site says more than 1,500 publishers, brands, technology companies, and standards organizations support it. Nobody knows yet whether any of these will work.
What to check now
- Open Security > Settings > AI Crawler and Agent access policies in Cloudflare, and set Search, Agent, and Training yourself.
- For Training, pick Disallow AI Training rather than Block unless you're happy to leave Google's, Apple's, and Microsoft's search.
- Make your Search decision and your Training decision separately in robots.txt.
OAI-SearchBotandGPTBotare different choices. - Put firm enforcement at the edge, and treat robots.txt as a statement of what you'd prefer.
References
- Cloudflare, "Your site, your rules: new AI traffic options for all customers", July 1, 2026
- Cloudflare, "Have it both ways: stay discoverable in search while disallowing AI training", September 15, 2026
- Cloudflare, "Pay Per Use: when AI uses your work, you should get paid", September 30, 2026
- Cloudflare documentation, "Block AI bots", updated November 2, 2026
- Cloudflare Radar, "A deeper look at AI crawlers: breaking down traffic by purpose and industry", August 28, 2025
- TechCrunch, "Cloudflare's new policy pushes AI companies to pay for publishers' content", July 1, 2026
- Content Signals Policy, contentsignals.org
Choose which AI bots you let in
Our guide to AI crawler controls lists the bots in use today, what each one does, and how to decide which to allow.