Agent Experiences

Cloudflare's September 15 Deadline: What Changes for Your Site

By Alan WeibelPublished Updated 7 min read

Update, November 15, 2026: Cloudflare changed its plans for September 15 before they took effect, and Pay Per Use launched in beta on September 30. This post has been revised to match. Our first version also said Pay Per Use replaced Pay Per Crawl. It didn't.

The short version

Since September 15, 2026, Cloudflare has offered new domains preset AI settings. For sites with ads, the preset allows search crawlers (the bots that visit pages to build a search index), asks AI companies not to train on the content, and blocks AI agents on pages with ads. On September 30 Cloudflare launched Pay Per Use in beta alongside Pay Per Crawl. It pays publishers when a buyer reports using their content.

Source: Cloudflare, September 15 and 30, 2026

If your site uses Cloudflare (a service that sits between a website and its visitors to speed it up and block attacks), the settings that decide which AI bots may read your content changed on September 15. This post covers what changed, who it affects, and the one setting that can take you out of search results if you pick it without thinking.

What Cloudflare changed

On July 1, 2026, Cloudflare announced new AI traffic options for all its customers. More details followed on 15 and September 30.

Pay Per Use joined Pay Per Crawl. Pay Per Crawl, launched in 2025, charges AI companies each time they fetch a page, using an HTTP 402 response (a standard web reply that means "payment required") to ask for the money. Cloudflare added Pay Per Use, which launched in beta on September 30, 2026. In Cloudflare's own words, "Pay Per Crawl, which we launched in 2025, charges for access. Pay Per Use pays for what happens next. Publishers can choose the model that suits them." The AI company buying the content decides what counts as a use and sets a price. Pay Per Crawl still exists. When Pay Per Use was first announced in July, TechCrunch reported that Cloudflare was starting with two partners, Ceramic.ai and You.com.

Access control now has three categories. Instead of one allow-or-block choice for each crawler (a bot that visits pages automatically), sites can set separate permissions for Search, Agent, and Training use, on every plan including the free one. This sits alongside the Content-Signal line Cloudflare added to robots.txt (the file that tells bots which pages they may visit) in late 2025, which lets a site say in that file which kinds of use it allows.

New ad-supported domains are offered a stricter setup. From September 15, 2026, Cloudflare offers new ad-supported domains a recommended setup: Search allowed, Disallow AI Training, and agents blocked on pages with ads. The owner can change it. Sites without ads are offered a setup that allows all three. Cloudflare's documentation, updated on November 2, 2026, describes the default for new domains as allowing Search, Agents, and Training, so check which setup your own domain ended up with.

Block and Disallow AI Training do different things

Several of the biggest crawlers do more than one job. Googlebot, Applebot, and Bingbot crawl for Google's, Apple's, and Microsoft's search, and their owners also use web content for AI training. For Training, Cloudflare now gives you two choices that sound alike and aren't.

Since September 15, the Block setting for Training also stops Googlebot, Applebot, and Bingbot from visiting, so you can drop out of their search results. "Block on pages with ads" does the same on any page that shows ads. Disallow AI Training keeps those crawlers indexing your pages while asking them not to train on your content. Cloudflare's September 15 post says the older Block AI setting, and earlier choices of Block for Training, were moved over to Disallow AI Training, so sites that set those before September 15 stay in search.

There's one gap. The same post says Bing doesn't yet support a "no training" request in robots.txt, so Disallow AI Training can't pass that wish on to Bing until Microsoft adds it, which it's aiming to do in early 2027.

Why the economics shifted

Search engines crawled your content and sent visitors back. AI companies crawl far more and send far fewer people.

Cloudflare Radar measured a crawl-to-refer ratio (how many times a company crawls your site for each visitor it sends back). In the first week of August 2025, Anthropic's ratio was nearly 50,000:1, OpenAI's was 887:1, and Perplexity's was 118:1.

Estimates for 2026 from other sources, which we haven't been able to match to a Cloudflare publication, put Anthropic much lower, at roughly 1,800:1, OpenAI between 850:1 and 1,300:1, Perplexity between 111:1 and 186:1, and Google Search at about 5:1. We don't know whether Anthropic's drop is real or a different way of counting. Even the lowest AI figure is more than 20 times Google's.

Bots now make more of the web's traffic than people do, too. In June 2026, Cloudflare's CEO Matthew Prince said that bots made 57.5% of web requests, against 42.5% from people. AI crawlers make up roughly a fifth of the bot traffic that can be traced to a known company.

What you should actually do

If you're on Cloudflare, open Security > Settings > AI Crawler and Agent access policies and set all three categories yourself. Don't assume the preset suits you, in either direction. An educational site that wants as much reach as possible and a subscription publisher protecting paid archives should end up with very different settings.

Decide Search and Training separately, because they're different questions. Allowing OAI-SearchBot (which powers citations in ChatGPT search) while blocking GPTBot (which gathers training data) is a sensible position. So is the reverse. Mixing the two up is how sites make themselves invisible by accident. A robots.txt that blocks OAI-SearchBot while the site tries to show up in ChatGPT can't work, because that's the bot ChatGPT search uses.

Remember that robots.txt is only a request. Some bots fetch a page because a person asked for something just now. They include ChatGPT-User, Claude-User, Perplexity-User, and Google's Google-Agent. AI companies generally treat these visits as the user acting rather than as crawling, and they may ignore crawler rules. If you need rules that are enforced and not just politely asked for, they belong at the edge (the network layer, such as Cloudflare, that sits in front of your site): firewall rules, rate limits (caps on how many requests a visitor can make), and bot verification.

Watch Web Bot Auth. Web Bot Auth lets a bot prove who it is with a digital signature, instead of a name anyone could copy. The standards group writing it hasn't finished yet. We think checking signed agents like this is where access control is heading.

The bigger picture

Cloudflare isn't alone. RSL (Really Simple Licensing), a standard for telling AI companies your licensing terms, is another attempt to put a price on content that used to be free. RSL's own site says more than 1,500 publishers, brands, technology companies, and standards organizations support it. Nobody knows yet whether any of these will work.

What to check now

  • Open Security > Settings > AI Crawler and Agent access policies in Cloudflare, and set Search, Agent, and Training yourself.
  • For Training, pick Disallow AI Training rather than Block unless you're happy to leave Google's, Apple's, and Microsoft's search.
  • Make your Search decision and your Training decision separately in robots.txt. OAI-SearchBot and GPTBot are different choices.
  • Put firm enforcement at the edge, and treat robots.txt as a statement of what you'd prefer.

References

Choose which AI bots you let in

Our guide to AI crawler controls lists the bots in use today, what each one does, and how to decide which to allow.