AI Opt-Out Controls

The Cloudflare Regime: AI Crawl Control and Pay Per Use

Cloudflare sits in front of a large share of the web, so its defaults function as de facto policy for many sites. AI Crawl Control has been generally available since August 2025, and customers now send over a billion HTTP 402 (Payment Required) responses a day to AI crawlers. If you are behind Cloudflare, these controls are the most consequential lever you have.

The robots.txt use parameter

Alongside the category controls, Cloudflare introduced a use parameter for robots.txt with three values — immediate, reference and full — escalating from the narrowest permitted use to the broadest. It is new and Cloudflare-originated rather than an established standard, so check the current definitions in Cloudflare's post before you deploy it, and do not treat it as a substitute for the access controls themselves.

Source: Cloudflare, Content Independence Day: new AI options

Choose a Posture First

Do not start from the config. Start from one question: what is your content worth to you, and where does its value come from? If your business runs on reach, blocking costs you more than it protects. If your content is the product, being an unpaid training input is a straight loss. Four coherent postures follow from that; pick the one that matches your economics, then copy the config.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# Training crawlers blocked
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Bytespider
Disallow: /

# Training opt-out tokens (not crawlers)
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /

# Deliberately allowed: OAI-SearchBot, Claude-SearchBot,
# PerplexityBot, DuckAssistBot — these are what put you in
# AI answers with a citation.

User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no
Allow: /

# Machine-readable licensing terms
License: https://yourdomain.com/license.xml

# Block AI ingestion pending a commercial agreement
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Bytespider
Disallow: /

User-agent: *
Content-Signal: search=yes, ai-input=no, ai-train=no
Allow: /

# Every AI token, both training and citation
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-AdsBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: MistralAI-User
Disallow: /
User-agent: DuckAssistBot
Disallow: /

# Conventional search left allowed — remove these two lines
# only if you also want out of Google and Bing.
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /

robots.txt Templates

Place at /robots.txt. These examples target common AI-related crawlers. Adjust to your policy.

Block Specific AI Crawlers

# Block AI training crawlers, keep search/citation crawlers
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: meta-externalagent
Disallow: /

User-agent: Amazonbot
Disallow: /

User-agent: Bytespider
Disallow: /

# Opt-out tokens, not crawlers: these signal "no AI training"
User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

# Left allowed on purpose — these index you for AI answers and
# citations. Disallowing them removes you from ChatGPT search,
# Claude search, and Perplexity results:
#   OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot

# Allow everyone else
User-agent: *
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Block All Except Major Search

# Allow conventional search engines only; block AI crawling entirely
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /

User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: OAI-AdsBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-SearchBot
Disallow: /
User-agent: Claude-User
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: MistralAI-User
Disallow: /
User-agent: DuckAssistBot
Disallow: /

User-agent: *
Disallow: /

Content Signals Policy: Stating Permitted Use, Not Access

The Content Signals Policy is a Cloudflare-originated robots.txt extension introduced in September 2025. It addresses a real gap: Allow and Disallow only express whether a crawler may fetch a page, and say nothing about what it may then do with what it fetched. Content Signals adds that second dimension, so you can let a crawler in while still declaring that training is not a permitted use.

It is live on 3.8 million or more domains, largely because Cloudflare added it to managed robots.txt files by default — so adoption reflects a platform rollout more than a considered choice by each site owner. Three signals are defined:

Syntax

A Content-Signal line goes inside a robots.txt group, alongside the usual Allow/Disallow rules. Each signal takes yes or no; omitting a signal expresses no preference on it, which is different from saying no.

# Content Signals Policy — https://contentsignals.org/
#
# search:    building a search index and providing search results
# ai-input:  inputting content into one or more AI models
# ai-train:  training or fine-tuning AI models
#
# Yes/no here express the site operator's preferences about how
# content may be used after it is accessed.

User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Per-crawler signals

Because it lives in a robots.txt group, you can state different permitted uses for different crawlers — useful if you have an agreement with one vendor and not others.

# Default: citable, not trainable
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

# A vendor you have licensed to: full permitted use
User-agent: ExampleLicensedBot
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

ai.txt (Emerging)

Place at /ai.txt. The ai.txt file communicates site AI usage policies. It is not a universal standard, but some vendors check it.

# ai.txt — Site policy for AI usage
# Scope: training (model development) vs inference (runtime usage)

owner: Your Company Name
contact: legal@yourdomain.com
policy: https://yourdomain.com/policies/ai-usage

# Explicit preferences
allow: inference
disallow: training

# Notes for crawlers
note: Please respect robots.txt and do not store content for model training.
updated: 2026-08-17

Licensing and Preference Standards, Honestly Assessed

Several efforts are trying to turn "please don't" into "here are the terms." They are at very different stages of maturity, and vendor marketing tends to blur that. Here is where each one actually stands as of August 2026.

Server-Side Controls (Optional)

Add mitigations beyond policy files. Examples shown for Nginx; adapt to your stack.

Block by User-Agent

# nginx.conf
map $http_user_agent $block_ai {
  default 0;
  ~*(GPTBot|OAI-SearchBot|ChatGPT-User|OAI-AdsBot|ClaudeBot|Claude-SearchBot|Claude-User|PerplexityBot|Perplexity-User|Google-Agent|meta-externalagent|Amazonbot|Bytespider|CCBot|MistralAI-User|DuckAssistBot) 1;
}

server {
  if ($block_ai) { return 403; }
  # ... rest of config
}

Rate Limit Suspicious Bots

# nginx rate limit example
limit_req_zone $binary_remote_addr zone=botlimit:10m rate=1r/s;

server {
  location / {
    if ($block_ai) { limit_req zone=botlimit burst=5 nodelay; }
  }
}

Warning: User-agents can be spoofed. Combine with IP reputation, behavior analysis, and WAF rules if this is critical.

Verifying Bots: What Works Now, and Where It's Going

Every user-agent-based rule above shares one weakness: the user-agent is just a string the client chose to send. If your policy has commercial consequences — allowing a paying partner, blocking an unpaid one — you need to verify identity rather than accept a claim.

What you can actually use today

# Verify a claimed Googlebot request
host 66.249.66.1
# -> crawl-66-249-66-1.googlebot.com
host crawl-66-249-66-1.googlebot.com
# -> must resolve back to 66.249.66.1

Web Bot Auth: the direction of travel

IP lists and reverse DNS both scale poorly — every site operator maintains their own copy of everyone else's infrastructure. Web Bot Auth replaces that with cryptographic proof: the bot signs its requests, and you verify the signature against a key the operator publishes.

Monitoring & Verification

• Log access to /robots.txt, /ai.txt, and high-value content paths.
• Alert on spikes in known AI user-agents (GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot, Claude-SearchBot, Claude-User, Google-Agent, PerplexityBot, Perplexity-User, meta-externalagent, Amazonbot, Bytespider, CCBot, MistralAI-User, DuckAssistBot). Note that Google-Extended and Applebot-Extended are robots.txt opt-out tokens only—they never appear in your logs.
• Watch for mismatches between user-agent and behavior (e.g., excessive parallel fetches).
• Document policy changes and include lastmod in your sitemap.xml for transparency.

The Legal Backdrop, Briefly

None of this is settled law, and the most important recent ruling cuts against the assumption that you can freely block agents acting for a user. This is context for your decision, not legal advice — talk to counsel before relying on it.