Cloudflare split AI crawler control into Search, Training and Agent

On 15 September 2026 Cloudflare published "Have it both ways: stay discoverable in search while disallowing AI training" and emailed its customers the next day. The old Block AI Bots switch "will be deprecated in favor of the more granular Search, Training, and Agent controls", and Managed Robots.txt will be deprecated in favor of Bot Preference Sync, the feature that writes these preferences into a zone's robots.txt.

Agent covers "user-directed agents visiting a page on behalf of a human, such as chat fetch bots and browser-use agents." Each behavior takes one of four settings.

• ·Allow: "All crawlers are allowed, unless blocked by another setting or a WAF rule."
• ·Disallow AI Training, Training only: "Bot Preference Sync publishes the applicable no-training preference in robots.txt. Accountable mixed-use crawlers remain allowed for search."
• ·Block on pages with ads: "Crawlers, including mixed-use crawlers, are blocked only on pages detected to be serving an ad."
• ·Block: "All crawlers, including mixed-use crawlers, are blocked."

The reversal worth noticing

On 1 July 2026 Cloudflare wrote that from 15 September "multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training". Read literally, disallowing training meant disappearing from search.

The 15 September post replaces that plan. "Apple, Google, and Microsoft all demonstrate that they meet the qualifications to be Accountable", so their crawlers stay allowed for search and receive the no-training preference through robots.txt. For Google that means a Disallow rule for Google-Extended, which Google states "does not impact a site's inclusion in Google Search". Every other training crawler is blocked, which Cloudflare says includes the training-only crawlers run by Amazon, Anthropic, Meta and OpenAI.

One gap remains. Microsoft is building support for a robots.txt no-training preference, "targeted for early 2027". Until then, Disallow AI Training does not convey that preference to Bing through robots.txt.

What happens to existing settings

Cloudflare migrates zones automatically from the legacy Block AI Bots value:

• ·Disabled becomes Search Allow, Training Allow, Agent Allow.
• ·Block becomes Search Allow, Training Disallow AI Training, Agent Block on pages with ads.
• ·Block on pages with ads becomes Search Allow, Training Disallow AI Training, Agent Block on pages with ads.

Asked what customers need to do, Cloudflare answers: "Nothing, in almost every case." Its customer email adds that the migration rolls out over the following week, that manual changes made meanwhile are kept, and that it is complete when the Block AI Bots switch disappears from the dashboard.

What it means for AI visibility

• ·Search crawlers are not training crawlers. OpenAI documents that "OAI-SearchBot is used to surface websites in search results in ChatGPT's search features", while GPTBot crawls content that may be used for training. Perplexity says PerplexityBot "is not used to crawl content for AI foundation models." Disallowing training is not the same as leaving AI search.
• ·Google Search is untouched by Disallow AI Training, because Googlebot stays allowed. Search Engine Journal notes that "a separate Search Console setting controls whether a site appears in AI Overviews, AI Mode, and Discover's generative AI features."
• ·robots.txt is a request, not a wall. Cloudflare's own documentation says "robots.txt compliance is voluntary." Enforcement happens only where Cloudflare blocks a crawler at its edge.

The ten second check

None of this reaches a hostname set to DNS only, the grey cloud. For those records Cloudflare says it "cannot optimize, cache, and protect those requests", so its edge rules never see the traffic, whoever hosts the site.

Run curl -sI https://yoursite.com and look for a cf-ray header. If it is there, Cloudflare proxies your traffic: open Security Settings and review Search, Training and Agent on every zone. If it is not, Cloudflare is not in the request path and nothing changes for you.

We ran it on our six sites on 16 September 2026. All six answered server: Netlify with no cf-ray header, and none of their robots.txt files carried a Cloudflare block. Nothing to change. Measure first, conclude second.

Sources

• ·Cloudflare Blog, Have it both ways: stay discoverable in search while disallowing AI training, 15 September 2026: https://blog.cloudflare.com/accountable-mixed-use-ai-crawlers/
• ·Cloudflare Blog, Say it once: introducing Bot Preference Sync, 21 August 2026: https://blog.cloudflare.com/bot-preference-sync/
• ·Cloudflare Blog, Content Independence Day AI options, 1 July 2026: https://blog.cloudflare.com/content-independence-day-ai-options/
• ·Cloudflare Docs, Managed robots.txt: https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
• ·Cloudflare Docs, Proxy status: https://developers.cloudflare.com/dns/proxy-status/
• ·Google Search Central, Google common crawlers: https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
• ·OpenAI, Overview of OpenAI crawlers: https://developers.openai.com/api/docs/bots
• ·Perplexity, Perplexity crawlers: https://docs.perplexity.ai/guides/bots
• ·Search Engine Journal, Cloudflare lets sites disallow AI training without blocking Googlebot, 15 September 2026: https://www.searchenginejournal.com/cloudflare-lets-sites-disallow-ai-training-without-blocking-googlebot/589559/