Cloudflare /crawl now enforces Content Signals: what use=reference and ai-train=no do

What changed on 31 August

Browser Run is Cloudflare's hosted browser service, renamed from Browser Rendering on 15 April 2026. Its /crawl endpoint, launched in beta on 10 March 2026, takes a starting URL and returns rendered pages as HTML, Markdown or structured JSON.

Since 31 August the endpoint reads a site's Content-Signal line before it crawls. The caller declares two things:

If the site's preferences are stricter than the declaration, the request is rejected at the start with a 400 and the message "Crawl disallowed by Content-Signal directive (purpose or use level)". A site with use=reference accepts contentUse reference and rejects the default full. A site with use=immediate rejects every crawl. A site with no Content-Signal line is crawled normally. robots.txt Disallow rules still apply URL by URL, reported with the status "disallowed".

• ·contentUse: "the level at which you intend to use the crawled content. Allowed values, from least to most permissive, are reference and full. The default is full."
• ·crawlPurposes: by default the endpoint "declares all three purposes": search, ai-input and ai-train.

Why default requests now fail on many sites

Cloudflare's managed robots.txt writes this block for the zones that use it: User-Agent: *, then Content-signal: search=yes, ai-train=no, use=reference, then Allow: /.

A default /crawl request declares ai-train and full, so it collides with that block twice, once on purpose and once on use level. To crawl such a site, the caller has to send contentUse reference and narrow crawlPurposes, for example to search only.

What it does not mean

• ·The use directive is still a test. Cloudflare's documentation says it "is testing content-use, an optional extension to Content Signals". It was introduced on 1 July 2026 with three levels: immediate, "interact, but store and reuse nothing"; reference, "index, excerpt, and link back", the default; and full, "summarize and reproduce".
• ·Other crawlers have not signed up. Search Engine Roundtable reported on 6 July 2026 that Google's John Mueller said of the directive: "it has no effects whatsoever for any crawler or llm." We found no public commitment from OpenAI, Anthropic or Perplexity to honor it.
• ·The standards track uses different words. The IETF AI Preferences vocabulary draft, version 08 of 14 September 2026, defines AI Training, AI Use and Search categories and states that its contents "DO NOT REFLECT CONSENSUS of the Working Group".

What it means for AI visibility

Content Signals are a preference that one enforcer honors today. A use=reference or ai-train=no line stops Cloudflare's /crawl customers. On the evidence we found, it does not change what ChatGPT, Gemini or Claude do. If you want AI engines to cite you, what decides that is still which crawlers you let in, at robots.txt and at your edge, and what they find when they arrive.

If your robots.txt carries a Content-Signal line you did not write, it may come from Cloudflare's managed robots.txt. Read it, and keep it only if it is the policy you chose.

We checked our eight sites on 16 September 2026. None carries a Content-Signal line, so a /crawl request reaches them normally. Measure first, conclude second.

Questions people ask

Why does Cloudflare /crawl return a 400 error?

Because the target site's robots.txt Content-Signal line is stricter than what the request declared. The error reads "Crawl disallowed by Content-Signal directive (purpose or use level)": either a purpose you declared is set to no, or your contentUse is above the site's use level.

How do I crawl a site that sets use=reference and ai-train=no?

Send contentUse reference and narrow crawlPurposes so it leaves out ai-train, for example search only. The default request, contentUse full with all three purposes, is rejected.

What does Content-Signal use=reference mean?

In Cloudflare's definition, reference means "index, excerpt, and link back". The stricter level, immediate, means "interact, but store and reuse nothing", and full means "summarize and reproduce". Cloudflare still describes use as a test extension.

Do Google or OpenAI respect Content Signals?

No public commitment that we could find. Google's John Mueller was reported on 6 July 2026 as saying the directive has no effect for any crawler or LLM. Cloudflare's /crawl is the only enforcement we could source.

Should I add a Content-Signal line to my robots.txt?

Only as a deliberate policy. Today it binds Cloudflare /crawl customers and signals intent to everyone else. It does not replace robots.txt rules for the crawlers you actually want to control.

Sources

• ·Cloudflare Changelog, Crawl endpoint now respects the Content Signals use directive, 31 August 2026: https://developers.cloudflare.com/changelog/post/2026-08-31-crawl-content-use/
• ·Cloudflare Docs, Browser Run /crawl endpoint: https://developers.cloudflare.com/browser-run/quick-actions/crawl-endpoint/
• ·Cloudflare Changelog, Browser Rendering renamed to Browser Run, 15 April 2026: https://developers.cloudflare.com/changelog/post/2026-04-15-br-rename/
• ·Cloudflare Changelog, Crawl entire websites with a single API call, 10 March 2026: https://developers.cloudflare.com/changelog/post/2026-03-10-br-crawl-endpoint/
• ·Cloudflare Docs, Managed robots.txt: https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/
• ·Cloudflare Blog, Content Independence Day AI options, 1 July 2026: https://blog.cloudflare.com/content-independence-day-ai-options/
• ·Cloudflare Blog, Content Signals Policy, 24 September 2025: https://blog.cloudflare.com/content-signals-policy/
• ·Search Engine Roundtable, Google: Cloudflare Content Signals Robots.txt Directive Has No Effects, 6 July 2026: https://www.seroundtable.com/google-cloudflare-content-signals-41631.html
• ·IETF, draft-ietf-aipref-vocab-08, 14 September 2026: https://datatracker.ietf.org/doc/draft-ietf-aipref-vocab/