---
title: robots.txt says yes, your server says no: why AI crawlers still get refused
canonical_url: https://tkmstudio.com/library/robots-txt-allows-ai-crawlers-server-blocks
language: en
last_updated: 2026-09-27
organization: TKM Studio, AI Visibility & Adoption Studio, Madrid, working worldwide
---

# robots.txt says yes, your server says no: why AI crawlers still get refused

A robots.txt that allows GPTBot, ClaudeBot and PerplexityBot is a request, not a door. The door is your CDN, your firewall and your bot protection, and they decide on their own. Where a yes turns into a no, why the fetchers that run while a buyer is asking depend on that layer most, and what each test can and cannot prove.

Two layers decide whether an AI crawler reads your page. robots.txt states what you would like crawlers to do. Your edge, meaning the CDN, the web application firewall and any bot protection in front of your server, decides what actually happens to each request. Cloudflare's own documentation says robots.txt compliance is voluntary. The reverse is also true: a robots.txt that says Allow does not stop the edge from saying no.

Most sites check the first layer and assume the second. The result is a robots.txt that welcomes AI search crawlers while the server in front of it refuses them, and nobody in the company knows.

## Four places a yes turns into a no

- An AI bot policy at the CDN. Since 1 July 2026 Cloudflare lets you set Search, Training and Agent separately, and on 15 September 2026 it retired the single Block AI Bots switch. From that date, new domains on every plan, Free included, get a default that blocks Training and Agent on pages that display ads and allows Search. Cloudflare places chat fetch bots such as ChatGPT-User in the Agent category. The block is enforced at the edge, whatever robots.txt says.
- A robots.txt you did not write. Cloudflare's managed robots.txt adds Disallow rules for crawlers including GPTBot, ClaudeBot and Google-Extended in front of your own file. The robots.txt a crawler receives can differ from the one in your repository.
- Generic bot protection. Cloudflare Bot Fight Mode challenges traffic that matches known bot patterns and cannot be skipped with WAF custom rules. Its documentation does not say whether verified AI crawlers are exempt, so do not assume they are. Vercel's bot protection excludes verified bots, but Vercel states it does not work behind a reverse proxy such as Cloudflare.
- Rules somebody added. User agent blockers such as Netlify's User Agent Blocker extension or Vercel's AI bots ruleset, both off by default, custom firewall rules and IP blocks all act before robots.txt is ever read.

## The fetchers that run while a buyer is asking

AI vendors run separate agents for training, for building a search index and for fetching a page at the moment a user asks. Their robots.txt behaviour is not the same.

OpenAI says robots.txt rules may not apply to ChatGPT-User, because a user triggers it. Perplexity says Perplexity-User generally ignores robots.txt. Anthropic says its bots, Claude-User included, honor robots.txt. For two of those three vendors, the edge is the only gate that reliably decides the live fetch, and live fetchers are also the category Cloudflare now blocks by default on pages that display ads, on new domains.

Perplexity's documentation goes furthest: it tells site owners to use its published IP ranges in their WAF configuration and to refresh them automatically. Perplexity's own guidance places the access decision in the firewall.

## The Google-Extended trap

Google-Extended controls whether your content is used to train future Gemini models and for grounding in Gemini Apps and Vertex AI. Google states it does not affect inclusion in Google Search and is not a ranking signal. It has no user agent of its own: Google fetches with its normal crawlers. A firewall rule that matches Google-Extended in the user agent matches nothing.

The opposite mistake is more expensive. AI Overviews and AI Mode are controlled through Search controls. To appear as a supporting link, a page must be indexed and eligible to show with a snippet. An edge rule that blocks Googlebot removes the page from Search and from AI Overviews together.

## How to test, and what each test proves

- Fetch the live robots.txt from your domain, not the file in your code. It shows what crawlers receive, including anything your CDN adds.
- Request your key pages with AI user agents and with a normal browser user agent, then compare status code and size. A difference shows a rule that acts on the user agent. This is the fastest test and it catches the most common self inflicted blocks.
- Know its limit. A request from your laptop with a crawler's user agent is not the crawler. Edge platforms verify real crawlers by published IP ranges, reverse DNS or signed requests, so a spoofed request can be treated differently. A 200 does not prove the real crawler gets through, and a 403 does not prove it is blocked.
- For decisions based on IP or verification, read your edge or server logs and filter by the IP ranges the vendors publish: OpenAI, Perplexity and Anthropic each publish JSON lists, and Google publishes its crawler ranges and a reverse DNS procedure. On Cloudflare, AI Crawl Control shows requests per AI crawler and its current Allow or Block setting.
- For Google, URL Inspection in Search Console shows whether Google can fetch and index a page.

## What we measured on our own site

On 27 September 2026 we requested tkmstudio.com/services/agent-access-check nine times: once with a Chrome user agent and once each with GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, PerplexityBot and Perplexity-User. All nine returned 200 with the same 32,668 bytes. The response carried server: Netlify and no cf-ray header, and the live robots.txt allows every crawler.

What that proves: no rule on our side acts on these user agents, and Cloudflare is not in the request path. What it does not prove: how a request from a vendor's own IP range is treated. That answer lives in logs. Measure first, then say only what the measurement supports.

## Questions people ask

### If my robots.txt allows GPTBot, can my CDN still block it?

Yes. robots.txt is read by the crawler and followed voluntarily. A CDN or firewall rule acts on the request before any page is served. On Cloudflare, an AI bot policy set to Block refuses the crawler at the edge whatever robots.txt says.

### Does ChatGPT follow robots.txt when a user asks it to open my page?

OpenAI says robots.txt rules may not apply to ChatGPT-User, because the fetch is triggered by a user. Whether that fetch reaches your page depends on your edge rules, not on robots.txt.

### Does blocking Google-Extended remove my site from AI Overviews?

No. Google states Google-Extended does not affect inclusion in Google Search. AI Overviews and AI Mode follow Search controls such as Googlebot access, noindex and snippet settings.

### Can I test AI crawler access with curl?

Partly. curl with a crawler's user agent shows whether a rule acts on that user agent. It cannot reproduce decisions based on the crawler's IP address or verification, because the request does not come from the vendor. For those, read your logs against the vendors' published IP ranges.

## Sources

- [OpenAI, Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots)
- [Perplexity, Perplexity Crawlers](https://docs.perplexity.ai/guides/bots)
- [Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler)
- [Google Search Central, Google's common crawlers](https://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers)
- [Google Search Central, AI features and your website](https://developers.google.com/search/docs/appearance/ai-features)
- [Google Search Central, Verify requests from Google crawlers and fetchers](https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests)
- [Cloudflare Docs, Block AI bots](https://developers.cloudflare.com/bots/additional-configurations/block-ai-bots/)
- [Cloudflare Changelog, New options to manage AI traffic, 1 July 2026](https://developers.cloudflare.com/changelog/post/2026-07-01-ai-traffic-options/)
- [Cloudflare Blog, Your site, your rules: new AI traffic options for all customers, 1 July 2026](https://blog.cloudflare.com/content-independence-day-ai-options/)
- [Cloudflare Docs, robots.txt setting](https://developers.cloudflare.com/bots/additional-configurations/managed-robots-txt/)
- [Cloudflare Docs, Bot Fight Mode](https://developers.cloudflare.com/bots/get-started/bot-fight-mode/)
- [Vercel Docs, Bot Management](https://vercel.com/docs/bot-management)
- [Netlify Docs, Block AI crawlers and bots](https://docs.netlify.com/build/build-with-ai/block-ai-crawlers/)

Check your agent access: https://tkmstudio.com/services/agent-access-check