--- title: Cloudflare and OpenAI's discovery pilot: hundreds of millions of new pages for ChatGPT search canonical_url: https://tkmstudio.com/library/cloudflare-openai-discovery-pilot-chatgpt-search language: en last_updated: 2026-10-17 organization: TKM Studio, AI Visibility & Adoption Studio, Madrid, working worldwide --- # Cloudflare and OpenAI's discovery pilot: hundreds of millions of new pages for ChatGPT search On 15 October 2026 Cloudflare and OpenAI shared early results of a research pilot that uses signals from Cloudflare's network to help OpenAI's search crawler find new pages. With about 1% of Cloudflare's network taking part, OpenAI discovered and indexed hundreds of millions of new web pages. The post does not say how sites joined the pilot, and it gives no traffic figures. What was announced, what it does not change, and what decides whether ChatGPT search can show your site today. ## What was announced On 15 October 2026 Cloudflare published "Cloudflare and OpenAI: Making AI discovery better for everyone" on its blog. The authors are Matthew Conroy, Jack Galilee, Henry Aspegren and James Balfour; the post lists Aspegren and Balfour as OpenAI. The post shares early results of a research pilot the two companies launched in July 2026. Cloudflare announced the pilot in a press release on 8 July 2026. The release describes it as a research pilot to explore how insights from participating websites across Cloudflare's network can help AI search engines discover and index relevant content on the open web. It names the kinds of signals in general terms: content freshness, traffic quality and actual page changes. ## What the pilot did - Cloudflare's network provides signals that can help OpenAI's search crawler find more useful pages. OpenAI then evaluates and ranks those pages independently. - The participating cohort was about 1% of Cloudflare's total network. - According to the post, the pilot has enabled OpenAI to discover and index hundreds of millions of new web pages, and has brought previously unknown pages into OpenAI's search index. - ChatGPT search links back to its sources, so people can open the original content and the sites behind it. ## Why pages with few links matter here The post's examples are a manual for an exact washing machine model, current passport requirements and fees, and a restaurant's latest menu and opening hours. In the authors' words, "these pages can be valuable even if few other sites link to them." The post contrasts this with the older web, where being discoverable meant ranking high on a results page and the visitor judged the content. In AI search, the system itself has to find sources, judge whether they are useful and connect people to them through answers, citations or a visit to the site. ## Less wasted crawling Finding a URL is only the start. The post lists patterns that waste a crawler's work, starting with automated traffic that can make a non-existent page look interesting: - Bots requesting files that do not exist. - Sites that redirect almost any address to the homepage. - Addresses that serve duplicate content. Recognising these patterns lets a crawler skip the fetch before it happens. Change signals tell the crawler when a page is worth another visit. The authors note a trade-off: discovering new pages can initially increase crawling while those pages are evaluated and indexed. The stated goal is to stay current while reducing the overall work. The authors are also testing signals that flag potentially harmful pages. The example is a legitimate site with one compromised page, such as a fake login form planted by an attacker. Flagging that single page, rather than the whole domain, is meant to protect people without penalising the site owner. The post says early results suggest these signals can help identify pages that should be kept out of an index. ## Permissions and training The post states that "a signal about a page doesn't grant permission to access it." It says the research uses metadata about the web for search and discovery, rather than sharing page content for model training. The post points site owners to Cloudflare's AI Crawl Control to understand and manage how AI systems access their sites. Cloudflare's documentation says AI Crawl Control is available on all plans. ## What the post does not say - How sites joined the cohort, how participants were chosen, or how a site can leave. The July press release does not say either. - Which OpenAI crawler is involved. The post only says "OpenAI's search crawler" and does not name OAI-SearchBot. - Any figures for crawl reduction, referral traffic or clicks. - A date for a wider rollout. The next phase, according to the post, will improve page selection, test faster and more reliable change signals, and measure what happens after content is discovered. ## What decides whether ChatGPT search can show your site The pilot is about discovery. Access depends on what your site allows OpenAI's crawlers to do. OpenAI's crawler documentation says OAI-SearchBot is used to surface websites in search results in ChatGPT's search features. Sites that are opted out of OAI-SearchBot will not be shown in ChatGPT search answers, although they can still appear as navigational links. OpenAI recommends allowing OAI-SearchBot in robots.txt and allowing requests from its published IP ranges. After a robots.txt change, OpenAI says it can take about 24 hours for its search systems to adjust. A robots.txt that allows a crawler is not the whole picture: a firewall or hosting rule can still block it. We covered that gap in [robots.txt says yes, your server says no](https://tkmstudio.com/library/robots-txt-allows-ai-crawlers-server-blocks). ## The short checklist - Open your robots.txt and confirm it does not disallow OAI-SearchBot. - If your site runs on Cloudflare, open AI Crawl Control and check which action is set for OAI-SearchBot. - Check that your firewall or hosting does not block OpenAI's published IP ranges for OAI-SearchBot. - Request an address that does not exist on your site and check that it returns a 404 instead of redirecting to the homepage. This check is ours. The post lists sites that redirect almost any address to the homepage among the patterns that waste a crawler's work. - Ask ChatGPT, with search on, the questions your customers ask, and record which pages it cites. Repeat after a few weeks. ## Questions people ask ### Is my site part of the pilot? The post and the July press release do not say how sites were selected or how to join. The post only says the cohort was about 1% of Cloudflare's total network. ### Does Cloudflare share my page content with OpenAI for model training? According to the post, no. The research uses metadata about the web for search and discovery, rather than sharing page content for model training. ### If I block OpenAI's crawler, does the pilot get around it? The post says a signal about a page does not grant permission to access it. Under OpenAI's documentation, a site opted out of OAI-SearchBot is not shown in ChatGPT search answers. ### Will this bring more visitors from ChatGPT? The post gives no traffic or click figures. It says ChatGPT search links back to sources and that the next phase will measure what happens after content is discovered. ### Do I need Cloudflare to appear in ChatGPT search? OpenAI's crawler documentation does not mention Cloudflare. It asks sites to allow OAI-SearchBot in robots.txt and to allow requests from OpenAI's published IP ranges. ## Sources - [Cloudflare Blog, Matthew Conroy, Jack Galilee, Henry Aspegren and James Balfour, Cloudflare and OpenAI: Making AI discovery better for everyone, 15 October 2026](https://blog.cloudflare.com/cloudflare-openai-discovery-agentic-web/) - [Cloudflare, Cloudflare Announces Research Pilot with OpenAI, press release, 8 July 2026](https://www.cloudflare.com/press/press-releases/2026/cloudflare-announces-research-pilot-with-openai/) - [OpenAI, Overview of OpenAI Crawlers](https://developers.openai.com/api/docs/bots) - [Cloudflare Docs, AI Crawl Control overview](https://developers.cloudflare.com/ai-crawl-control/) - [Cloudflare Docs, Manage AI crawlers](https://developers.cloudflare.com/ai-crawl-control/features/manage-ai-crawlers/) See where engines find you: https://tkmstudio.com/services/answer-share-audit