NEWS
Moving to a CDN without losing Google: three failure points and Google's own fixes
On 11 November 2026 Google's Search Central blog explained what happens to crawling when a site moves behind a Content Delivery Network (CDN): why crawler traffic to the origin server can jump, why the new cache does not help at the start, and how blanket bot protection can push pages out of Search. What Google says, and the checks to run before and after the switch.
What Google published
On 11 November 2026 the Google Search Central blog published "Moving to a CDN without overwhelming your origin (or blocking Googlebot)". Google writes that it sometimes hears from site owners that, right after putting their site behind a CDN, crawler traffic to their server spikes almost immediately. The post explains the transition phase and points to two existing guides: how crawl budget works, and how to reduce Google's crawl rate in emergencies.
Failure point 1: faster responses mean faster crawling
Google's crawling infrastructure calculates a crawl capacity limit for each site, also known as hostload. When response times increase, Google's crawlers slow down. When a server responds quickly and consistently, and there is crawl demand, the crawl capacity limit gradually goes up.
Google writes that the moment your DNS points to a fast CDN edge network, connection and response times drop, and to its crawling systems it looks like your server just got a massive hardware upgrade. The crawlers start ramping up their request rate to match. Google adds that crawlers do not ramp up immediately, and that many CDNs include built-in connection pacing that helps smooth out traffic spikes.
Failure point 2: a cold cache passes requests to your origin
A new CDN starts with an empty, or cold, cache. When an edge location has no copy of a URL yet, that request is a cache miss: the edge server fetches the response from your origin, stores a cacheable copy and then serves it. Only later requests for the same URL are served from the edge.
Google points out the difference between visitors and crawlers. Human visitors cluster around a relatively small set of popular pages, which warm up in the cache quickly. Crawlers systematically explore the breadth of your URL inventory, so every uncached URL they request goes straight through to your origin. Until the edge caches are warm, Google writes, your origin server is still doing the heavy lifting.
Failure point 3: blanket bot protection blocks Googlebot
Seeing a surge of automated requests in origin logs, an admin may switch on blanket bot protection in the CDN dashboard. Google explains that blanket rules without exemptions treat all automated traffic the same, including the crawlers you care about. When a CDN or Web Application Firewall (WAF) intercepts automated requests with a challenge like a CAPTCHA, Google's crawlers fail the test.
Google describes the result: crawlers get stuck on an error page that is not useful for Search or other products that rely on crawled data, the site appears deliberately restricted or broken, the crawlers back off, and pages and shopping listings begin to drop out of Search.
Google's fixes
- ·If you need to block bots, allow verified bots in your CDN and firewall settings. Google notes that most major CDNs and WAFs offer a built-in toggle or managed rule for known or verified bots, and that you can identify Google's crawlers via its published IP ranges. Google also suggests using robots.txt or rules that target specific unwanted bots instead of blocking all automated clients.
- ·Pre-warm your caches before routing all traffic through the CDN: fetch your highest-traffic pages, upcoming peak-traffic pages such as seasonal launches, and your most frequently crawled URLs through the CDN at a controlled pace.
- ·If crawl volume still puts critical load on your origin, return a 429 (Too Many Requests) or 503 (Service Unavailable) status code for a short window. The blog says a few hours, or 1 to 2 days at most. Google's guide on reducing the crawl rate warns that keeping this up for longer than 1 to 2 days may hurt how the site appears in Google products, and that a URL returning these codes for multiple days may be dropped from Google's index.
Quick checklist
- ·Before the switch, list your most visited and most crawled URLs and plan a controlled cache warm-up.
- ·Check every bot protection, challenge or CAPTCHA rule in the CDN and WAF, and confirm verified crawlers are exempt.
- ·Target specific unwanted bots in robots.txt or in rules, not all automated clients.
- ·After the switch, watch origin load and server logs for crawler requests that pass the cache.
- ·Use 429 or 503 only as a short brake, never for more than 1 to 2 days.
Questions people ask
Why did crawler traffic to my server go up after moving to a CDN?
Google says fast and stable responses raise the crawl capacity limit, so crawlers ramp up their request rate. With a cold cache, each request for a URL the edge has not stored yet still reaches your origin.
Will a CDN bot protection setting block Google?
It can. Google says blanket rules without exemptions treat all automated traffic the same, and its crawlers fail CAPTCHA challenges. Allow verified bots, or target specific unwanted bots instead.
How do I slow Google down without blocking it?
Return 429 or 503 for a short window, a few hours or 1 to 2 days at most. Google warns that longer periods may affect how the site appears in its products, and URLs can be dropped from the index if the codes persist for multiple days.
Does this apply to AI crawlers too?
The post is about Google's crawlers. Google writes that a blocked crawler sees an error page that is not useful for Search or other products relying on crawled data. For other AI crawlers, see our article on why AI crawlers still get refused when robots.txt allows them.
Sources
- ·Google Search Central blog, Moving to a CDN without overwhelming your origin (or blocking Googlebot): https://developers.google.com/search/blog/2026/11/crawling-cdn
- ·Google Crawling, Optimize your crawl budget: https://developers.google.com/crawling/docs/crawl-budget
- ·Google Crawling, Reduce the Google crawl rate: https://developers.google.com/crawling/docs/crawlers-fetchers/reduce-crawl-rate
- ·Google Crawling, Verify requests from Google crawlers and fetchers: https://developers.google.com/crawling/docs/crawlers-fetchers/verify-google-requests