You increased your server capacity, submitted a fresh sitemap, and expected Googlebot to ramp up crawling within days. Instead, crawl stats in Search Console flatlined for three weeks while a competitor with half your page count gets crawled twice daily. The disconnect exists because crawl budget is not a single dial — it is two independent systems, crawl rate limit and crawl demand, and Google calculates each from different signal sets that rarely move in sync. This article breaks down both allocation mechanisms, the signals that feed each one, and the points where you can actually influence the balance.
Crawl rate limit is a server health ceiling, not an SEO signal
Crawl rate limit (Google’s current terminology is “crawl capacity limit”) defines the maximum number of simultaneous parallel connections and the time delay between fetches that Googlebot will use on a given host. Google’s crawl budget documentation states this ceiling exists to “provide coverage of all your important content without overloading your servers.” The calculation is entirely mechanical. It does not factor in content quality, backlink profiles, or domain authority.
Two inputs control the ceiling. First, crawl health: if a site responds quickly over a sustained period, the limit increases, allowing more parallel connections. If the site slows down or returns server errors, the limit drops and Googlebot crawls less. Second, Google’s own infrastructure constraints. Google allocates finite crawler resources across billions of hosts, and your share competes with every other domain on the same IP range or data center segment.
The rate limit adjusts dynamically during a crawl session. Googlebot monitors response times in near real-time. A server that starts a session at 150ms TTFB but degrades to 900ms under load will see connections throttled mid-session. This is not a penalty. It is a protective mechanism that prevents Googlebot from contributing to a site outage. The adjustment happens at the connection scheduler level, well before any quality or indexing system evaluates the content.
One common misunderstanding: increasing server capacity (adding more CPU, more RAM) does not automatically raise the rate limit. The limit responds to observed latency and error rates during actual crawl sessions, not to theoretical capacity. A server with 64 cores that still takes 600ms to serve HTML due to unoptimized database queries will receive a lower rate limit than a 4-core server responding in 80ms.
Since January 2024, the manual crawl rate limiter tool in Search Console has been deprecated. Google’s position, stated in the deprecation announcement, is that improvements to their crawling logic make the manual tool unnecessary. Googlebot now reacts to HTTP response behavior directly, slowing down automatically when it encounters persistent 500-series errors or elevated response times.
Crawl demand scores URLs independently based on staleness and value signals
Crawl demand determines which URLs Googlebot actually wants to fetch, regardless of available server capacity. Even if the crawl rate limit allows 100 requests per second, Googlebot will not use that capacity unless demand justifies it. Google’s documentation is explicit: “if crawl demand is low, Googlebot will crawl your site less.”
The primary demand signals, as described by Gary Illyes in the original 2017 crawl budget post and subsequent clarifications, are popularity and staleness. Popularity correlates with how often a URL appears in search results, receives clicks, and attracts external links. Staleness reflects how long it has been since Googlebot last fetched a URL relative to its observed change frequency. Google builds a predictive model per URL, estimating when content is likely to change based on historical patterns.
A third signal, perceived inventory, represents how many URLs Google believes exist on a site. Without explicit signals like robots.txt directives or noindex tags, Googlebot attempts to crawl every URL it discovers. If a large portion of those URLs are duplicates, parameter variations, or thin pages, the demand system wastes allocation on low-value fetches. This is the single most controllable factor in crawl demand management.
Site-wide events also trigger demand spikes. A domain migration, a major redesign detected through changed URL patterns, or a sudden influx of new external links can all temporarily increase demand. Gary Illyes confirmed on LinkedIn in 2024 that internal links also impact crawl demand, meaning changes to site architecture can shift which URLs receive crawl priority.
The demand system operates independently per crawler. AdsBot has its own demand profile (higher when dynamic ad targets are active), Googlebot-Image responds to image inventory changes, and the primary Googlebot evaluates the main HTML document corpus. Each crawler’s demand draws from the same rate limit ceiling, which is why sites running Google Shopping feeds or dynamic ad campaigns sometimes observe reduced main crawl capacity.
The interaction model between rate limit and demand creates four crawl states
The two systems combine into a simple matrix that explains most crawl behavior patterns observed in Search Console crawl stats and server logs.
State 1: High rate limit, high demand. Googlebot crawls aggressively. This is the target state for large sites. The crawl stats report shows consistently high request counts with low average response times and minimal error rates.
State 2: High rate limit, low demand. Available server capacity goes unused. This is the most common state for small to medium sites with stable content. It is also the normal state, not a problem to solve. Google’s documentation explicitly states that most sites do not need to worry about crawl budget at all.
State 3: Low rate limit, high demand. Googlebot wants to crawl more URLs than the server can handle. This produces the symptoms that trigger most crawl budget audits: important pages going stale, new content taking weeks to appear in the index, and crawl stats showing flat request counts despite growing content. The fix is always on the rate limit side, specifically server performance.
State 4: Both low. The server is slow and the content does not generate demand. This state often indicates deeper problems: thin content, excessive duplication, or a site that has lost external link equity. Fixing server speed alone will not help because demand remains insufficient to trigger increased crawling.
Diagnosing which state applies requires correlating two data sources. Search Console’s crawl stats report shows request volume, response time distribution, and response code breakdown. Server logs show actual Googlebot hit patterns, including which URL segments receive crawls and which are ignored. The crawl stats report alone cannot distinguish between State 2 (unused capacity) and State 4 (both low) without log-level URL segment analysis.
Server response latency is the fastest lever for shifting crawl rate limit
Reducing average server response time is the single most effective intervention for increasing crawl rate, and the results appear faster than any other crawl budget optimization. When TTFB drops from 800ms to 200ms, the observable effect in crawl stats typically appears within days, not weeks. Google’s documentation confirms this directly: “a speedy site is a sign of healthy servers, so it can get more content over the same number of connections.”
The relationship between latency and crawl rate is nearly linear within the operational range. At 100-200ms TTFB, Googlebot can sustain high connection counts. Between 200-500ms, crawling proceeds at moderate rates. Above 1,000ms consistently, crawl rates drop substantially. John Mueller has noted that response times between 100ms and 500ms are generally associated with efficient crawling.
TTFB matters more than full page load time for crawl rate purposes. Googlebot’s initial fetch is an HTTP request for raw HTML. Rendering happens separately, in a different system, on a different timeline. The rate limit scheduler only sees the document fetch latency. A page that takes 4 seconds to fully render in a browser but delivers HTML in 120ms will receive a high rate limit.
CDN configuration affects the calculation in a specific way. If a CDN serves cached HTML responses, Googlebot sees the CDN edge latency (typically 20-80ms), not the origin server latency. This effectively decouples the rate limit from origin server performance. However, CDN cache misses that fall through to a slow origin will spike latency and trigger rate limit reductions. Consistent cache hit ratios above 90% for Googlebot requests are the threshold where CDN benefits stabilize.
One overlooked factor is DNS resolution time. During each crawl session, Googlebot performs DNS lookups. A DNS server responding in 200ms instead of 20ms adds 180ms of overhead per request. Over a crawl session touching thousands of URLs, this accumulates into measurable rate limit reduction.
Content change signals are the fastest lever for shifting crawl demand
Googlebot builds a per-URL prediction model for content change frequency. If a URL changes every Tuesday, Googlebot learns to recrawl on or near Tuesdays. If a URL has not changed in six months, recrawl intervals extend accordingly. Influencing this model requires actual content changes that Googlebot can detect and verify.
Cosmetic changes do not count. Updating a timestamp, rotating a session ID in a widget, or changing a sidebar ad unit will not register as meaningful content change in Google’s systems. The change detection operates on the main content of the page. Google’s systems compare the substantive content between crawls, not the raw HTML byte-for-byte. This is consistent with how the indexing pipeline works: it extracts main content, strips boilerplate, and evaluates the delta.
The lastmod tag in XML sitemaps interacts with the change prediction model, but only when used accurately. Google’s documentation states that it uses lastmod values “if it’s consistently and verifiably accurate.” Sites that update lastmod on every page generation regardless of actual content changes train Google to ignore the signal entirely. Accurate lastmod usage, where the date changes only when substantive content changes, reinforces the prediction model and can accelerate recrawling of genuinely updated pages.
The changefreq tag in sitemaps is effectively ignored by Google. It is described as “only a hint” in Google’s documentation and has no observable impact on crawl scheduling in practice.
Publishing patterns also matter at the site-section level. A /blog/ directory that publishes daily will develop a higher baseline demand than a /legal/ directory that changes annually. Googlebot allocates demand at both the URL level and the URL-pattern level, meaning new URLs published in a high-frequency section inherit a higher initial demand than new URLs in a dormant section. This is one mechanism behind the observation that content published in active site sections gets crawled faster than content added to neglected areas.
Common interventions that fail to move crawl budget and why
Increasing server capacity without reducing latency. Adding more servers behind a load balancer does not help if the application layer still takes 700ms to generate each response. The rate limit responds to observed response time, not to infrastructure specifications. This is the most expensive failed intervention, often involving hosting upgrades that show no change in crawl stats.
Submitting URLs through the Indexing API for non-eligible content. The Indexing API is restricted to JobPosting and BroadcastEvent structured data types. Submitting other content types through this API has no effect on crawl scheduling. Google has been explicit about this restriction, and abuse of the API for general content has been addressed in multiple Google Search Central office hours sessions.
Requesting indexing via the URL Inspection tool. The “Request Indexing” function in Search Console submits a single URL for priority crawling. It does not affect the site’s overall crawl budget allocation. It is useful for individual URL emergencies but does not scale, and Google rate-limits the tool to prevent abuse. Repeated use across thousands of URLs will not shift the crawl demand curve.
Blocking low-value URLs with robots.txt to “redirect” budget. Google’s crawl budget documentation addresses this directly: “Google won’t shift this newly available crawl budget to other pages unless Google is already hitting your site’s serving limit.” In State 2 (high rate limit, low demand), blocking URLs with robots.txt simply reduces total crawl volume. The freed capacity does not transfer to other pages because the rate limit was never the constraint.
Changing URL parameters in Search Console (deprecated). The URL Parameters tool was deprecated and removed. Even when it existed, its primary function was to help Google understand parameter-based duplication, not to directly allocate crawl budget. The removal reflects Google’s improved ability to handle parameters automatically.
The interventions that do work consistently are reducing server latency (rate limit side), consolidating duplicate content (demand side), improving internal linking to high-value pages (demand side), and publishing content that generates genuine search demand (demand side). Each targets one of the two systems directly, rather than attempting to manipulate crawl budget as a single entity.
Does AdsBot and Googlebot-Image share the same crawl rate limit as the primary Googlebot?
All Googlebot variants draw from the same crawl rate limit ceiling for a given host. AdsBot, Googlebot-Image, and the primary Googlebot each maintain separate demand profiles, but the total simultaneous connections they open count against one shared capacity limit. Sites running Google Shopping feeds or dynamic ad campaigns frequently observe reduced main crawl throughput because AdsBot requests consume part of that shared ceiling during overlapping sessions.
Does crawl budget matter for sites with fewer than 10,000 pages?
For most sites under 10,000 pages, crawl budget is not a practical concern. Google’s crawl demand system can cover the entire URL inventory well within default rate limits. The crawl budget framework becomes relevant only when URL count, duplication, or server latency prevents Googlebot from reaching important pages within a reasonable recrawl interval. Sites at this scale benefit more from content quality and internal linking improvements than from crawl budget optimization.
Does switching CDN providers reset the crawl rate limit Googlebot has learned for a domain?
Googlebot’s rate limit is tied to observed response latency per host, not to a specific CDN or IP range. Switching CDN providers changes the IP addresses Googlebot connects to, and the rate limit recalibrates based on the new response times. If the new CDN delivers faster TTFB, the rate limit adjusts upward within a few crawl sessions. A CDN migration that temporarily increases latency through misconfigured cache rules or elevated cache miss rates will trigger a temporary rate limit reduction until performance stabilizes.
Sources
- Large Site Crawl Budget Management — Google’s official guidance on crawl capacity limit, crawl demand, and budget management for large sites
- What Crawl Budget Means for Googlebot — Gary Illyes’ foundational 2017 post defining crawl rate limit and crawl demand as the two components of crawl budget
- Myths and Facts About Crawling — Google’s crawling infrastructure documentation debunking common crawl budget misconceptions
- Crawl Rate Limiter Deprecation Announcement — Google’s explanation of why the manual crawl rate tool was removed and how Googlebot self-adjusts
- Crawling December: How and Why of Googlebot Crawling — December 2024 updated guidance from Martin Splitt and Gary Illyes on crawling behavior
