The optimal strategy is to actively suppress crawl access to the low-value 85% (through robots.txt disallow for URLs that don’t need to be crawled at all, and canonicalization or noindex for URLs that need to remain accessible but shouldn’t be independently indexed), so that Google’s finite crawl demand and rate allocation concentrates on the 15% of URLs that actually drive traffic. At this scale, crawl budget is a genuine, binding constraint, unlike a small site where Google can reasonably crawl everything without difficulty, and the practical lever isn’t asking Google to crawl more, it’s reducing the size of the space competing for the crawl attention that already exists.
The mechanism: crawl budget as a real constraint only at scale
Google’s Large Site Owner’s Guide to managing crawl budget is explicit that this concept only meaningfully applies to sites in the range of many thousands to millions of URLs; for the vast majority of smaller sites, Google states crawl budget isn’t something to worry about since existing crawl capacity comfortably covers the site’s actual URL count. A 10M+ URL site sits squarely in the category where the guide’s concern is real: Google allocates a rate (bounded by what the server can handle without strain) and a demand (how much value Google’s systems estimate in crawling that URL, driven by perceived popularity, freshness needs, and quality), and both are finite relative to the sheer number of URLs competing for attention.
When 85% of URLs generate no organic traffic, that’s a strong indicator that a large share of the crawlable space consists of low-value duplication, parameter combinations, thin or empty-result pages, or otherwise low-priority content that’s nonetheless consuming crawl requests that could otherwise go toward the 15% that matters. Every crawl request Google spends discovering, fetching, and processing a low-value URL is a request not spent on a URL that could be ranking and driving traffic; at this scale, that tradeoff is a real, measurable cost rather than a theoretical one.
Building the tiering framework
A practical approach segments the URL space into three tiers rather than treating it as a single undifferentiated mass.
Tier 1: revenue/traffic-driving pages. The confirmed 15% (or whatever the actual data shows), identified via existing Search Console and analytics data on which URLs receive organic clicks or conversions. These should have the strongest internal linking, be included in sitemaps, and receive priority attention for technical health (fast response times, clean status codes, accurate canonical signals).
Tier 2: supporting pages. URLs that don’t independently drive much traffic but serve a legitimate structural or user-experience purpose, category pages that aid navigation, pagination, or filtered views with some genuine search demand behind them. These can remain crawlable and indexable but don’t need the same priority treatment as tier 1, and are good candidates for lighter internal linking and exclusion from priority sitemap sections.
Tier 3: low-value duplicates and combinations to suppress. Parameterized URLs (sort, filter, session, tracking parameters that don’t produce meaningfully different content), thin or zero-result pages, and other programmatic combinations that don’t correspond to real search demand. These are the primary target for suppression.
Hypothetically, imagine a marketplace, “Boltwork Industrial Supply,” with 10 million URLs where log analysis shows Googlebot spending a large share of its daily crawl activity on parameterized session and sort-order variants of a few thousand product-listing pages, while genuinely new product pages sit uncrawled for weeks after launch. After tiering, Boltwork might disallow the sort- and session-parameter patterns entirely in robots.txt, apply canonical tags to genuine near-duplicate size/color variants that still need to remain indexable, and leave the roughly 15% of URLs with confirmed organic traffic fully open with strong internal linking. The expected result, checked against server logs rather than assumed, would be crawl activity gradually shifting away from the parameter combinations and toward new and existing revenue-driving pages.
Choosing the right suppression mechanism matters
A common and costly mistake at this scale is reaching for noindex when the actual goal is stopping Google from crawling a URL at all. Noindex still requires Google to crawl the page to see the noindex directive in the first place, so it doesn’t save any crawl budget, it only prevents the URL from being indexed after it’s already been crawled. For URLs that should be kept out of the crawl queue entirely (parameter combinations, faceted-navigation dead ends, internal search results pages), a robots.txt disallow is the mechanism that actually prevents the crawl request from happening. Noindex remains the correct tool for URLs that do need to be crawled for other reasons (to pass through link equity, or because Google needs to see them to understand canonicalization) but shouldn’t appear independently in the index. Canonicalization is the right tool for genuine near-duplicate content where consolidating signals to a single preferred URL is the goal, rather than blocking crawl access outright.
Practical implication
Start from actual traffic and click data (Search Console performance data segmented by URL pattern, cross-referenced with server logs showing current crawl distribution) to confirm which URL patterns fall into which tier, rather than guessing at the segmentation. Apply robots.txt disallow to entire URL patterns that are structurally low-value and don’t need individual-URL evaluation (most parameterized and faceted-navigation combinations), reserve canonical tags for genuine near-duplicate content where consolidation rather than exclusion is the goal, and use noindex selectively for pages that must remain crawlable but shouldn’t independently rank. There’s no fixed daily crawl “quota” Google publishes or guarantees, so track the practical result via Crawl Stats trends (crawled URL counts by section, ideally cross-referenced against server logs) to confirm that suppression of the low-value 85% is actually shifting a measurably larger share of ongoing crawl activity toward the tier 1 pages that drive the site’s real value, rather than assuming the reallocation happened just because the low-value URLs were blocked.
