Robots.txt disallow prevents crawling, it does not prevent discovery, and those are two separate mechanisms in Google’s pipeline. When a URL is disallowed, Googlebot stops fetching its content, but it can still learn the URL exists (through internal links, sitemaps, or other references) and will keep that URL in its crawl frontier, periodically checking whether it’s still disallowed. If your site keeps generating new search-result URLs (a new one for essentially every unique query typed into the search box), each new variant gets discovered the same way, through crawlable links to it, even though none of them will actually be fetched once discovered.
Why robots.txt disallow doesn’t stop discovery of search URLs
Google’s robots.txt documentation is explicit that disallow is a crawling instruction, not an indexing or discovery instruction. Discovery, crawling, and indexing are three distinct stages: discovery is Google learning a URL exists at all (via a link, a sitemap entry, or any other reference); crawling is actually requesting and fetching that URL’s content; indexing is deciding whether and how to include what was fetched in the search index. Disallow interrupts the crawling stage specifically. It does nothing to stop discovery, and because Googlebot never gets to fetch the page, it also never sees a noindex directive that might have been present in that page’s HTML, since reading a noindex tag itself requires a crawl to happen first.
This creates a specific, self-perpetuating pattern for internal site search: if your search form or search results pages generate crawlable links (a “did you mean” suggestion, a “recent searches” widget, a paginated results structure, faceted filters within search results), Googlebot keeps encountering new link targets it hasn’t seen before, adds them to its discovery queue, attempts to check them, gets blocked by robots.txt each time, and moves on, only to encounter more newly-generated variants later. The URLs are never crawled, but they’re also never definitively resolved or dropped from consideration, since nothing has told Google’s systems these paths are dead ends rather than simply temporarily blocked.
By contrast, if these pages were noindex instead of (or in combination with) being crawlable, Google would fetch them once, see the noindex directive, and generally stop prioritizing recrawls of URLs it has confirmed are excluded, and the confirmed-excluded status can reduce how aggressively they get re-enqueued going forward. Robots.txt alone skips that resolution step entirely, leaving the URLs in a permanent “still checking” state from Google’s perspective.
A concrete worked example makes the compounding math tangible. Imagine an internal search results page that includes a “related searches” or “people also searched” widget, a common UX pattern that suggests five additional query variations as clickable links beneath the main results. Each of those five suggestions is itself a link to another internal search results URL, one that likely has its own “related searches” widget generating five more suggestions in turn. On a site with meaningful search traffic, that single widget pattern can generate an enormous number of unique crawlable URLs over time, since every distinct query a real user types produces a results page, and that results page then spins off five more linked query variations that a user never typed but that now exist as crawlable, linked destinations regardless. None of these get fetched for content once disallowed, but each one still has to be discovered, added to the crawl frontier, checked against robots.txt, and rejected, over and over, as Googlebot periodically revisits its list of known-but-blocked URLs. On a large site with years of accumulated search query variety and a busy related-searches widget, this isn’t a handful of stray URLs, it’s a continuously regenerating, effectively unbounded set, and robots.txt disallow does nothing to shrink or cap that set since it only ever intervenes at the fetch step, well after discovery has already happened.
There’s also a diagnostic difference worth knowing about in Search Console’s URL Inspection tool, since it’s one of the more direct ways to observe this mechanism in practice rather than just reasoning about it abstractly. A URL that’s blocked by robots.txt reports a status along the lines of “blocked by robots.txt” indefinitely, essentially forever, for as long as the disallow rule remains in place and Google keeps that URL in its awareness, because that status simply reflects the current robots.txt rule each time the tool checks, not a historical resolution. A URL that was instead allowed to be crawled once, found to carry a noindex tag, and consequently excluded reports a materially different status, something to the effect of “excluded by noindex tag,” which reflects an actual fetch having happened and a directive having been read and honored. The first status is a standing, repeatedly-reconfirmed condition; the second is closer to a settled, resolved outcome tied to a specific crawl event. If you’re auditing a site and pulling representative URLs from the disallowed search-page pattern into URL Inspection, seeing “blocked by robots.txt” persist unchanged over weeks or months, rather than ever transitioning into a resolved-and-quiet state, is itself evidence of the exact discovery-without-resolution loop this mechanism produces.
How to actually resolve crawlable internal search URLs
- Recognize that robots.txt disallow, noindex, and “stopping Google from knowing the URL exists” are three different outcomes, and pick the tool based on which outcome you actually need.
- The most durable fix is preventing the URL pattern from being generated as crawlable links in the first place, adding
rel="nofollow"or removing hrefs from search-suggestion/related-search widgets so new search-result URLs stop entering the discovery queue at the source, rather than relying entirely on robots.txt to catch them after the fact. - If crawl-budget conservation is the primary goal and you’re comfortable with these pages never resolving to a confirmed excluded state, robots.txt disallow is still the right call, since it accomplishes the actual budget-saving goal even if Googlebot keeps periodically rechecking blocked status.
- If getting these URLs to a fully resolved, no-longer-checked state matters (cleaning up historical index bloat, for instance), a temporary noindex-and-allow-crawl phase, followed by disallow once Google has confirmed the noindex, is the documented sequence for permanently resolving already-known URLs, rather than jumping straight to disallow.
- Don’t assume a disallowed URL is invisible to Google in any sense. It is not being fetched, but it can still appear as a URL Google is “aware of,” including in some reporting contexts, and continued periodic discovery-stage requests are the visible symptom of that ongoing awareness.
- Pull a sample of disallowed search-result URLs into Search Console’s URL Inspection tool periodically and check whether the status is a static, unchanging “blocked by robots.txt” versus a settled “excluded by noindex tag,” since that difference is a quick, concrete way to confirm whether a given batch of URLs is actually resolved or still stuck in the discovery loop.
- Audit raw server log files for actual Googlebot request volume against the disallowed search-path pattern before deciding how much engineering effort the fix deserves. Filtering server logs (by user agent and by the relevant path prefix) shows the real request count Googlebot is generating against these blocked URLs over a given period, which turns “this is probably wasting some crawl budget” into an evidence-based number you can compare against the site’s total crawl activity for the same window, and that comparison is far more persuasive to engineering stakeholders than a general statement about how the mechanism works.
- When reviewing log data, look specifically for repeat requests to search-URL patterns that were disallowed months or years earlier, since finding Googlebot still periodically requesting decade-old blocked query-string patterns is direct, site-specific confirmation that the discovery queue for this pattern never actually closes out on its own.
The mechanism in one line: disallow stops fetching, not knowing, and newly-generated URLs keep re-entering a queue that robots.txt alone can never fully close out.
