Blocking a URL in robots.txt only prevents Googlebot from crawling that URL going forward, it contains no instruction to remove anything already in the index. If a page was indexed before the disallow rule was added, or gets indexed afterward through some other discovery path like an external link, robots.txt has no mechanism to reach into the index and pull it out. The result is often a page that lingers in search results with a degraded snippet, frequently something to the effect of no information being available for that page, because Google can see the URL exists (from links, sitemaps, or its own prior crawl history) but is now forbidden from crawling it to confirm what’s actually there or whether it should still be indexed.
Direct answer
Disallowing a path in robots.txt and removing a page from Google’s index are two entirely different operations, and conflating them is one of the most common technical SEO mistakes. Robots.txt is exclusively a crawling directive. It tells Googlebot “do not fetch this URL,” nothing more. It does not tell Google “and also delete whatever you already know about this URL.” If you want a page actually removed from the index, you need a mechanism that operates on indexing directly, primarily a noindex directive, and robots.txt actively gets in the way of that because it prevents Google from ever crawling the page to see the noindex tag in the first place.
Why a disallowed-but-indexed page persists
Google’s crawling and indexing processes are logically separate stages, even though they’re often talked about as one pipeline. Crawling is the act of fetching a URL’s content. Indexing is the decision to store and potentially serve that URL in search results. A URL can end up indexed based on signals other than Google having crawled and read its full content, most notably when other pages link to it. Google can observe the URL, the anchor text pointing to it, and contextual signals, and decide to index it even with limited direct knowledge of the page’s content. This is exactly the scenario that produces the degraded “no information is available for this page” snippet: Google has an index entry for the URL but is barred by robots.txt from crawling it to generate a proper description.
Once robots.txt disallows the path, this situation becomes self-reinforcing rather than self-correcting. Because Googlebot respects the disallow rule, it will not crawl the page to discover a noindex tag, an updated canonical, a 404 status, or any other signal that might otherwise prompt removal. The disallow rule freezes Google’s knowledge of the page at whatever state existed when crawling stopped, and that stale index entry has no natural path to expiring just because you’ve decided you don’t want it crawled anymore. Google’s own documentation on removing content is explicit that using robots.txt disallow as a removal method is a mistake for precisely this reason: it can prevent Google from seeing directives meant to actually deindex the page.
The correct multi-step deindexing workflow
Step one: ensure the page remains crawlable, and add a noindex directive. Before anything else, confirm the URL is not blocked in robots.txt, since a noindex tag is worthless if Googlebot can never reach the page to read it. Add either a meta robots tag in the page’s head (<meta name="robots" content="noindex">) or an X-Robots-Tag HTTP header carrying the same instruction for non-HTML resources. This is the actual removal instruction. It tells Google directly, at the page level, “you may crawl this, but do not include it in your index.”
Step two: wait for Google to recrawl the page and observe the new directive. This isn’t instantaneous. Google has to revisit the URL on its normal or expedited crawl schedule, parse the response, and act on the noindex signal. For pages Google already considers low-priority or infrequently updated, this can take longer than site owners expect. Requesting indexing via URL Inspection in Search Console can sometimes prompt a faster recrawl, but it’s a request, not a guarantee of immediate processing.
Step three: confirm removal via URL Inspection and the Index Coverage (Page Indexing) report. Use the URL Inspection tool to check the current indexed status of the specific URL and confirm Google’s last crawl picked up the noindex tag. At a broader level, the Page Indexing report in Search Console will eventually reflect the URL under an “Excluded” status specific to noindex, confirming the directive was honored across the set of URLs you’re tracking, not just the one you spot-checked.
Step four: only block in robots.txt afterward, once removal is confirmed, if ongoing crawl prevention is actually desired. Once you’ve verified the page is out of the index, it’s safe to add a robots.txt disallow if your goal is also to stop Google from spending any further crawl attention on that path. At that point, the disallow isn’t fighting against a needed directive, since the removal has already happened. Adding it earlier is what breaks the workflow, adding it after is just crawl management.
Optional interim step: use the Search Console Removals tool for urgent, temporary hiding. If a page needs to disappear from search results quickly, for legal, privacy, or reputational reasons that can’t wait for the natural crawl-and-noindex cycle, the Removals tool can hide a URL from search results on an expedited basis. This is explicitly documented as temporary, generally covering a window of roughly six months, not a permanent removal mechanism. It buys time while the actual permanent fix (noindex, then confirmation, then optionally robots.txt) works through its normal cycle. Relying on Removals alone, without also implementing noindex, means the page will eventually resurface in results once the temporary hiding period lapses.
As a hypothetical example: imagine a hypothetical site, “Site T,” that adds a robots.txt disallow rule for an old /promo/ path to try to get a batch of outdated promotional pages out of search results. Hypothetically, if those pages had already been indexed months earlier and were linked from a few external directories, the disallow rule would freeze Google’s existing index entries in place rather than removing them, and Site T would likely see those URLs continue appearing in results for months afterward with a degraded snippet, since Googlebot is now barred from recrawling the pages to ever discover a noindex tag, even one added later.
Common failure pattern to avoid
The failure mode this whole workflow exists to prevent is doing steps out of order: blocking the path in robots.txt first, in the belief that this constitutes removal, then being confused weeks or months later when the URL still appears in search results with a degraded snippet and no path to fixing it short of temporarily unblocking the page again so the noindex tag can actually be seen. If you find yourself in that state already, the fix is to reverse the mistake: remove the robots.txt disallow, add the noindex directive, let Google recrawl and process it, confirm removal, and only then reapply any crawl blocking you still want in place. There is no shortcut that lets Google act on a directive it is simultaneously being blocked from reading.
