In a survey of 150 SEO practitioners, 43% believed that adding a disallow rule in robots.txt would eventually remove already-indexed pages from Google’s search results. It does not. Robots.txt controls crawling, not indexing — and these are separate systems. A disallow rule prevents Googlebot from fetching the page, which means Google cannot see a noindex tag even if one exists. The page remains in the index, potentially indefinitely, displaying whatever snippet Google last cached. This misunderstanding is one of the most common causes of failed deindexing projects.
Robots.txt disallow prevents crawling, which prevents Google from reading removal signals
The pipeline sequence makes this limitation unavoidable. When a URL is blocked by robots.txt, Googlebot cannot make an HTTP request to fetch the page content. Since the noindex directive exists as either a <meta name="robots" content="noindex"> tag in the HTML or an X-Robots-Tag: noindex HTTP header, both require Googlebot to actually request the page. A disallowed URL never receives that request.
Google’s documentation is direct on this point: “a robots.txt file is used mainly to avoid overloading your site with requests; it is not a mechanism for keeping a web page out of Google.” John Mueller reiterated in September 2024 that robots.txt is a crawling control, not an indexing control. The two systems are architecturally separate.
The implication is that combining robots.txt blocking with noindex is self-defeating. Google’s documentation states explicitly: “For the noindex rule to be effective, the page or resource must not be blocked by a robots.txt file, and it has to be otherwise accessible to the crawler.” If both are applied, the robots.txt block prevents Google from ever seeing the noindex tag. The page remains indexed.
Google considered supporting a noindex directive within robots.txt itself. Internal discussions explored whether it should be added to the robots.txt standard. The conclusion was against it, as Mueller explained, because robots.txt files are frequently copied and pasted without careful review, making it too easy to accidentally deindex critical site sections. Google officially retired all code handling the unsupported noindex directive in robots.txt in September 2019.
Already-indexed pages behind robots.txt blocks can persist in search results for years
When a previously indexed URL is blocked by robots.txt, Google does not remove it from the index. Instead, the URL continues appearing in search results with a restricted listing. The listing typically shows the URL and possibly a title derived from anchor text or historical data, but with no snippet or a message indicating the description is unavailable due to robots.txt.
These “zombie” listings persist because Google never forgets a URL it has discovered. The robots.txt block prevents recrawling, which means Google cannot update its information about the page, confirm it still exists, or detect any removal signals. The last-crawled version remains in the index in a frozen state.
The persistence timeline varies but can extend indefinitely. Observed cases show blocked-but-indexed URLs remaining in search results for 2+ years without any sign of natural decay. Google’s systems treat the block as “this page exists but we cannot access it,” not as “this page should be removed.” Without an explicit removal signal that Google can detect, the URL has no reason to leave the index.
These zombie listings are not benign. They occupy positions in search results that could be filled by active pages. They present outdated information to users who click them. They generate crawl errors in Search Console’s coverage report (“Indexed, though blocked by robots.txt”), creating noise that obscures genuine issues. On sites with thousands of such pages, the cumulative effect dilutes the site’s overall quality signals.
The Correct Multi-Step Deindexing Workflow Using Noindex
The reliable deindexing sequence requires temporary crawl access so Google can detect the removal signal.
Step 1: Remove the robots.txt disallow rule. Edit robots.txt to allow Googlebot access to the target URLs. Wait for the robots.txt cache to refresh (up to 24 hours, though usually faster for active sites). Verify in server logs that Googlebot is now crawling the previously blocked URLs.
Step 2: Add noindex to the target pages. Implement either <meta name="robots" content="noindex"> in the HTML head or X-Robots-Tag: noindex in the HTTP response header. The X-Robots-Tag method is preferable for non-HTML resources or when modifying HTML templates is impractical.
Step 3: Wait for Googlebot to crawl and process. Monitor the URL Inspection tool in Search Console. When Googlebot crawls the page and detects the noindex directive, the page transitions to “Excluded by noindex tag” status. This typically takes 1-4 weeks depending on the URL’s crawl frequency.
Step 4: Confirm deindexation. Verify using a site: search that the URL no longer appears in Google results. Check the URL Inspection tool for confirmation of exclusion.
Step 5 (optional): Re-add robots.txt block. After confirmed deindexation, the robots.txt disallow can be re-added to prevent further crawl resource consumption. The noindex directive should remain in place as the primary deindexing control. Be aware that if the robots.txt block prevents future crawls, Google will eventually be unable to confirm the noindex is still present, though it will not re-index a page it has already deindexed without a positive signal.
URL Removal Tool Limitations and Six-Month Expiration
Google’s URL Removal tool in Search Console hides URLs from search results for approximately six months. It does not permanently deindex them. The tool works by temporarily suppressing the URL’s appearance in results, not by removing it from Google’s index.
When the six-month suppression expires, the URL reappears in search results unless a permanent removal signal exists: a noindex directive, a 404/410 status code, or the content being genuinely removed. Using the URL Removal tool without implementing a permanent signal guarantees the problem recurs.
The tool’s legitimate use cases are narrow. It is appropriate for emergency removals (sensitive content that needs to disappear from results immediately while a permanent solution is deployed) and for confirming that a deindexation signal is being processed (if a noindex page is still appearing, the removal tool provides temporary relief while Google processes the directive).
Using the URL Removal tool as the sole deindexing method, which many practitioners do, creates a recurring maintenance burden. Every six months, the removal expires, the URL reappears, and the removal must be resubmitted. On sites with hundreds of URLs requiring removal, this manual cycle is unsustainable.
Status code alternatives: 404 and 410 as permanent deindexing signals
Returning a 404 (Not Found) or 410 (Gone) status code for pages that should be permanently deindexed is the most decisive removal method. Both signal to Google that the page no longer exists and should be removed from the index.
404 (Not Found) tells Google the page does not exist. Google processes the signal, reduces crawl frequency over time, and eventually removes the URL from the index. The timeline for full removal varies but typically completes within 4-8 weeks.
410 (Gone) provides a stronger signal that the page is permanently removed, not just temporarily missing. Google’s documentation and observed behavior confirm that 410 is processed faster than 404 for deindexing purposes. For content that will never return, 410 is the appropriate choice.
The advantage of status codes over noindex is that they do not require robots.txt coordination. A page returning 404 or 410 can be blocked in robots.txt simultaneously without conflict, because the status code is part of the HTTP response header that Googlebot receives even when it encounters the URL through other signals (backlinks, cached queue entries). However, for the fastest deindexation, allowing Googlebot to access the URL and receive the 404/410 directly produces the quickest results.
The decision framework: use 404/410 for pages that are permanently removed and whose URLs should never return. Use noindex for pages that should remain accessible to users (via direct URL or internal navigation) but should not appear in search results. Use the URL Removal tool only as a temporary emergency measure alongside a permanent solution.
Does the X-Robots-Tag HTTP header work for deindexing pages that are blocked by robots.txt?
The X-Robots-Tag is an HTTP response header, which means Googlebot must fetch the page to receive it. If robots.txt blocks the URL, Googlebot never makes the request, so it never sees the X-Robots-Tag header. The header is only effective for pages that Googlebot can access. To deindex a blocked page, first remove the robots.txt block, add the noindex directive (via meta tag or X-Robots-Tag), wait for Googlebot to crawl and process the directive, then optionally re-block the URL after deindexation is confirmed.
Does Google’s URL Removal tool work on pages that are not indexed but still appear in search results as title-only listings?
The URL Removal tool can suppress both fully indexed pages and title-only listings from search results. Title-only listings occur when Google knows a URL exists through links but has not crawled the content. The removal tool hides the URL from search results for approximately six months, regardless of indexation status. A permanent solution still requires a 404/410 response or a noindex directive that Googlebot can access, as the removal tool’s effect is temporary.
Does adding a noindex meta tag to a page that also has a canonical tag pointing elsewhere create a conflict?
This combination sends mixed signals. The canonical tag tells Google to consolidate signals onto the target URL, while the noindex tag tells Google not to index the current page. In practice, Google typically respects the noindex directive and drops the page from the index. However, the canonical tag may still cause Google to associate some signals with the canonical target. For clean deindexation, removing the canonical tag and relying solely on the noindex directive avoids ambiguity.
Sources
- Block Search Indexing with noindex — Google’s documentation on noindex implementation and the requirement that pages must be crawlable for noindex to work
- Robots.txt Introduction and Guide — Google’s documentation stating robots.txt is not a mechanism for keeping pages out of Google
- Google On Robots.txt: When to Use Noindex vs. Disallow — Summary of John Mueller’s September 2024 clarification on robots.txt vs. noindex separation
- Indexed Though Blocked by robots.txt — Google Search Central community thread explaining the blocked-but-indexed state and resolution workflow
