The question publishers keep asking is how to block AI systems from training on their content while still appearing in AI-generated search answers. As of 2026, approximately 5.6 million websites block OpenAI’s GPTBot and 5.8 million block ClaudeBot, a 70% increase from early July 2025. But the belief that you can selectively block training while preserving citation visibility has become more nuanced than the original binary framing suggested. AI companies now operate multiple crawlers with distinct purposes, and the distinction between training crawlers and retrieval crawlers has created a selective blocking strategy that did not exist when the first wave of blocking decisions was made.
AI crawler blocking removes content from both training pipelines and retrieval indices simultaneously when applied indiscriminately
The misconception originates from treating “AI crawler” as a single category. In the early phase of AI crawler blocking (2023-2024), most publishers added blanket blocks against GPTBot, ClaudeBot, and Google-Extended in robots.txt. These directives blocked the same crawl infrastructure that feeds both training data collection and real-time retrieval indices. The result was complete removal from both pipelines.
The technical architecture that makes blanket blocking problematic is straightforward. For a retrieval-augmented generation system to cite your content in a real-time response, it must first retrieve your content, which requires crawling and indexing it. Blocking the crawler prevents the crawl, which prevents indexing, which prevents retrieval, which prevents citation. The chain from crawl access to citation is linear, and blocking at any point breaks the entire sequence.
BuzzStream’s study found that 79% of top news sites block AI training bots via robots.txt. More critically, 71% of sites also block AI retrieval bots. This means the majority of blocking publishers have inadvertently removed themselves from both training data and real-time AI search citation, achieving maximum content protection at the cost of complete AI search invisibility.
The crawl-to-referral imbalance data explains why many publishers accept this trade-off. OpenAI’s crawl-to-referral ratio sits at approximately 1,700:1 as of June 2025, meaning for every click OpenAI sends to a publisher, its crawlers consume roughly 1,700 pages. Anthropic’s ratio is even more extreme at 73,000:1. From a cost-benefit perspective, the traffic returned by AI search systems is negligible compared to the content consumed, which justifies blocking for publishers who prioritize content protection over AI visibility.
The crawl landscape has evolved: training crawlers and retrieval crawlers are now separable
The original binary framing, block everything or allow everything, no longer accurately describes the available options. AI companies have separated their crawling infrastructure into distinct user agents with different purposes, enabling selective blocking that was not possible in 2024.
OpenAI operates GPTBot for training data collection and ChatGPT-User (also called OAI-SearchBot) for real-time browsing and retrieval. Blocking GPTBot prevents training data inclusion while allowing ChatGPT-User permits content to appear in ChatGPT’s real-time search answers. This separation is the most widely adopted selective blocking pattern in 2026.
Anthropic now operates ClaudeBot for training data collection, Claude-User for real-time page fetching when users ask questions, and Claude-SearchBot for indexing content for search results. Each has its own robots.txt user-agent string, enabling granular control. Blocking ClaudeBot while allowing Claude-SearchBot prevents training data contribution while maintaining search result visibility.
Google’s approach differs. Google-Extended controls whether content is used for Gemini model training and improvement, but Google has stated that blocking Google-Extended does not affect search rankings or inclusion in AI Overviews. This means Google AI Overviews draw from the standard Googlebot index rather than from a separate AI-specific crawl, and blocking Google-Extended has no visibility impact for Google’s AI search features.
# Selective blocking: block training, allow retrieval
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: ChatGPT-User
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: Claude-SearchBot
Allow: /Selective blocking strategies are practical but come with limitations
The selective blocking approach, blocking training crawlers while allowing retrieval crawlers, is technically viable and represents the current best practice for publishers who want to participate in AI search without contributing to training datasets. But the strategy has practical limitations that prevent it from being a clean solution.
First, the separation between training and retrieval is maintained by AI companies’ own infrastructure design, not by any technical standard or protocol. Nothing prevents an AI company from changing which user agent handles which function, or from using retrieval-crawled content for training purposes. The trust boundary is policy-based, not technically enforced.
Second, compliance with robots.txt directives is voluntary and inconsistently observed. Tollbit reported that 13.26% of AI bot requests ignored robots.txt directives in Q2 2025, up from 3.3% in Q4 2024. Some publishers have moved to server-level blocking using user-agent detection and IP range blocking for more reliable enforcement, but this requires more technical infrastructure than robots.txt.
Third, selective blocking produces inconsistent brand representation across AI systems. If your content enters one AI system’s retrieval index but not another’s, your brand appears in some AI search results and not others. This inconsistency can confuse enterprise marketing teams tracking AI visibility and makes performance measurement more complex.
Fourth, the long-term strategic implications of blocking training while allowing retrieval are unclear. A brand that blocks training crawlers contributes no content to future parametric knowledge, meaning the LLM’s baseline understanding of that brand erodes over successive training cycles. The brand becomes entirely dependent on real-time retrieval for AI visibility, which may produce less consistent or authoritative responses than brands with strong parametric representation.
The actual decision: full participation, selective participation, or full protection, each with distinct business consequences
The strategic decision is no longer strictly binary, but it remains a consequential choice between three options with distinct business consequences.
Full participation means allowing all AI crawlers, including training crawlers, to access your content. Your content enters training datasets, building parametric knowledge that makes the LLM more likely to recommend your brand even without retrieval. Your content appears in real-time AI search results. The cost is that your content is used to train models that may reduce your organic search traffic. This option favors brands where AI visibility and recommendation frequency drive significant business value.
Selective participation means blocking training crawlers while allowing retrieval crawlers. Your content does not enter future training datasets, providing some content protection. Your content can appear in real-time AI search answers through retrieval. Parametric brand knowledge degrades over time as new training runs exclude your content. This is the most popular approach in 2026, balancing content protection with visibility.
Full protection means blocking all AI crawlers. Your content does not enter training data or retrieval indices. You are invisible in AI-generated responses. Your content is fully protected from AI usage. This option suits publishers whose business model depends on direct content consumption and who cannot afford content being summarized by AI systems.
The revenue impact calculation for each option involves estimating three values: the traffic and conversion value of AI search citations (currently small but growing), the brand value of being recommended in AI responses (significant for consideration-stage queries), and the content protection value of preventing AI systems from using your content (difficult to quantify but meaningful for publishers). The option that maximizes total value varies by business model, with consumer brands favoring participation and premium publishers favoring protection.
Does blocking Google-Extended affect whether content appears in Google AI Overviews?
No. Google has stated that blocking Google-Extended prevents content from being used for Gemini model training and improvement but does not affect search rankings or inclusion in AI Overviews. AI Overviews draw from the standard Googlebot index, which is controlled by the regular Googlebot user-agent directive. Blocking Google-Extended is a low-risk decision for publishers who want to prevent Gemini training data contribution without sacrificing Google AI search visibility.
What percentage of AI crawler traffic actually respects robots.txt blocking directives?
Compliance is declining. Tollbit reported that 13.26% of AI bot requests ignored robots.txt directives in Q2 2025, up from 3.3% in Q4 2024. This means robots.txt-based blocking is not fully reliable. Publishers requiring stricter enforcement are implementing server-level blocking using user-agent detection and IP range blocking, which provides more reliable control but requires more technical infrastructure to maintain as AI companies periodically update their crawler IP ranges.
Is the selective blocking strategy (block training, allow retrieval) sustainable long-term as AI systems evolve?
The sustainability depends on AI companies maintaining the separation between training and retrieval crawlers, which is a policy decision rather than a technical guarantee. Nothing prevents a provider from merging crawler functions or using retrieval-crawled content for training. Additionally, brands that block training crawlers see their parametric representation erode over successive training cycles, becoming entirely dependent on real-time retrieval for AI visibility. This dependency creates fragility if retrieval systems change or if the brand’s content does not rank well in retrieval indices.
Sources
- BuzzStream: Which News Sites Block AI Crawlers in 2025 — Data on 79% of top news sites blocking AI training bots and 71% blocking retrieval bots
- Playwire: The Real Cost of Blocking AI, Traffic and Revenue Impact Analysis — Crawl-to-referral ratio data showing 1,700:1 for OpenAI and 73,000:1 for Anthropic
- Search Engine Journal: Anthropic’s Claude Bots Make Robots.txt Decisions More Granular — Documentation of separate training, user, and search crawler user agents
