Search Console distinguishes between “Discovered – currently not indexed” and “Crawled – currently not indexed,” and that distinction contains the diagnosis. “Discovered” means Google knows the URL exists but has not allocated crawl resources to fetch it, indicating crawl budget exhaustion. “Crawled – currently not indexed” means Google fetched and rendered the content, then decided after evaluation that the page did not merit indexation, indicating quality-based filtering. Cross-referencing server logs with Search Console coverage status for non-indexed programmatic URLs reveals the ratio between these two causes. In the majority of large programmatic deployments analyzed, quality filtering accounts for the dominant share. The remediation strategies are opposite: crawl budget problems require technical optimization of crawl efficiency, while quality filtering requires content improvement on the pages Google already evaluated and rejected.
The Diagnostic Separation Between Crawl Budget and Quality Filtering
Crawl budget exhaustion and quality-based deindexation produce the same visible symptom, pages not in the index, but leave different diagnostic traces in your data.
Crawl budget issues manifest as pages never being crawled. Server log analysis reveals that Googlebot has not visited these URLs at all, or has visited them only once many months ago. The Search Console status for these pages shows “Discovered – currently not indexed,” meaning Google knows the URLs exist (through sitemaps or internal links) but has not allocated crawl resources to fetch them.
Quality filtering manifests differently. The pages were crawled, Google fetched and rendered the content, but decided after evaluation that the pages did not meet indexation quality thresholds. The Search Console status shows “Crawled – currently not indexed,” confirming that Google invested crawl resources but declined to add the pages to the index.
The first diagnostic step correlates server log crawl data with Search Console coverage status. Export the list of non-indexed programmatic URLs from Search Console. Cross-reference against server logs to determine whether Googlebot visited each URL within the past 90 days. URLs with recent Googlebot visits but “Crawled – currently not indexed” status are quality-filtered. URLs with no recent Googlebot visits and “Discovered – currently not indexed” status are crawl-budget-constrained. The ratio between these two groups determines the primary remediation path. [Observed]
The “Crawled – Currently Not Indexed” Pattern Analysis
When Search Console shows a growing population of “Crawled – currently not indexed” pages concentrated in programmatic URL patterns, quality filtering is the primary diagnosis. The pattern analysis determines the severity and identifies the specific quality factors driving the filtering.
Track the growth rate of this status category over time by exporting index coverage data weekly. A steady increase in “Crawled – currently not indexed” pages indicates active quality filtering where Google is crawling new pages and rejecting them. A stable count indicates that Google filtered a batch of pages and has stopped re-evaluating them. A declining count (pages moving from “Crawled – currently not indexed” to “Valid”) indicates quality improvement being recognized.
Correlate the growth rate with template changes or data source updates. If the “Crawled – currently not indexed” count spiked after a specific template modification or data source change, the temporal correlation identifies the likely cause. If the count has been growing steadily since the programmatic section launched, the original template design is the constraint.
Segment by URL pattern to determine whether specific programmatic page types are more affected. If pages targeting low-volume queries are disproportionately in “Crawled – currently not indexed” while pages targeting higher-volume queries are indexed, Google may be applying a demand-based quality threshold where pages without search demand fail a different quality criterion. If all page types are equally affected, the template-level quality is the binding constraint. [Observed]
The Indexation Ratio Benchmark Test
The indexation ratio, indexed pages divided by crawled pages, is the key diagnostic metric. A healthy programmatic page set maintains an indexation ratio above 80%. This ratio isolates quality filtering from crawl budget issues by focusing only on pages Google has actually crawled and evaluated.
Calculate the ratio by dividing the number of programmatic pages in “Valid” (indexed) status by the sum of “Valid” plus “Crawled – currently not indexed” status. Exclude “Discovered – currently not indexed” pages from this calculation because they represent crawl budget constraints, not quality decisions.
Benchmarks by programmatic page type provide context for interpretation. Directory pages listing businesses or services: healthy ratio is 75-90%. Comparison pages with substantive content: healthy ratio is 85-95%. Data-only pages with minimal contextual content: healthy ratio is 40-60%. If your ratio falls below the benchmark for your page type, quality filtering is confirmed as active and above the expected baseline for your content depth.
Segmenting the ratio by template variant identifies which specific templates trigger quality-based rejection. If Template A produces an indexation ratio of 85% while Template B produces 35%, Template B’s design is the primary quality problem. This segmentation prevents wasted effort on improving Template A when Template B is the binding constraint. [Reasoned]
Confirming the Diagnosis With Controlled Re-Submission Tests
The definitive diagnostic confirmation involves improving content quality on a small test cohort of non-indexed pages and requesting re-indexation through Search Console’s URL inspection tool. If re-submission with improved content results in indexation while re-submission without changes does not, quality filtering is confirmed as the root cause.
The test design requires two cohorts of 25-50 non-indexed pages each. The test cohort receives content quality improvements: additional unique content blocks, contextual interpretation of data, expanded data fields, or improved template formatting. The control cohort receives no changes. Request re-indexation for all pages in both cohorts through the URL inspection tool.
The sample size of 25-50 pages per cohort provides sufficient signal for a binary outcome (indexed vs. not indexed) while remaining manageable for manual content improvement. Smaller samples risk inconclusive results from individual page variability. Larger samples are unnecessary for diagnostic purposes (though they provide more statistical confidence).
The specific content improvements most likely to flip pages from filtered to indexed include: adding 200-300 words of contextual content that interprets the page’s data, adding conditional sections that vary based on data characteristics, improving the unique content ratio above 30%, and adding structured data markup that enhances the page’s information presentation. Monitor indexation status for both cohorts over four weeks. If the improved cohort achieves 60%+ indexation while the control remains under 20%, quality filtering is confirmed with high confidence. [Reasoned]
How do you distinguish between quality-based deindexation and a Google algorithm update affecting programmatic pages?
Algorithm updates produce sudden, correlated ranking drops across many pages within a narrow time window, typically aligning with confirmed Google update dates. Quality-based deindexation is gradual, with the “Crawled – currently not indexed” count growing steadily over weeks or months. Check Google’s Search Status Dashboard for confirmed update dates and correlate against your index coverage timeline. If the deindexation trend predates or persists independently of update dates, quality filtering rather than an algorithm change is the primary cause.
Should you request re-indexation through Search Console for all deindexed programmatic pages simultaneously?
No. Mass re-indexation requests without content improvements are ineffective because Google will re-evaluate and re-reject pages that failed quality thresholds. Instead, improve content quality on a test batch of 25-50 pages first, request re-indexation for that batch, and measure the indexation success rate over four weeks. Only after confirming that the improvements produce indexation should the changes be rolled out across the full page set with staggered re-indexation requests.
Can moving programmatic pages to a subdomain help recover from site-wide quality contamination caused by deindexed pages?
Migrating programmatic pages to a subdomain isolates their quality signals from the main domain, which can halt cross-section quality drag on editorial content. However, the subdomain starts with reduced domain authority inherited from the parent, and the programmatic pages still need to pass quality thresholds independently. The migration is a containment strategy that protects editorial rankings while programmatic content is improved, not a fix for the underlying quality problems causing deindexation.
