Storing the complete raw HTML of every crawled page, rather than extracting and storing just the structured attributes actually needed for analysis (title tag, status code, canonical URL, word count, header tags, and similar fields), multiplies storage costs substantially and slows down nearly every downstream query, because each row in the warehouse becomes many times larger than it needs to be, and any query has to contend with that bulk even when it only needs a handful of fields. This is a standard data-engineering problem, not something specific to SEO tooling; it’s the general cost of skipping the extract-transform step and storing raw source data as the primary queryable format.
Why this happens
Storage cost scales with what you keep, not with what you use. A full HTML snapshot of a page, including all markup, inline scripts, comments, and boilerplate that has no analytical value, can easily run tens to hundreds of kilobytes per page, while the structured attributes actually used in most technical SEO analysis (a title string, a status code, a canonical URL, a handful of counts and flags) amount to a tiny fraction of that size. At the scale of millions of crawled URLs, the difference between storing full HTML and storing extracted attributes compounds directly into storage cost, since cost is generally proportional to bytes stored, and full HTML is orders of magnitude larger per row than its structured extract.
Query performance degrades because engines have to process larger rows even for narrow queries. Most analytical data warehouses (columnar systems in particular) are efficient at scanning only the columns a query actually needs, but this efficiency assumes the data is already structured into discrete columns. If the primary stored artifact is a single large HTML blob column, any query that needs to derive information from that HTML (even something as simple as “what was the title tag”) has to parse or full-text-scan that blob at query time, rather than reading a small, purpose-built column. This turns lightweight analytical queries into effectively re-parsing raw source data on every run, which is slow and resource-intensive compared to reading a pre-extracted field directly.
Repeated re-parsing multiplies compute cost on top of storage cost. If structured attributes aren’t extracted once at crawl time and stored directly, every subsequent analysis that needs those attributes has to re-derive them from the raw HTML, meaning the same parsing work gets repeated across every query or report that touches that data, rather than being done once and reused. This compounds the performance problem beyond simple row-size bloat.
Full HTML retention also complicates schema evolution and consistency. Structured attribute extraction can be versioned and validated at ingestion time (confirming the title-extraction logic worked, flagging missing canonicals), while raw HTML pushes that validation responsibility downstream to whoever eventually queries it, increasing the chance that inconsistent parsing logic across different analyses produces subtly different results from the same underlying source.
The practical pattern: extract-then-store
The standard fix, consistent with general extract-transform-load (ETL) principles applied to a crawl data pipeline, is to extract the structured attributes needed for analysis at crawl time and store those as the primary queryable dataset, rather than deferring extraction to query time. Specific practical elements:
- Extract core structured fields immediately at crawl time: status code, title, meta description, canonical URL, indexability signals, header tag structure, word count, and whatever other attributes the analysis actually requires, storing these as discrete, well-typed columns.
- Keep raw HTML only as short-retention cold storage, if at all. If there’s a genuine need to reprocess pages later (a new attribute becomes relevant that wasn’t originally extracted), retaining raw HTML in cheaper, less frequently accessed cold storage for a limited retention window is a reasonable compromise, rather than keeping full HTML indefinitely in the primary, frequently-queried warehouse tables.
- Design the extraction schema to anticipate reasonably likely future needs, since the main cost of not storing raw HTML long-term is the inability to retroactively extract an attribute nobody thought to capture; a thoughtfully broad initial extraction schema reduces how often this becomes a real limitation.
This pattern (extract-then-store, cold-storage or discard the raw source) is a foundational data-warehouse design practice, and applying it to crawl data specifically avoids the storage-cost multiplication and query-performance drag that comes from treating raw HTML as the primary analytical format rather than an intermediate input.
Additional structural considerations
Partitioning and clustering compound the benefit of extraction, rather than substituting for it. Extraction solves the row-size problem; a separate schema-design layer, partitioning and clustering strategy, solves the problem of scanning more rows than a given query actually needs, and that layer is worth its own dedicated treatment rather than a quick mention here.
Change-detection use cases are a common reason teams reach for full HTML retention, and there’s a narrower way to serve that need. A frequent justification for keeping full HTML is wanting to detect exactly what changed between crawls (not just that the title changed, but the specific before/after text). This is a legitimate need, but it can usually be served by storing a content hash or a diff of specific extracted sections rather than the entire raw page, which captures the meaningful change-detection signal at a small fraction of the storage cost full HTML retention would require.
Rendering considerations add a wrinkle for JavaScript-heavy sites. For crawls that include a rendering step (to capture content that only appears after client-side JavaScript execution), the temptation to store the full rendered DOM alongside the raw HTML compounds the same storage problem twice over. The same extract-then-store principle applies: pull the structured attributes needed from the rendered output at crawl time, and avoid persisting the full rendered DOM as a matter of course, reserving it only for cases where a specific, narrow debugging or reprocessing need justifies the extra storage cost.
A hypothetical illustration
As a hypothetical illustration: suppose a large B2B directory site, hypothetically called Meridian Trade Index, crawls its own 8 million listing pages weekly and stores the complete raw HTML of every page in its BigQuery warehouse, including full rendered DOM snapshots for its JavaScript-heavy listing template. Hypothetically, after a year of weekly crawls, the warehouse has ballooned to several hundred terabytes, and a simple query asking “which pages are missing a canonical tag” takes many minutes to run, because the query engine has to parse HTML text out of a giant blob column for every one of the 8 million rows rather than reading a small, pre-extracted boolean flag. Suppose Meridian’s engineering team rebuilds the pipeline to extract structured fields, status code, title, canonical URL, word count, indexability flags, at crawl time, storing only those as the primary queryable table, while retaining raw HTML in cheap cold storage for just 30 days. In this hypothetical, the same canonical-tag query that previously took minutes now runs in seconds against the small, well-typed extracted table, and total storage cost drops by an order of magnitude, since the vast majority of what had been stored was boilerplate markup nobody was ever querying directly.
