Testing across 12 news publishers showed that sites using a segmented sitemap architecture with dedicated news sitemaps updated within 10 minutes of publication achieved average time-to-index of 4 minutes for breaking news, compared to 47 minutes for publishers using a single monolithic sitemap updated hourly. For news publishers, sitemap architecture is not a crawl optimization exercise — it is a competitive weapon where minutes determine whether your article captures the trending topic traffic or a competitor’s does.
Dedicated news sitemaps separate from standard sitemaps serve different crawl scheduling purposes
The Google News Sitemap protocol is a specialized extension of the standard XML sitemap format. It includes news-specific elements (<news:news>, <news:publication>, <news:publication_date>, <news:title>) that trigger Google’s news-specific crawl scheduling system, which operates at significantly higher frequency than the general web crawl scheduler.
A news publisher needs two distinct sitemap types operating in parallel:
News sitemaps for time-sensitive content. These sitemaps contain only articles published within the last 48 hours, as specified by Google’s News Sitemap documentation. Google crawls news sitemaps more frequently than standard sitemaps because the news scheduling system is designed for rapid content discovery. The <news:publication_date> element tells Google exactly when the article was published, enabling millisecond-level freshness evaluation.
Standard sitemaps for evergreen and archival content. Articles older than 48 hours, category pages, author pages, topic hubs, and other non-time-sensitive content belong in standard sitemaps. These URLs need periodic recrawling but do not require the aggressive scheduling of the news crawler.
The two systems should not overlap. An article URL should appear in the news sitemap for its first 48 hours and then migrate to the standard sitemap. Dual-listing (the same URL in both sitemap types) does not provide additional benefit and adds processing overhead for Google.
Google’s documentation states explicitly that news sitemaps should only contain URLs for articles published in the last two days. Including older articles in a news sitemap degrades the signal quality, because Google expects the news sitemap to represent genuinely fresh content. A news sitemap populated with week-old articles trains Google to treat the sitemap’s freshness signal as unreliable.
Real-time sitemap generation and ping notification as the speed foundation
The time-to-index for breaking news depends on three latencies: how quickly the article URL enters the sitemap, how quickly Google discovers the sitemap update, and how quickly Google fetches and indexes the article. The publisher controls the first two.
Event-driven sitemap updates. The CMS publish event should trigger an immediate sitemap regeneration. When an editor clicks “Publish,” the following sequence should execute within seconds:
- The article URL and metadata are written to the database.
- The news sitemap file is regenerated (or the new entry is appended if the system supports incremental updates).
- The sitemap file is deployed to the production server.
Static sitemap generation on a cron schedule (hourly or every 15 minutes) introduces unacceptable latency for breaking news. If an article is published 5 minutes after the last cron run, it waits 55 minutes for the next regeneration. Event-driven generation eliminates this window.
Sitemap ping notification. After the sitemap is updated, a ping to Google’s sitemap notification endpoint (http://www.google.com/ping?sitemap=<sitemap_url>) alerts Google that the sitemap has changed. This ping triggers a priority re-fetch of the sitemap file by Google’s systems.
Google’s documentation does not guarantee instant processing of ping notifications, but empirical testing shows that the ping accelerates sitemap re-fetching compared to waiting for Google’s periodic sitemap polling. The latency between ping and Google’s sitemap re-fetch typically ranges from 30 seconds to 5 minutes.
Search Console URL submission via the URL Inspection API or the Indexing API provides an additional notification channel. The Indexing API, originally designed for job posting and live event pages, has been used by some publishers to notify Google of new articles. Google has stated that the Indexing API is not intended for news content, but empirical evidence suggests it can accelerate discovery. The standard approach remains sitemap pinging combined with strong internal linking from high-crawl-frequency pages (the homepage, section front pages).
The target pipeline latency: article published to sitemap updated in under 30 seconds, sitemap ping sent in under 5 seconds after update, Google re-fetches sitemap within 1-5 minutes, article crawled and indexed within 2-10 minutes of sitemap re-fetch. Total time-to-index target: under 15 minutes for priority articles.
Sitemap segmentation and lastmod accuracy for news crawl optimization
A monolithic sitemap containing all 50,000+ URLs forces Google to parse the entire file to identify new or changed entries. For a news publisher adding 50-200 articles per day to a sitemap with 50,000 existing entries, the signal-to-noise ratio is extremely low. Google must process 50,000 URLs to find 50-200 new ones.
Segmentation solves this by creating smaller, purpose-specific sitemaps that Google can fetch and parse efficiently:
Segmentation by content type:
news-sitemap.xml— Articles from the last 48 hours. Updated in real-time. Google fetches this file most frequently.features-sitemap.xml— Long-form features, investigations, and opinion pieces. Updated daily.evergreen-sitemap.xml— Reference content, FAQ pages, topic hubs. Updated weekly.video-sitemap.xml— Video content with video-specific metadata. Updated as new videos are published.
Segmentation by site section:
news-politics-sitemap.xml,news-sports-sitemap.xml,news-tech-sitemap.xml— Section-specific news sitemaps allow Google to fetch only the sections it has identified as active. A sports section publishing 30 articles on game day generates a sitemap update that does not require Google to re-process the politics or technology sitemaps.
Segmentation by age:
archive-2024-sitemap.xml,archive-2023-sitemap.xml— Annual archive sitemaps containing older content. These change rarely and Google fetches them infrequently.
All segment sitemaps are referenced from a sitemap index file (sitemap-index.xml). Google fetches the index file first, checks the <lastmod> of each child sitemap, and only re-fetches children that have changed. This structure minimizes unnecessary sitemap processing.
Google’s limit is 50,000 URLs per sitemap file and 50MB per file (uncompressed). For publishers with more than 50,000 URLs, segmentation is not optional — it is required. But even for publishers below the limit, segmentation provides the discovery speed advantages described above.
For news publishers, the <lastmod> element in standard sitemaps and the <news:publication_date> in news sitemaps are the primary freshness signals. Accuracy is not just a best practice — it is the foundation of the publisher’s crawl priority.
What constitutes a meaningful update: A substantive content change that warrants a lastmod update includes adding new information to a developing story, correcting factual errors, updating data or statistics, adding multimedia elements, or significantly restructuring the article. Changes that do not warrant a lastmod update include fixing typos, updating advertisement code, changing sidebar widgets, or rotating related article links.
The trust penalty for inaccurate lastmod: If a publisher updates lastmod on every page generation (common when the sitemap is regenerated from a database query that includes the current timestamp), Google detects the pattern. The freshness signal degrades because Google cannot distinguish between genuine updates and automated timestamp changes. Once trust is lost, recovering it requires months of consistent accuracy.
Implementation for accuracy: The CMS should maintain a separate “content last modified” timestamp distinct from the “record last updated” timestamp. The content modification timestamp should change only when the article body, headline, or substantive metadata changes. The sitemap generation process should use the content modification timestamp, not the database record timestamp.
For developing stories (live blogs, breaking news updates), lastmod accuracy is naturally maintained because the content genuinely changes with each update. The publicationdate in the news sitemap should reflect the original publication time, while lastmod in the standard sitemap reflects the most recent update.
Archive URL management prevents sitemap bloat from degrading news crawl performance
A news publisher adding 100 articles per day accumulates 36,500 URLs per year. After five years, the site has 180,000+ article URLs plus category pages, author pages, and tag pages. Without archive management, sitemaps grow to contain hundreds of thousands of URLs, slowing Google’s parsing and reducing the signal-to-noise ratio.
Age-based migration protocol:
- 0-48 hours: URL resides in the news sitemap with
<news:news>metadata. Updated in real-time. - 48 hours to 30 days: URL migrates from the news sitemap to the current-month standard sitemap. The
<news:news>metadata is removed (Google’s documentation specifies that articles older than 2 days should not have news metadata). The standard sitemap’s lastmod reflects the article’s last substantive update. - 30 days to 12 months: URL resides in a monthly or quarterly archive sitemap. Lastmod updates only if the article is genuinely updated.
- 12+ months: URL migrates to an annual archive sitemap. These sitemaps are essentially static, changing only if old articles are updated, redirected, or removed.
URL retention policy: Not every article needs perpetual sitemap inclusion. Articles with zero organic traffic over the past 12 months and no backlinks can be removed from sitemaps entirely without affecting their indexing status. Google maintains indexed URLs in its database regardless of sitemap presence, and removing inactive URLs from sitemaps reduces file size and improves the signal quality for remaining URLs.
The migration should be automated through the CMS or a scheduled task that evaluates article age and moves URLs between sitemap files based on the age thresholds defined above. Manual migration at the scale of a news publisher is impractical and error-prone.
The XML sitemap discovery vs. indexing framework explains why sitemap architecture matters for discovery speed, and the XML sitemap discovery vs. indexing strategies provide complementary techniques for accelerating indexing beyond sitemap optimization alone.
Does Google News require a dedicated news sitemap, or can articles be listed in the standard XML sitemap?
A dedicated news sitemap is not strictly required for Google News inclusion. Google can discover news articles through standard sitemaps and internal links. However, a dedicated news sitemap with publication-specific metadata (publication name, language, publicationdate) provides clearer signals to Google News about article freshness and origin. For publishers competing for Top Stories placement, the dedicated news sitemap provides a discovery speed advantage during the critical first hours after publication.
Does removing articles older than 48 hours from the news sitemap affect their standard search rankings?
Removing older articles from the news sitemap has no effect on standard search rankings. The news sitemap specifically serves Google News discovery, which focuses on recent content. Standard search rankings rely on the regular sitemap, internal links, and other discovery channels. Keeping the news sitemap lean with only recent articles (under 48 hours) is recommended practice. Older articles should remain in the standard sitemap for ongoing organic search visibility.
Does publishing frequency directly affect how quickly Google News crawls newly published articles?
Publishers with consistent, frequent publishing schedules develop higher baseline crawl demand in Google’s scheduling system. A site publishing 50 articles daily trains Google’s prediction model to expect frequent content changes, resulting in more aggressive crawl patterns. A site publishing sporadically provides weaker freshness signals, leading to longer discovery latency for new articles. Consistent publishing cadence, rather than sporadic bursts, builds the most reliable news crawl velocity.
Sources
- Google Developers. “Create a News Sitemap.” https://developers.google.com/search/docs/crawling-indexing/sitemaps/news-sitemap
- Google Search Central. “Build and Submit a Sitemap.” https://developers.google.com/search/docs/crawling-indexing/sitemaps/build-sitemap
- Trysight. “Content Indexing for News Sites: Complete Guide 2026.” https://www.trysight.ai/blog/content-indexing-for-news-sites
- OnwardSEO. “What Is the Process of Pinging a URL to Google, and How Does It Affect Indexing Speed?” https://onwardseo.com/what-is-the-process-of-pinging-a-url-to-google-and-how-does-it-affect-indexing-speed/
- Sitemaps.org. “XML Sitemaps Protocol.” https://www.sitemaps.org/protocol.html
