Design a multi-level-safe sitemap index structure for a site with far more URLs than one sitemap index file can reference
domain: developers.google.com · 6 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Split URLs into individual sitemap files respecting the 50,000-URL and 50MB (uncompressed) per-sitemap limits
Group those sitemap files under a sitemap index file, which itself is capped at 50,000 <sitemap> entries and 50MB uncompressed
For sites large enough to exceed 50,000 sitemaps (i.e. potentially billions of URLs), create multiple separate root sitemap index files rather than nesting one index inside another, since search engines do not support multi-level index nesting
Submit each root sitemap index separately via the Search Console Sitemaps API or robots.txt Sitemap: directive
Gzip each sitemap file to reduce transfer size and regenerate/replace files atomically so crawlers never fetch a half-written file
Track per-sitemap submitted vs. indexed counts in Search Console's Sitemaps report to spot sitemaps Google is ignoring or throttling
Known gotchas
A sitemap index referencing other sitemap index files (rather than plain sitemaps) is invalid per the sitemap protocol and may be silently ignored
lastmod values that are inaccurate or bumped on every regeneration train Google to distrust your freshness signal and can reduce recrawl priority
Very large numbers of low-value or near-duplicate URLs in sitemaps can waste crawl budget rather than help it, even if the sitemap technically validates
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?