Verify special-case crawlers (AdsBot) and user-triggered fetchers (Site Verifier) using Google's separate special-crawlers.json and user-triggered-fetchers.json IP-range files
domain: developers.google.com · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗
Steps
Recognize Google publishes three separate crawler/fetcher categories, each with its own JSON file, rather than one unified Googlebot list
For standard organic crawling, match request IPs against common-crawlers.json (the current name for the file formerly called googlebot.json)
For special-purpose bots like AdsBot that may not obey robots.txt, match against special-crawlers.json instead, since they're excluded from the common list
For user-triggered fetches (e.g. Google Site Verifier, or RSS fetches from a GCP-hosted site), match against user-triggered-fetchers.json or user-triggered-fetchers-google.json, whose reverse-DNS hostnames differ from crawl-*.googlebot.com
Refresh all IP lists on a schedule rather than hardcoding a static snapshot, since Google updates the ranges periodically
Known gotchas
Treating all three JSON files as one 'Googlebot' list causes false negatives/positives — special-case and user-triggered fetchers use different hostnames and may ignore robots.txt entirely
The file location moved to https://developers.google.com/static/crawling/ipranges/ — old bookmarked paths can point to stale or missing files
IPs in the JSON files are CIDR ranges, not individual addresses; naive exact-string matching misses most legitimate requests
Give your agent this knowledge — and 15,500+ more routes
One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp
Need this verified for your stack — or a route we don't have yet?