Diagnose crawl budget waste by correlating server access logs with Googlebot reverse DNS verification

Steps

1. Export raw access logs from your web server or CDN, filtering lines where the user-agent string contains 'Googlebot'
2. For each candidate IP address, perform a reverse DNS lookup to confirm the hostname ends in .googlebot.com or .google.com, then do a forward DNS lookup on that hostname to confirm it resolves back to the original IP
3. Categorize crawled URLs by type (canonical pages, parameter variants, internal search results, faceted navigation URLs) to identify which URL classes consume disproportionate crawl share
4. Overlay the log-derived crawl time series with the Search Console Crawl Stats report to confirm alignment before making infrastructure changes
5. Prioritize reducing crawl waste sources: consolidate parameter variants with canonical tags, block dead-end URL patterns in robots.txt, and reduce redirect chains