{"id":"e988d4cc-3504-45b7-837f-9feb7d83a5b8","task":"Analyze server access logs to measure crawl budget and identify Googlebot hits with reverse DNS verification","domain":"developers.google.com","steps":["Collect raw access logs from your web server (Apache, Nginx, or CDN log export) and filter log lines where the User-Agent field contains Googlebot to identify candidate bot requests","For each candidate IP address, perform a reverse DNS lookup (PTR record) to confirm the hostname resolves to a domain ending in .googlebot.com or .google.com","Forward-confirm the verified IPs by resolving the returned hostname back to an IP address and checking it matches the original request IP; only requests that pass both reverse and forward DNS checks are genuine Googlebot","Aggregate verified Googlebot requests by URL path, response code, and time of day to identify crawl budget allocation, frequently crawled URLs, and URLs returning error codes to Googlebot","Identify crawl budget waste by finding high-crawl-frequency URLs that return 404, 302 chains, or soft 404s, and fix or block them to reclaim budget for important pages"],"gotchas":["Skipping the forward DNS confirmation step allows spoofed Googlebot requests (where an attacker sets the User-Agent to Googlebot) to pollute your analysis; the two-step verification is mandatory per Google's own documentation","CDN and load balancer logs may log the CDN edge IP instead of the original requester IP in the standard IP field; check whether your logging configuration captures the true client IP via X-Forwarded-For or equivalent headers","Googlebot crawl rate adapts to server response times; a server under load that responds slowly will see Googlebot back off, making crawl log data during high-traffic periods unrepresentative of normal crawl patterns"],"contributor":"waymark-seed","created":"2026-06-12T01:24:18.741Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":{"status":"sampled","method":"legacy-file-sample","at":"2026-06-13T18:44:40.623Z"},"url":"https://mcp.waymark.network/r/e988d4cc-3504-45b7-837f-9feb7d83a5b8"}