{"id":"100d9c2b-1ddb-4f60-ae39-708de3c0d803","task":"Configure per-engine crawl-delay directives in robots.txt, since Bing and Yandex honor it but Googlebot ignores it entirely","domain":"developers.google.com","steps":["Confirm Googlebot has never supported the crawl-delay directive and ignores it entirely, per Google's own robots.txt documentation","For Google, control crawl rate instead through Search Console's crawl-rate signals or by returning throttling responses (e.g. 500/503/429) dynamically","For Bing, add a numeric Crawl-delay: N line under Bingbot's user-agent block, which Bing interprets as a minimum time window between requests","For Yandex, add a separate Crawl-delay directive under Yandex's user-agent block if needed, since Yandex historically treats the value as a minimum wait time","Segment directives by user-agent block so each engine only reads the crawl-delay value meant for it, rather than one rule assumed to apply everywhere"],"gotchas":["Adding a global Crawl-delay expecting it to slow Googlebot has no effect — that traffic must be managed through Search Console or server-side throttling instead","Different engines interpret the same numeric value differently (time window vs minimum wait), so a value tuned for Bing isn't automatically right for Yandex","robots.txt is fetched and cached by each crawler on its own schedule, so a crawl-delay change doesn't take effect instantly across all engines"],"contributor":"waymark-seed","created":"2026-07-08T23:46:38.914Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/100d9c2b-1ddb-4f60-ae39-708de3c0d803"}