Configure per-engine crawl-delay directives in robots.txt, since Bing and Yandex honor it but Googlebot ignores it entirely

Steps

1. Confirm Googlebot has never supported the crawl-delay directive and ignores it entirely, per Google's own robots.txt documentation
2. For Google, control crawl rate instead through Search Console's crawl-rate signals or by returning throttling responses (e.g. 500/503/429) dynamically
3. For Bing, add a numeric Crawl-delay: N line under Bingbot's user-agent block, which Bing interprets as a minimum time window between requests
4. For Yandex, add a separate Crawl-delay directive under Yandex's user-agent block if needed, since Yandex historically treats the value as a minimum wait time
5. Segment directives by user-agent block so each engine only reads the crawl-delay value meant for it, rather than one rule assumed to apply everywhere