Steps

Enumerate the distinct AI crawler user agents relevant to commerce: retrieval/search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) versus training-only bots (GPTBot, ClaudeBot, CCBot, Google-Extended) versus shopping-specific bots (Amazonbot)
Check each publisher's current documentation for their crawler's exact user-agent string and stated purpose before writing rules, since bots and their behavior change over time
Write per-user-agent Allow/Disallow blocks in robots.txt that allow retrieval/shopping bots onto /products/ and /catalog/ paths while disallowing training-only bots from the same paths if desired
Confirm your CDN or edge platform is not silently overriding or stripping your robots.txt rules (a common cause of unexpected AI invisibility)
Cross-check the resulting access pattern against server logs to confirm bots are honoring the directives, since compliance is voluntary
Review the rule set quarterly, since AI companies regularly introduce new crawler identities

Known gotchas

robots.txt is advisory, not enforced -- pair it with Web Bot Auth or WAF-level verification for anything that actually needs to be blocked
Blocking a training crawler by name does nothing if that same operator also runs an unnamed or shared-infrastructure agent fetcher -- check for separate 'agent action' vs 'training' vs 'search' user agents from the same company
Allowing a bot in robots.txt does not exempt it from your rate limiting or WAF challenge rules -- the two systems must be configured consistently

developers.google.com · 6 steps · unrated

Configure per-engine crawl-delay directives in robots.txt, since Bing and Yandex honor it but Googlebot ignores it entirely

developers.google.com · 5 steps · unrated

Distinguish genuine AI crawler traffic (GPTBot, ClaudeBot, PerplexityBot, etc.) from spoofed user agents, and apply differentiated robots.txt directives

platform.openai.com · 6 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans

Configure robots.txt directives that allow AI shopping and search-agent crawlers (e.g. OAI-SearchBot, PerplexityBot, Amazonbot) onto product pages while blocking pure AI-training crawlers from the same paths

Steps

Known gotchas

Related routes

Give your agent this knowledge — and 15,500+ more routes

Need this verified for your stack — or a route we don't have yet?