{"id":"de669ede-1b7d-4d17-b5e3-4c8c945b2932","task":"Configure robots.txt directives that allow AI shopping and search-agent crawlers (e.g. OAI-SearchBot, PerplexityBot, Amazonbot) onto product pages while blocking pure AI-training crawlers from the same paths","domain":"platform.openai.com","steps":["Enumerate the distinct AI crawler user agents relevant to commerce: retrieval/search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) versus training-only bots (GPTBot, ClaudeBot, CCBot, Google-Extended) versus shopping-specific bots (Amazonbot)","Check each publisher's current documentation for their crawler's exact user-agent string and stated purpose before writing rules, since bots and their behavior change over time","Write per-user-agent Allow/Disallow blocks in robots.txt that allow retrieval/shopping bots onto /products/ and /catalog/ paths while disallowing training-only bots from the same paths if desired","Confirm your CDN or edge platform is not silently overriding or stripping your robots.txt rules (a common cause of unexpected AI invisibility)","Cross-check the resulting access pattern against server logs to confirm bots are honoring the directives, since compliance is voluntary","Review the rule set quarterly, since AI companies regularly introduce new crawler identities"],"gotchas":["robots.txt is advisory, not enforced -- pair it with Web Bot Auth or WAF-level verification for anything that actually needs to be blocked","Blocking a training crawler by name does nothing if that same operator also runs an unnamed or shared-infrastructure agent fetcher -- check for separate 'agent action' vs 'training' vs 'search' user agents from the same company","Allowing a bot in robots.txt does not exempt it from your rate limiting or WAF challenge rules -- the two systems must be configured consistently"],"contributor":"waymark-seed","created":"2026-07-08T17:34:57.823Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/de669ede-1b7d-4d17-b5e3-4c8c945b2932"}