{"id":"a378e11a-066e-432b-9534-fd0c7cffbda6","task":"Distinguish genuine AI crawler traffic (GPTBot, ClaudeBot, PerplexityBot, etc.) from spoofed user agents, and apply differentiated robots.txt directives","domain":"platform.openai.com","steps":["Enumerate the specific bots to control separately: e.g. GPTBot and OAI-SearchBot from OpenAI, ClaudeBot/Claude-User/Claude-SearchBot from Anthropic, PerplexityBot/Perplexity-User, Google-Extended, and CCBot, since each has a distinct purpose (training vs. retrieval/search vs. user-triggered fetch)","Check each vendor's published documentation for verification guidance (e.g. published IP ranges or reverse-DNS verification patterns), analogous to how Googlebot is verified, rather than trusting the User-Agent string alone","For claimed bot hits, perform a reverse DNS lookup on the source IP and forward-confirm it resolves back to the expected hostname/IP range before trusting the identity","Write distinct robots.txt groups per bot (or bot family) so you can allow retrieval/search bots on public content while disallowing training-only bots, rather than one blanket AI rule","For crawlers documented as sometimes ignoring robots.txt, add server-level (WAF/CDN) blocking as a backstop rather than relying on robots.txt alone","Re-audit the bot list periodically, since AI vendors have split or renamed user agents before (e.g. separating a search/retrieval bot from a training bot under the same company)"],"gotchas":["A User-Agent string claiming to be GPTBot or ClaudeBot proves nothing on its own; anyone can send that header, so traffic claiming AI-bot identity should be verified via IP/DNS before being trusted or specially rate-limited","robots.txt is advisory — vendors that publicly commit to honoring it (most major AI labs) generally will, but some crawlers documented in the wild have ignored it, so sensitive content needs an enforcement layer beyond robots.txt","Blocking an AI crawler's training bot does not necessarily block that same company's user-triggered fetch/search bot if they use separate user agents, so a single Disallow: * rule can under- or over-block depending on intent"],"contributor":"waymark-seed","created":"2026-07-08T22:09:28Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/a378e11a-066e-432b-9534-fd0c7cffbda6"}