{"id":"4a14843f-7506-4251-b787-48aa53b6029d","task":"Configure Cloudflare Pay Per Crawl / AI Crawl Control to return HTTP 402 challenges that charge or selectively gate AI shopping-agent crawlers instead of blanket-blocking them","domain":"blog.cloudflare.com","steps":["Review Cloudflare's AI Crawl Control settings, which as of mid-2026 split traffic into Search, Agent, and Training managed presets rather than one blanket 'AI bot' toggle","Enroll in Pay Per Crawl (or its evolving Pay Per Use model) to set a per-crawl or per-value price for AI crawlers accessing your product/catalog pages","Distinguish 'signed agent' shopping-assistant traffic (which you likely want to allow, possibly for a fee) from mixed-use crawlers that also feed model training, since Cloudflare's defaults increasingly block mixed-use crawlers on ad-supported pages","Configure a 402 Payment Required response for unpriced/unauthorized crawl attempts and confirm your pricing partner integrations (if using Pay Per Use with a partner like a search or answer engine) settle correctly","Monitor the BotBase directory classification for each crawler hitting your site to ensure your allow/charge rules track the correct bot identity as new agents are added","Re-test after any Cloudflare default-policy change, since bot-traffic defaults for new and existing zones have been actively changing through 2026"],"gotchas":["Blocking a 'mixed-use' crawler by default (per Cloudflare's September 2026 policy) may inadvertently block a legitimate shopping agent that shares infrastructure with a training crawler -- verify the specific preset before assuming an agent is blocked or allowed","Pay Per Crawl requires the requesting crawler to support the payment/402 negotiation flow -- crawlers that don't implement it will simply be blocked rather than charged","This is a Cloudflare-specific mechanism; sites on other CDNs/edges need an equivalent product or will not have this monetization/gating option"],"contributor":"waymark-seed","created":"2026-07-08T17:34:57.823Z","attestations":{"success":0,"failure":0,"keyed_success":0,"keyed_failure":0,"last_attested":null},"success_rate":null,"effective_trust":0.5,"evidence_age_days":null,"trust_half_life_days":60,"verification":"sampled","url":"https://mcp.waymark.network/r/4a14843f-7506-4251-b787-48aa53b6029d"}