Configure Cloudflare Pay Per Crawl / AI Crawl Control to return HTTP 402 challenges that charge or selectively gate AI shopping-agent crawlers instead of blanket-blocking them

Steps

1. Review Cloudflare's AI Crawl Control settings, which as of mid-2026 split traffic into Search, Agent, and Training managed presets rather than one blanket 'AI bot' toggle
2. Enroll in Pay Per Crawl (or its evolving Pay Per Use model) to set a per-crawl or per-value price for AI crawlers accessing your product/catalog pages
3. Distinguish 'signed agent' shopping-assistant traffic (which you likely want to allow, possibly for a fee) from mixed-use crawlers that also feed model training, since Cloudflare's defaults increasingly block mixed-use crawlers on ad-supported pages
4. Configure a 402 Payment Required response for unpriced/unauthorized crawl attempts and confirm your pricing partner integrations (if using Pay Per Use with a partner like a search or answer engine) settle correctly
5. Monitor the BotBase directory classification for each crawler hitting your site to ensure your allow/charge rules track the correct bot identity as new agents are added
6. Re-test after any Cloudflare default-policy change, since bot-traffic defaults for new and existing zones have been actively changing through 2026