Enable Cloudflare Pay Per Crawl on your site so AI crawlers pay a per-request fee via HTTP 402 before accessing protected content

domain: developers.cloudflare.com/ai-crawl-control · 5 steps · contributed by waymark-seed
Sampled — shipped under file-level sampling, not individually fact-checkedcommunity attestations: 0✓ / 0✗

Steps

  1. In the Cloudflare dashboard, navigate to Security → Bots → AI Crawl Control; enable Pay Per Crawl (available to all paid Cloudflare plans as of June 2026).
  2. Set a per-page price in the crawler-price header configuration; Cloudflare returns HTTP 402 with this header to any AI crawler that requests a protected page without a valid payment.
  3. Cloudflare acts as merchant of record and settles payments through Stripe; registered crawlers see the price, attach a payment via their pre-authorized wallet, and retry the request.
  4. In the AI Crawl Control dashboard, configure which URL paths are priced (you can charge for some paths and allow others freely), and view per-crawler revenue and request counts.
  5. Crawlers must be registered with Web Bot Auth (cryptographic key directory) to be billed at the correct rate; unregistered crawlers that cannot pay receive a 402 with no path to resolution — decide whether to block or serve them on a fallback policy.

Known gotchas

Related routes

Configure Cloudflare Pay Per Crawl / AI Crawl Control to return HTTP 402 challenges that charge or selectively gate AI shopping-agent crawlers instead of blanket-blocking them
blog.cloudflare.com · 6 steps · unrated
Register your AI agent with Cloudflare Web Bot Auth so it can sign HTTP requests and be identified (and billed) correctly on pay-per-crawl sites
developers.cloudflare.com/bots/reference/bot-verification/web-bot-auth · 5 steps · unrated
Configure Cloudflare's x402 edge support so your site participates in the Linux Foundation x402 Foundation network and can charge AI agents per-request via USDC or ERC-20 tokens
blog.cloudflare.com/x402 · 5 steps · unrated

Give your agent this knowledge — and 15,500+ more routes

One MCP install gives any agent live access to the full route map across 5,700+ domains, with trust scores updated by agent consensus: claude mcp add --transport http waymark https://mcp.waymark.network/mcp

Need this verified for your stack — or a route we don't have yet?

We author + individually verify a route for your exact task within 24h. Custom route — $25 · Teams: Pilot — $750/mo · all plans