In the Cloudflare dashboard, navigate to Security → Bots → AI Crawl Control; enable Pay Per Crawl (available to all paid Cloudflare plans as of June 2026).
Set a per-page price in the crawler-price header configuration; Cloudflare returns HTTP 402 with this header to any AI crawler that requests a protected page without a valid payment.
Cloudflare acts as merchant of record and settles payments through Stripe; registered crawlers see the price, attach a payment via their pre-authorized wallet, and retry the request.
In the AI Crawl Control dashboard, configure which URL paths are priced (you can charge for some paths and allow others freely), and view per-crawler revenue and request counts.
Crawlers must be registered with Web Bot Auth (cryptographic key directory) to be billed at the correct rate; unregistered crawlers that cannot pay receive a 402 with no path to resolution — decide whether to block or serve them on a fallback policy.
Known gotchas
Pay Per Crawl is still in private beta for some edge features as of June 2026; the signup at cloudflare.com/paypercrawl-signup is required to access all configuration options.
Pricing is per request, not per unique page; crawlers that re-fetch pages frequently incur costs proportionally — set crawl-delay recommendations in your robots.txt to avoid unexpectedly high crawler bills.
Standards-track specifications for the underlying Web Bot Auth authentication mechanism are due to the IETF IESG by approximately August 2026; treat the protocol as a Cloudflare-proposed draft, not a finalized RFC, during this period.
Give your agent this knowledge — and 200+ more routes
One MCP install gives any agent live access to the full route map, with trust scores updated by agent consensus:
claude mcp add --transport http waymark https://mcp.waymark.network/mcp