Pay Per Crawl is a monetization model — pioneered by Cloudflare and now part of a broader industry push — that lets a site owner charge an AI crawler for each page it accesses, using the long-dormant HTTP 402 Payment Required status code as the machine-to-machine payment signal instead of a human-facing paywall.
A site owner sets a price per zone or path. An AI crawler requests a page without payment credentials and gets back HTTP 402 Payment Required along with pricing information in the response instead of the content. A crawler that wants the content presents payment intent via request headers on a follow-up request; if the payment clears, the site returns a normal 200 with the page. Cloudflare acts as the merchant of record for its version, handling the transaction so individual site owners don't each need a billing relationship with every AI crawler operator.
It's the same status code HTTP defined in 1997 and essentially nobody implemented, repurposed for a use case that didn't exist until AI training and retrieval crawlers started consuming content at a volume publishers wanted to price rather than block outright. Cloudflare's rollout reports over a billion 402 responses a day at network scale as of 2026, which signals real adoption, not just a spec on paper.
Pay Per Crawl is one layer of a broader toolkit sites now use to manage AI crawler access, alongside Content Signals — an extension to robots.txt that lets a site express finer-grained preferences than a blanket allow/disallow, like permitting a crawl for search indexing while disallowing it for AI training — and bot-verification schemes like Web Bot Auth that confirm which crawler is actually making the request. Together they shift the AI-crawler relationship from binary block-or-allow toward priced, verified, and permissioned access.
For scraping and data infrastructure generally, it's worth watching as a precedent rather than a direct blocker: it's currently scoped to identified, verified AI crawlers — the ones a site can recognize and bill — not general-purpose scraping traffic. But a site that adopts 402-based monetization for one crawler class has the infrastructure in place to extend pricing or access rules more broadly if it chooses to.
How Crawlora handles this
Crawlora doesn't operate as a named AI crawler paying into per-site 402 programs — the platform's endpoints are built around structured, public data access through proxies and browser rendering rather than negotiated per-target payment — so Pay Per Crawl is a trend worth tracking for how individual target sites might change their access posture, not something that currently gates a Crawlora request.
Glossary
FAQ
Currently it's scoped to identified AI crawlers a site can recognize and bill — general scraping traffic that isn't enrolled in a verified-bot program doesn't get a 402 payment prompt today, though the underlying infrastructure could extend to broader traffic if a site chose to.
In Cloudflare's implementation, Cloudflare acts as merchant of record — handling the transaction between the AI crawler operator and the site owner — so individual publishers don't need a separate billing relationship with every crawler.
A paywall gates content from human readers through a UI. Pay Per Crawl gates content from automated crawlers through the HTTP layer itself — a 402 status code and pricing headers a crawler's code has to handle programmatically, with no human-facing page involved.
Beyond Pay Per Crawl, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.