Web Bot Auth is an emerging standard — backed by Cloudflare, Akamai, AWS, and OpenAI, and now under an IETF working group — that lets a bot prove its identity by cryptographically signing each request with RFC 9421 HTTP Message Signatures, instead of relying on a User-Agent string a site has to trust at face value.
Every signed request carries three headers: Signature-Input, which names the key ID, a creation and expiry timestamp, and a tag identifying the scheme as web-bot-auth; Signature, the cryptographic signature itself, computed with the operator's private Ed25519 key over a canonical subset of the request; and Signature-Agent, a URL pointing at the operator's public key directory. A site — or in practice the CDN in front of it — fetches that directory, verifies the signature against the published public key, and checks the timestamps haven't expired.
The public keys live at a well-known path (/.well-known/http-message-signature-directory) as a JSON Web Key Set, so any verifier can fetch them without a bespoke integration per operator. Cloudflare, which has pushed the spec furthest into production, validates signatures automatically at its edge for any zone that opts in, and already recognizes agents from Anthropic, OpenAI, Perplexity, and Common Crawl out of the box.
A User-Agent header is just a string a client sets — nothing stops a scraper from claiming to be Googlebot, and nothing lets a site verify that claim. IP allowlisting solves that but breaks the moment an operator's IP ranges change, and it doesn't scale to a world of many AI agents crawling from cloud infrastructure. A signed request ties an identity to a private key the operator actually controls, so a site can grant favorable treatment — higher rate limits, skipped challenges — to traffic it can cryptographically verify, and treat everything else as unverified.
For infrastructure that isn't an already-recognized operator, Web Bot Auth doesn't block anything by itself — unsigned requests still get served, just without the verified-bot fast lane. The practical effect so far is a widening gap between traffic that opts into verifiable identity and gets preferential treatment, and everything else that keeps competing on IP reputation and fingerprint quality the way it always has.
How Crawlora handles this
Crawlora doesn't market its requests as a verified AI-agent identity today — the platform's approach to access is proxy and browser-rendering quality rather than a signed-request program — so Web Bot Auth is worth tracking as a category: a site enrolled in a verified-bots program may start treating unsigned traffic more strictly over time, which is one more reason request quality (fingerprints, IP reputation, pacing) stays the baseline defense regardless of which crawlers get a cryptographic fast lane.
Glossary
FAQ
Not automatically — unsigned requests still get normal responses today. What changes is that verified-bot programs can grant signed traffic preferential treatment (higher limits, fewer challenges), which raises the relative cost of staying unverified over time rather than blocking it outright.
No. robots.txt is a voluntary, unauthenticated request, and a CAPTCHA challenges an unknown visitor. Web Bot Auth is a cryptographic identity claim — it proves who is making a request, using RFC 9421 HTTP Message Signatures, rather than asking a client to behave a certain way.
Technically yes — it's an open specification with public libraries, and any operator can generate an Ed25519 key pair and publish a signature directory. The practical barrier is being recognized: a site's verified-bot list has to actually include your key for the signature to earn different treatment.
Beyond Web Bot Auth, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.