A proof-of-work challenge is an anti-scraping technique that forces a visitor's browser to solve a small, deliberately expensive cryptographic puzzle before a page loads — cheap enough that one human visitor never notices, but costly enough that a crawler fetching thousands of pages pays a real compute bill. Anubis, an open-source reverse proxy Xe Iaso released in January 2025, is the tool that popularized the approach against AI scraper traffic.
Anubis's challenge, modeled on the decades-old Hashcash scheme, hands the client a random number and asks it to find a second number — a nonce — such that the SHA-256 hash of the two concatenated together starts with a set number of leading zero bits. There's no shortcut: finding a matching nonce means brute-force guessing, but verifying a proposed answer is nearly instant, which is the whole point of a proof-of-work scheme. The default difficulty (5 leading zeros) solves in a couple of seconds on an ordinary laptop; each additional required zero roughly doubles the expected work, so a site can tune the cost up for traffic it trusts less.
Once solved, the client gets an EdDSA-signed token, often stored as a cookie, valid for a configurable window — so a real visitor only pays the cost once per session rather than on every page load. The challenge only triggers for requests that look like a browser (a User-Agent containing "Mozilla") and is skipped for low-harm paths like robots.txt, so it adds friction to bulk automated fetching specifically rather than to every request a site receives.
Anubis exists because of a specific, now-common problem: open-source infrastructure — Git forges, wikis, mailing-list archives — getting hammered by AI-training crawlers with no budget for an enterprise bot-management contract, and in some documented cases crawlers that ignored robots.txt entirely. Xe Iaso built it after Amazon's crawler overloaded a Git server this way; it's since been adopted by projects including UNESCO, GNOME's GitLab instance, the Linux kernel mailing list archive, Wine, and FFmpeg — infrastructure that needed a free, self-hosted deterrent rather than a managed anti-bot platform's pricing and integration overhead.
The tradeoff is real: a proof-of-work challenge doesn't distinguish a well-behaved research scraper from an aggressive training crawler — both look the same to it — and a determined, well-resourced scraper can simply pay the compute cost, so it mainly raises the bar for mass, careless scraping rather than stopping a targeted one. It's also a genuinely different mechanism from a CAPTCHA (no human interaction or image/text puzzle involved) and from a WAF or fingerprint-based anti-bot system (it doesn't inspect TLS or browser signals at all — the compute cost is the only gate).
How Crawlora handles this
A proof-of-work wall is one more rendering obstacle Crawlora's browser-based `render: "auto"` escalation is built to clear — the request needs a real browser context to execute the challenge's JavaScript and retain the resulting signed token across the session, the same escalation pattern Crawlora already uses to get through Cloudflare's and DataDome's JS-based checks.
Related reading
Glossary
FAQ
No. A CAPTCHA asks a visitor to prove they're human by solving a puzzle a bot struggles with. A proof-of-work challenge doesn't care who's asking — it imposes a compute cost on every browser-presenting client, human or not, and lets anyone willing to pay it through.
Yes, technically — it's compute, not a hard block, so a well-resourced operation can pay the cost at scale. It mainly deters low-effort, high-volume scraping rather than stopping a determined, well-funded one.
No — it doesn't inspect TLS fingerprints, browser signals, or behavior at all. It only adds a compute tax to any client that presents itself as a browser, which is a different, complementary layer to fingerprint-based anti-bot systems.
Beyond Proof-of-Work Challenge, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.