Cloudflare Will Crawl the Web for You. It's Locked Out of 29% of Its Own Customers.

Cloudflare's /crawl endpoint will crawl an entire site for you, and by design it plays fair: it is, in Cloudflare's words, "a verified bot (intermediary agent) that respects robots.txt and AI Crawl Control by default." We joined our top-1M anti-bot census with our top-1M robots.txt census — 818,614 reachable sites, both open datasets, an exact domain-for-domain join — to measure what that politeness costs it. A site sitting behind Cloudflare is 2.6× more likely to be closed to Cloudflare's own crawler than a site that isn't: 29.1% versus 11.3%. On 60,528 of those sites, Cloudflare's own bot management actively challenged us. Web-wide, 12.8% of the top 100,000 sites is shut to it.

Two gates, not one

Almost everything written about /crawl since its March 10, 2026 launch has framed the limit as an anti-bot problem: how much of the web has a wall that will stop it. That framing is wrong, and it is wrong in a way that is easy to miss.

Cloudflare's crawler identifies itself as CloudflareBrowserRenderingCrawler/1.0, and the user agent is not customizable for /crawl. It does not bypass "CAPTCHAs, Turnstile challenges, or any other bot protection mechanisms," and Cloudflare confirms that Browser Run requests "are always identified as bot traffic." So a site is closed to it if either of two independent gates is shut:

• The technical gate — the site actively challenged or blocked our probe. This is the definition already published on our Anti-Bot Adoption Index: a fingerprinted vendor wall plus active enforcement.
• The policy gate — robots.txt carries User-agent: * / Disallow: /. This is not our approximation of what stops Cloudflare's crawler. It is the exact rule Cloudflare's own documentation tells site owners to use, describing it as "the most restrictive configuration," one that "blocks all compliant bots, not just Browser Run."

The two gates measure different things, and — this is the part that matters — they barely overlap. Count only walls and you miss 44.9% of the closed web; count only robots.txt and you miss 53.3%.

Show the flows

Reachable → Open to /crawl	673,753 (69.9%)
Reachable → Closed	144,861 (15%)
Closed → Technical gate only	77,214 (8%)
Closed → Policy gate only	65,026 (6.7%)
Closed → Both gates	2,621 (0.3%)

Show the two-gate table (counts and % of reachable)

Gate	Sites	% of reachable
Policy gate only (robots.txt says no; no wall)	65,026	7.94%
Technical gate only (actively challenged; no blanket disallow)	77,214	9.43%
Both gates	2,621	0.32%
Union — closed to /crawl	144,861	17.70%
Open on both axes	673,753	82.30%

The number we are not going to lead with

That 17.70% is literally true, and it is the number this study was expected to produce. We are not leading with it, because we do not believe it means what a reader would take it to mean.

Split the policy gate by Tranco rank and it behaves strangely. The technical gate is nearly flat from the head of the web to the tail — around 10% everywhere. The policy gate collapses through the middle of the ranking and then triples in the tail.

Technical gate (wall)9.86% → 9.7%Policy gate (robots.txt)6.62% → 8.85%

Top 1,000

9.86%

6.62%

1k–10k

10.8%

2.86%

10k–100k

10.15%

2.93%

100k–1M

9.7%

8.85%

Break that tail spike down by top-level domain and it stops looking like publisher behaviour:

TLD	Top 1,000	1k–10k	10k–100k	100k–1M
.us	—	0.0%	4.7%	78.2%
.nl	—	0.0%	1.5%	40.8%
.uk	0.0%	7.3%	1.1%	39.2%
.ca	0.0%	0.0%	1.3%	39.1%
.xyz	—	12.5%	2.4%	23.4%
.se	—	0.0%	1.1%	17.2%
.de	0.0%	1.4%	1.7%	17.7%
.com	7.0%	2.9%	3.2%	7.4%
.net	12.5%	5.9%	3.7%	4.5%
.ru	14.3%	2.4%	3.2%	3.0%
.org	0.0%	1.5%	1.9%	2.3%
.br	0.0%	1.0%	2.2%	1.9%

.com, .org, .net, .ru and .br are flat across the whole ranking, which is what a genuine editorial decision looks like. .us goes from 4.7% to 78.2% across a single rank boundary. .nl, .uk and .ca all jump by a factor of thirty in the same place.

Sampling those domains explains it in one glance. They are combinatorially generated: brightblackjacklounge.nl, clearroulettedaily.nl, coreslotspro.nl, betrixslots.co.uk, blogculturehub.co.uk, smartpalace883.com. Bulk-registered gambling and SEO networks, built from a shared template, and the template ships a blanket Disallow: /. Their robots.txt files are real — every one returned HTTP 200 — but they encode a hosting default, not a publisher's decision about who may read their work.

So we report three numbers and lead with the most conservative:

Universe	Sites	Policy gate	Technical gate	Closed to /crawl
All reachable top 1M (raw)	818,614	8.26%	9.75%	17.70%
Excluding the six affected ccTLDs	753,939	5.57%	9.51%	14.74%
Top 100,000 (no TLD shows the anomaly)	81,577	2.96%	10.21%	12.79%

12.8% of the top 100,000 sites is closed to Cloudflare's crawler. That is the figure we will defend. It uses a standard universe with no ad-hoc exclusions, and it is the one place where every TLD behaves consistently. It is also the smallest of the three, which is the right direction for a number published by a company that sells the alternative.

The casualty of that correction is a finding we liked. In the raw top-1M data, blanket-disallow sites are less likely to actively challenge (3.9% vs 10.3%) — a tidy story about the sign and the guard being substitutes, sites putting up a notice instead of a wall. It is an artefact. Restrict to the top 100,000 and the correlation reverses: sites that say no in robots.txt are slightly more likely to also challenge (a 1.26× lift, against 0.40× in the raw data). We are reporting it because we went looking for it and it was not there.

Cloudflare's own footprint is its worst territory

Cloudflare guards 369,775 reachable sites in our census — 45.2%, comfortably the largest anti-bot footprint on the web. (That groups Cloudflare's WAF with Turnstile; the WAF alone is the 45.0% published on the index.) Its crawler does worst exactly there.

Sites behind Cloudflare29.1%

369,775 sites — closed to Cloudflare's own crawler

Sites not behind Cloudflare11.3%

448,839 sites

Both gates push the same way. Cloudflare-guarded sites actively challenge at 16.4% and carry a blanket disallow at 13.2%; the rest of the web does 7.8% and 4.2%. And the gap is not an artefact of the tail — it holds at 1.76×–1.84× in every band above rank 100k, where the domain-network contamination is absent.

Show the Cloudflare-versus-rest table, by rank band

Rank band	CF sites	CF closed	Non-CF sites	Non-CF closed	Ratio
Top 1,000	175	25.71%	565	14.34%	1.79×
1k–10k	2,492	20.99%	4,807	11.90%	1.76×
10k–100k	29,690	19.83%	43,848	10.77%	1.84×
100k–1M	337,418	30.03%	399,619	11.39%	2.64×
All reachable	369,775	29.14%	448,839	11.34%	2.57×

The mechanism is not mysterious, and it is not really about the wall. It is selection. Cloudflare sold bot defence to the population of sites least willing to be crawled, and it now sells a crawler that honours their wishes. The product is structurally weakest on its own customer base.

The literal version of that: 60,528 sites run Cloudflare bot management that actively challenges Cloudflare's crawler.

Worth being fair to Cloudflare here, because the obvious reading is the wrong one. It is not blocking its own crawler on customers' behalf: its FAQ notes that Cloudflare "does not enforce bot protection by default — that is the customer's choice," and 83.6% of its guarded sites did let our plain request through passively. The company shipped the wall and left it switched off. Its customers turned it on.

Show Cloudflare's enforcement posture across its 369,775 guarded sites

Posture	Sites	% of Cloudflare-guarded
Passive edge (wall present, let us through)	309,247	83.63%
Active challenge	44,177	11.95%
Active block	16,351	4.42%
Active total	60,528	16.37%
Blanket robots.txt disallow	48,926	13.23%
Closed to /crawl (either gate)	107,770	29.14%

The Enterprise allowlist trap

Suppose you are a Cloudflare customer who wants Cloudflare's crawler to read your own site — you are building a RAG index over your own documentation, say. You are one of the 60,528. Cloudflare's own Browser Run FAQ answers this directly: allowlisting Browser Run needs a WAF custom rule, and "You must be on an Enterprise plan to allowlist Browser Run on your own website because WAF custom rules require access to Bot Management fields."

There is no caller-side escape either. There is no parameter to ignore robots.txt, the user agent cannot be overridden for /crawl, and requests carry a Signature-agent header under Web Bot Auth, so they are cryptographically attributable. The compliance is genuinely not bypassable — which is a design virtue, not a flaw.

But it means a Free, Pro, or Business customer on Cloudflare has no supported way to let Cloudflare's crawler through Cloudflare's wall on their own website. Cloudflare sits on both sides of the transaction, and the self-serve fix exists only at the top tier.

Two years of blocking GPTBot bought nothing here

11.12% of reachable sites (91,020) block at least one named AI crawler — GPTBot, CCBot, ClaudeBot, and the rest of the list the web has spent two years learning to add. None of it binds Cloudflare's crawler. CloudflareBrowserRenderingCrawler is not on those lists, it launched in March 2026, and our robots census ran in June. Only the * group binds it at web scale.

Meanwhile 26.58% of reachable sites (217,615) serve no robots.txt at all — no policy restriction of any kind, for any crawler.

The web's accumulated crawler policy is mostly a list of proper nouns, and it goes stale the moment someone ships a new crawler.

The strongest argument against this post

We should state the obvious objection in its strongest form, because it lands.

"12.8% closed means 87.2% open. You have just proved Cloudflare's crawler works fine for the overwhelming majority of the web — and you are a scraping vendor with an obvious motive to say otherwise."

That is fair, and the honest answer is: yes. For most of the web, /crawl is genuinely sufficient. If you are crawling documentation sites, blogs, or your own properties, it will do the job, it is cheap, and during the beta render: false is not billed at all. We would rather say that plainly than pretend otherwise.

The claim worth defending is narrower and survives the objection: the closed portion is not randomly distributed. It concentrates on Cloudflare's own customers (29.1% versus 11.3%), and it concentrates on sites that made a deliberate choice to defend themselves. Sites that defend themselves are disproportionately the sites with data worth defending — marketplaces, real estate, forums, classifieds. In our category difficulty data, those four sit at the top of the difficulty ranking. The average site is open. The site you actually wanted is likelier not to be.

There is one piece of independent corroboration, and it is not ours. When The Web Scraping Club tested /crawl by hand against a set of WAF-protected sites in May 2026, it failed against every one of them, across vendors. That is a small sample and we are citing it as such — but it is consistent with the mechanism, since a fixed, self-identifying, non-spoofable user agent is the easiest possible thing for any vendor to filter on. We have not run /crawl against a vendor matrix ourselves, and we are not claiming we have.

A second objection: robots.txt is advisory, so this measures politeness rather than capability. True — Cloudflare says so itself, noting that the standard "is a voluntary protocol" that only well-behaved bots follow. But that is the point rather than a flaw. /crawl's entire value proposition is that it is the polite one. For a crawler whose selling point is compliance, an advisory gate is a hard gate.

What we could not measure

Four honest holes, all of which push the same direction — our number is a floor.

1. star_disallow_root catches only full-site blocks. Path-scoped Disallow: lines under * — /search, /cart, /admin, /api — bind Cloudflare's crawler page by page and are not counted at all. The share of content it will not fetch is materially higher than the share of sites it will not enter.

2. AI Crawl Control is invisible to us. It is an edge-side toggle; no external robots.txt scan can see it. It applies specifically to Cloudflare-guarded sites, so it makes the 29.1% an understatement. Cloudflare's July 1, 2026 announcement — that from September 15 it will default new ad-supported domains to blocking the Agent and Training categories, and that it is "no longer viewing Verified as 'default allowed'" — will move this number in a direction we cannot observe. We covered that change separately.

   There is an open question underneath it we could not close. Cloudflare's promoted use cases for /crawl are knowledge bases, RAG, and training AI systems, which reads as Training or Agent rather than Search. If its own crawler is classified that way, then from September 15 Cloudflare's defaults would begin blocking Cloudflare's crawler on new ad-supported customer domains. Cloudflare has not published where it classifies CloudflareBrowserRenderingCrawler, and the BotBase directory that would say is an Enterprise-only dashboard — so this is the question we would put to Cloudflare, not a finding.

3. Content Signals are unmeasured. Our scanner does not parse Content-Signal: lines, and Cloudflare-managed robots.txt now ships ai-train=no by default.

4. Homepage-level, datacenter vantage, two scans six days apart (anti-bot 2026-06-14, robots 2026-06-20). Deep pages are more walled than homepages — 48.4% versus 40.5% in our deep-page census — so the technical gate is a floor too.

And one conflict of interest, stated plainly: we sell an API that reaches sites Cloudflare's crawler will not. That is exactly why both underlying datasets are open, the join is an inner join on a shared seed list, and the reproduction script is committed alongside this post. Recompute it and tell us if we are wrong.

Where this leaves /crawl

It is a good product with an honest constraint. If your targets are open — and most are — use it. It is a verified, well-behaved crawler with generous limits: 100,000 pages per job, seven-day runtime, HTML, Markdown, or JSON output, a 0.5-second default delay that honours Crawl-delay.

If your targets are the defended part of the web, its compliance is not a setting you can change, and the remedy on Cloudflare's own network costs an Enterprise plan. That is where a managed unblocking API is a different tool rather than a competing one.

Three things follow if you are choosing a crawler:

• Check the policy gate, not just the wall. An anti-bot check answers barely half the question. A site can return a clean 200 and still be off-limits to anything that reads robots.txt.
• Know which kind of crawler you are running. A compliant crawler and a resilient one fail on different sites, and this data says those sets overlap far less than you would guess. Choosing between them is a policy decision, not a technical one.
• The two questions are separable. You can check any single URL's wall with our free anti-bot checker, and the whole robots.txt picture in the AI-Crawler Blocking Index.

Reach the 12.8%

Crawlora's API handles the transport, fingerprinting and access layer for public data on sites that challenge a polite crawler — with the same open datasets behind this study.

Check any URL freeBrowse the Anti-Bot Index

Method

Two censuses, one inner join on registrable domain, restricted to sites the anti-bot scan reached.

• Anti-bot census — run top1m-20260614, 998,497 sites scanned from the Tranco top 1M, 818,614 reachable. Open dataset: anti-bot-adoption-index-data (CC BY 4.0). Technical gate = protected == true and enforcement ∈ {active_block, active_challenge} — the definition published on /anti-bot-index.
• Robots census — run aicrawler-top1m-20260620, same seed list. Open dataset: ai-crawler-blocking-index-data (CC BY 4.0). Policy gate = star_disallow_root.
• Join coverage is 100% — all 818,614 reachable domains are present in both. The join reproduces the published anti-bot figures exactly (818,614 reachable, 437,857 walled, 79,835 actively challenged, Cloudflare at 45.0% present and 16.4% active rate), which is the check that the two datasets are keyed the same way.
• One definitional note. The Cloudflare-versus-rest comparison counts active enforcement even where no vendor could be fingerprinted. The stricter published definition requires a vendor fingerprint, which a Cloudflare site always has and a non-Cloudflare site may not — using it would have inflated the ratio to 3.5× instead of 2.6×. We report the lower figure.

Both datasets are public and CC BY 4.0, and the join is one pass over each file (about 39 MB of download):

python3 scripts/anti-bot-index/cloudflare-crawl-coverage.py --data data --json figures.json

It prints every table in this post, including the validation block that reproduces the published anti-bot figures. Cite it as "Crawlora Anti-Bot Adoption Index" with a link.

Related reading

• How Much of the Web Runs Anti-Bot? We Scanned the Top 1,000,000 Sites — the technical gate in full.
• The Anti-Bot Adoption Index — searchable, every site, filterable by vendor and difficulty.
• The AI-Crawler Blocking Index — the policy gate: who blocks GPTBot, CCBot and ClaudeBot by name.
• Cloudflare Blocks AI Crawlers by Default in 2026 — the September 15 AI Crawl Control change, which neither census can see.
• Scraping sites that block bots — what each vendor actually does.