What percentage of websites run an anti-bot or WAF?

In an independent 2026 scan of the top 1,000,000 sites (seeded from the Tranco ranking), 53.5% of the 818,614 reachable sites ran a managed anti-bot, WAF, or access-control vendor. Cloudflare alone accounts for 45% of reachable sites.

What percentage of websites use Cloudflare?

Crawlora detected Cloudflare's active anti-bot posture on 45% of reachable top-1M sites (84.2% of every protected site). W3Techs independently reports Cloudflare on about 48.7% of the top 1M and 23.2% of all websites — consistent with this scan.

Which anti-bot vendor is most common?

Cloudflare is the most common by far (45% of reachable top-1M sites). CAPTCHA widgets such as reCAPTCHA and hCaptcha are next at about 12.6%. Enterprise bot-management (DataDome, PerimeterX, Akamai Bot Manager) is under 1% across the whole web but rises to about 15.5% among the major, high-value sites people actually scrape.

How much of internet traffic is bots?

Per Imperva's 2025 Bad Bot Report, automated traffic is now 51% of all web traffic (bad bots 37%). That is the demand side; on the supply side, Crawlora's scan found 53.5% of the reachable top-1M sites deploy a managed anti-bot or WAF to manage it.

Data study · updated June 14, 2026

How much of the web runs anti-bot?

We scanned the full top 1,000,000 sites on the Tranco ranking and recorded the anti-bot/WAF vendor on each. Just over half of the reachable web runs managed bot defense — and it’s overwhelmingly Cloudflare. Below, we zoom into 1005 hand-picked sites people actually scrape for the full vendor, CAPTCHA and difficulty breakdown. Paste any URL to check it yourself — protection is per-URL.

53.5%

of the 818,614 reachable top-1M sites run a managed anti-bot / WAF — Cloudflare alone covers 45%.

12.6%

show a CAPTCHA widget (reCAPTCHA, hCaptcha)

0.9%

use enterprise bot-management (DataDome, PerimeterX, Akamai)

Independent scan of the top 1,000,000 sites (Tranco, 2026-06-11). Homepage-level, datacenter-IP — a lower bound, since deep pages are more protected.

Across the whole web

The long tail is Cloudflare.

Across the 818,614 reachable sites, Cloudflare covers 45% — 84.2% of every protected site. Bot defense at web scale is effectively one vendor plus a layer of cheap CAPTCHA widgets (12.6%). The specialist bot-management vendors — DataDome, PerimeterX, Akamai Bot Manager — are rare here (0.9%) but jump to 15.5% among the high-value sites people actually scrape (below). Adoption and sophistication are different axes: the head is defended differently from the tail.

Cloudflare45% · 368,609 sites

Google reCAPTCHA7.5% · 61,238 sites

hCaptcha3.5% · 28,249 sites

Unidentified WAF / bot protection2.1% · 17,383 sites

Cloudflare Turnstile1.3% · 10,910 sites

Protection by traffic rank

The busiest sites run the least detectable anti-bot.

Across the full top-1M, protection and Cloudflare both climb as rank descends while enterprise bot-management falls — so the head and the tail of the web are defended by different vendors. W3Techs corroborates the direction (Cloudflare is lower in the very top-1k than across the top-1M).

Protected44.2% → 53.6%Cloudflare23.4% → 45.6%Enterprise bot-mgmt7.6% → 0.6%

top 1k

44.2%

23.4%

7.6%

1k-10k

50.7%

34%

5.3%

10k-100k

52.3%

40.2%

2.5%

100k-1M

53.6%

45.6%

0.6%

% of reachable sites in each Tranco rank band. Hover a band to read it off.

What kind of wall

Edge WAF47.6%

CAPTCHA widget9.4%

Access control2.1%

Dedicated bot-mgmt0.8%

Rate limit0.2%

Of reachable sites (overlapping). Most protection is edge WAF; dedicated bot-management is rare at web scale.

% of reachable sites with a detectable wall in each Tranco rank band. Homepage-level, datacenter-IP — and passive fingerprinting is blindest exactly at the head: the giants keep homepages open and lean on invisible in-house/behavioural defenses a single GET can’t see. Read the top-1k as least fingerprintableprotection, not least protected — the gap is partly real, partly a measurement floor.

Protection has a geography

The web's walls cluster by country code.

Across the full top-1M, the share of reachable sites running a managed anti-bot wall swings ~8× by top-level domain. The Anglophone and Commonwealth web is the most defended; the East-Asian and German web the most open.

.us89%n=6,788

.uk79.5%n=28,849

.ca69.6%n=7,769

.nl64.5%n=12,747

.au62.8%n=6,908

.com58.8%n=357,443

.io56.3%n=7,930

.ai54.5%n=3,090

.org51.5%n=34,065

.net46.4%n=31,964

.fr45.4%n=9,312

.br44.3%n=15,169

.in42.4%n=9,997

.ru30.5%n=27,961

.de24.3%n=24,393

.jp18.6%n=13,921

.kr11.2%n=3,615

% of reachable sites in each TLD running a managed anti-bot wall (run top1m-20260614). Hover a row to isolate it.

.us (89%), .uk (79.5%), .ca (69.6%) and .au (62.8%)sit at the top — business-heavy registries where Cloudflare is near-ubiquitous.

At the other end, .kr (11.2%), .jp (18.6%) and .de (24.3%) run the least: East-Asian and German sites lean on local infrastructure our scan rarely flags as a managed wall.

.com — 357,443 reachable sites, the bulk of the web — sits mid-table at 58.8%.

Who guards the web, by rank

Scroll the ranking: the gatekeeper changes.

Same scan, sorted by traffic rank. Move from the web's busiest sites down into the long tail and the dominant anti-bot vendor flips — Cloudflare takes over, Akamai fades, cheap CAPTCHA rises. (This is the popularity ranking, not a timeline.)

Top 1,000

Cloudflare23.4%

Akamai (edge + Bot Manager)11.2%

Google reCAPTCHA4.1%

hCaptcha0.7%

Tranco rank band — the web's most popular sites at the top, the long tail at the bottom. % of reachable sites in the band. This is a ranking, not a timeline.

Top 1,000

The web's 1,000 busiest sites are a mixed field — Akamai's enterprise stack (11%) nearly rivals Cloudflare (23%), and cheap CAPTCHA barely registers. The giants run their own heavy machinery.

Cloudflare23.4%

Akamai (edge + Bot Manager)11.2%

Google reCAPTCHA4.1%

hCaptcha0.7%

1k–10k

Drop to the next tier and Cloudflare pulls ahead (34%) as Akamai starts to thin out (7%).

Cloudflare34%

Akamai (edge + Bot Manager)7.3%

Google reCAPTCHA3.7%

hCaptcha0.3%

10k–100k

Cloudflare keeps climbing (40%); Akamai halves again (3%) while turnkey reCAPTCHA ticks up.

Cloudflare40.2%

Akamai (edge + Bot Manager)3.4%

Google reCAPTCHA5.2%

hCaptcha1.3%

100k–1M

In the long tail — millions of smaller sites — Cloudflare guards nearly half of everything (46%), Akamai has all but vanished (1%), and cheap CAPTCHA (reCAPTCHA 8%, hCaptcha 4%) has taken its place.

Cloudflare45.6%

Akamai (edge + Bot Manager)0.9%

Google reCAPTCHA7.8%

hCaptcha3.7%

Most walls are asleep

Half the web has a bot wall — only ~1 in 10 actually used it on us.

Of the 818,614 reachable sites, 53.5% expose a managed anti-bot or WAF — but only 9.8% actively challenged our homepage request. The rest run the wall passively: a matched request just gets a 200. (Deeper pages challenge far more — treat this as a homepage lower bound.)

Where the million sites go. Hover a node to trace its flows.

Show the flows

Scanned → Reachable	818,614 (36.3%)
Reachable → Has a wall	437,857 (19.4%)
Reachable → No wall	380,757 (16.9%)
Has a wall → Passively present	358,022 (15.9%)
Scanned → Unreachable	179,883 (8%)
Has a wall → Actively challenged	79,835 (3.5%)

Detected vs actually challenging

Detected onActively challenged

Cloudflare45% → 7.4%16.4% active

Google reCAPTCHA7.5% → 0.3%3.4% active

hCaptcha3.5% → 0.2%5% active

Unidentified WAF / bot2.1% → 1.6%75.8% active

Cloudflare Turnstile1.3% → 0.1%4.9% active

Imperva (Incapsula)0.6% → 0.1%21.1% active

Akamai Bot Manager0.6% → 0.04%6.8% active

Both as % of reachable sites; the gap is passive coverage. Hover a row to isolate it.

Snapshot top1m-20260614 (998,497 sites scanned). “Active” = the homepage GET got a challenge/block, not a 200.

Which CAPTCHA you actually hit

Three products are almost every CAPTCHA on the web.

When a wall throws an interactive challenge, it's overwhelmingly one of three: Google reCAPTCHA, hCaptcha, or Cloudflare Turnstile. Everything else — including the new anti-AI proof-of-work walls — is a rounding error.

Google reCAPTCHA61,238

hCaptcha28,249

Cloudflare Turnstile10,910

GeeTest2,034

AWS WAF CAPTCHA864

Anubis (PoW)204

Sites detected running each CAPTCHA product, full top-1M scan (run top1m-20260614). Hover a row to isolate it.

Google reCAPTCHA alone (61,238 sites) is on more sites than every other CAPTCHA product combined.

The one to watch is Anubis— the proof-of-work wall built specifically to stop AI crawlers. It’s on just 204 sites today, but it’s the fastest-moving new gate, adopted by open-source projects and public institutions tired of LLM scrapers.

Zoom in · the 1005 sites people scrape

What the data says.

We hand-picked 1005 representative public sites across 28 categories and ran the full vendor + CAPTCHA + difficulty breakdown. 5 takeaways — each a single, dated, citable number.

77.2%

Two vendors run the show — Cloudflare and Akamai protect this share of every defended site between them.

85%

of these sites need no browser at all — a plain HTTP request or a matched TLS fingerprint reaches them. Even 78.4% of Cloudflare/Akamai sites answer without one: a vendor being present isn’t the same as it actively challenging.

86.1%

Protection tracks money — Crypto is the most defended, while search engines stay open (17.6%): the sites that live by crawling don’t wall off crawlers.

4.2/10

Real estate is the hardest vertical: when these sites defend, it’s DataDome or PerimeterX, not a Cloudflare toggle — adoption and difficulty are different axes.

sites flip difficulty band by IP — the same page reads clean from a fresh residential IP and challenges from a flagged datacenter one, so a single grade is a midpoint, not a guarantee.

From 1005 reachable sites in the curated set. Blocks are mostly a fingerprint problem, not a proxy one.

Vendor share · curated

Who guards the sites you scrape.

Cloudflare33.1% · 333 sites

Akamai Bot Manager11% · 111 sites

Akamai (edge)5.3% · 53 sites

DataDome3% · 30 sites

Imperva (Incapsula)1.5% · 15 sites

PerimeterX (HUMAN)1.5% · 15 sites

Difficulty by category

Where the hard-to-scrape sites actually live.

Across our 1005 curated sites (~36 per category), the share that's genuinely hard to reach swings from roughly half down to nothing. Real estate, marketplaces and forums are the hard wall; search, social and SaaS are wide open — and crypto runs a wall on 86% of sites yet only 6% are actually hard (challenged, not blocked).

Category	Easy	Medium	Hard
Real estate72% protected	25%	28%	47%
Marketplaces81% protected	33%	22%	44%
Forums & community75% protected	39%	31%	31%
Travel & hospitality72% protected	36%	39%	25%
Jobs & recruiting69% protected	47%	33%	19%
Classifieds44% protected	58%	19%	22%
E-commerce72% protected	50%	31%	19%
Reference & wiki64% protected	53%	25%	22%
Other67% protected	56%	25%	19%
Food & delivery72% protected	56%	31%	14%
Gaming53% protected	67%	8%	25%
Education50% protected	69%	8%	22%
AI tools78% protected	58%	33%	8%
Finance & markets75% protected	64%	19%	17%
Automotive58% protected	61%	25%	14%
Healthcare53% protected	64%	28%	8%
Telecom67% protected	56%	39%	6%
Crypto86% protected	64%	31%	6%
Entertainment42% protected	72%	17%	11%
Developer & tech56% protected	72%	17%	11%
Music & audio43% protected	69%	23%	9%
Government47% protected	64%	33%	3%
News & media39% protected	75%	19%	6%
Search engines18% protected	71%	29%	0%
Sports33% protected	75%	22%	3%
Social media25% protected	81%	17%	3%
SaaS & business50% protected	81%	17%	3%
Video & streaming39% protected	81%	17%	3%

Share of each category's curated sites by scraping difficulty — green easy, amber medium, red hard (rows sum to ~100%; ≈36 sites/category, run top1m-20260614). Sorted hardest-first. Hover a row to isolate it.

The hard end

Where the hardest targets cluster.

Beyond Cloudflare and Akamai, the specialist bot-management vendors — DataDome and PerimeterX — cluster on exactly the verticals where scraped data has direct dollar value (pricing, inventory, leads).

DataDome30 sites

Concentrated in Marketplaces, News & media, Real estate.

fnac.com wayfair.com allegro.pl autoblog.com cruisecritic.com etsy.com

PerimeterX (HUMAN)15 sites

Concentrated in Real estate, E-commerce, Finance & markets.

bloomberg.com calm.com

When a request didn't pass

A block isn't a block.

From a datacenter IP, the reason a request fails tells you the fix. A rate limit or an IP ban calls for rotating IPs — not a browser. A bot challenge or CAPTCHA needs a real browser. A login wall isn’t public at all. We read the status, headers and cookies to tell them apart.

Passed931 · 92.6%

Bot challenge55 · 5.5%

CAPTCHA18 · 1.8%

Geo-blocked1 · 0.1%

Homepages undersell it

The page you actually scrape is more walled than the homepage.

We re-probed the same top-50,000 sites at their deep pages — product, listing, profile, search — not just the homepage. Managed walls show up more often there, and login gates far more often.

Homepage40.5%

of the same top-50k sites carry a managed wall

Deep pages48.4%

product · listing · profile · search pages

Share with a managed anti-bot / WAF wall present, same top-50k sites (run top50k-deep-20260613).

Deep pages are also ~30× more likely to sit behind a login— 0.4% of deep pages vs 0.01% of homepages. Profiles, accounts and checkout are the usual gates.

Homepage-tier (datacenter HTTP) probe: it detects a managed wall’s presence, not full browser difficulty — a browser-tier deep pass would grade the rest. Treat it as a lower bound.

Reference tables

The headline numbers, as tables.

The same figures behind the charts above, as plain HTML tables — easy to copy, and machine-readable for search engines and AI answer engines that can't parse a chart.

Anti-bot vendor share — reachable top-1M

Vendor	% of reachable sites	Sites
Cloudflare	45%	368,609
Google reCAPTCHA	7.5%	61,238
hCaptcha	3.5%	28,249
Unidentified WAF / bot protection	2.1%	17,383
Cloudflare Turnstile	1.3%	10,910
Imperva (Incapsula)	0.6%	5,249

Protection by category — curated set

The data

Every site we tested (1005).

Search, filter by difficulty, vendor or category, sort any column, and download the result as CSV. Open = no managed anti-bot detected at the homepage.

Deep-page census (top-50k): a site's profile / org / listing pages are often protected very differently from its homepage. Review blocked deep pages · all deep pages · homepage census · download deep data

1005 sites Full dataset (GitHub)

Site	Category

By category

Protection tracks money.

Crypto86.1% protected · avg 2/10

Marketplaces80.6% protected · avg 3.9/10

AI tools77.8% protected · avg 2.3/10

Finance & markets75% protected · avg 2.3/10

Forums & community75% protected · avg 3.2/10

E-commerce72.2% protected · avg 2.7/10

This check can be inaccurate or out of date

Anti-bot is deliberately dynamic, so a snapshot like this can be wrong in both directions — and the vendors update their models constantly, often daily.

Homepages are the open front door. Login, checkout, search and deep listings are usually protected more heavily than the homepage we tested.
Difficulty is a moving snapshot, not a fixed property.What you see depends on the IP and its live reputation: the same page returns content from a clean residential IP but a challenge from a flagged datacenter one — and even a single IP flips over time, so a site can read “medium” one run and “hard” the next. This study ran from a datacenter; read each band as a sample, not a guarantee.
Challenges are conditional. Cloudflare managed challenge, DataDome and PerimeterX only trip on suspicious signals, so a protection can be present but invisible to a passive scan.
Vendors ship updates and per-customer configs constantly.Akamai added JA4 fingerprinting in 2026; a signature that’s correct today can be renamed or reconfigured tomorrow.
“Not detected” does not mean “easy.”It can mean a protection we didn’t recognise, a challenge that hadn’t triggered, or behavioural/TLS defenses that don’t show up in passive HTML. This bites hardest at the head of the web, where the largest sites are the most likely to run undetectable in-house systems — so the top-rank “protected” rates are a floor, not a ceiling.

semanticscholar.org	Education	Easy· 1/10	Plain HTTP client	—
collegeboard.org	Education	Easy· 1/10	Plain HTTP client	Akamai Bot Manager
imdb.com	Entertainment	Medium· 3/10	Browser-impersonation HTTP (matched TLS)	AWS WAF + 1 more· 1 CAPTCHA
eventbrite.com	Entertainment	Easy· 1/10	Plain HTTP client	—
giphy.com	Entertainment	Easy· 1/10	Plain HTTP client	—
people.com	Entertainment	Medium· 4/10 varies by IP	Browser-impersonation HTTP (matched TLS)	Cloudflare
pbs.org	Entertainment	Easy· 1/10	Plain HTTP client	—
letterboxd.com	Entertainment	Hard· 6/10	Headless browser	Cloudflare
buzzfeed.com	Entertainment	Easy· 1/10	Plain HTTP client	—
tenor.com	Entertainment	Easy· 1/10	Plain HTTP client	—
themoviedb.org	Entertainment	Easy· 1/10	Plain HTTP client	—
variety.com	Entertainment	Easy· 1/10	Plain HTTP client	—
ticketmaster.com	Entertainment	Hard· 6/10	Headless browser	Unidentified WAF / bot protection
hollywoodreporter.com	Entertainment	Easy· 1/10	Plain HTTP client	—
rottentomatoes.com	Entertainment	Easy· 1/10	Plain HTTP client	Akamai (edge)
tmz.com	Entertainment	Easy· 1/10	Plain HTTP client	—
myanimelist.net	Entertainment	Easy· 1/10	Plain HTTP client	—
deadline.com	Entertainment	Easy· 1/10	Plain HTTP client	—
timeout.com	Entertainment	Easy· 1/10	Plain HTTP client	—
screenrant.com	Entertainment	Easy· 1/10	Plain HTTP client	—
atlasobscura.com	Entertainment	Medium· 3/10	Browser-impersonation HTTP (matched TLS)	Cloudflare
collider.com	Entertainment	Easy· 1/10	Plain HTTP client	—
ew.com	Entertainment	Medium· 4/10 varies by IP	Browser-impersonation HTTP (matched TLS)	Cloudflare
eonline.com	Entertainment	Easy· 1/10	Plain HTTP client	Akamai (edge)
fandango.com	Entertainment	Easy· 1/10	Plain HTTP client	Akamai (edge)
cbr.com	Entertainment	Easy· 1/10	Plain HTTP client	—
comicbook.com	Entertainment	Easy· 1/10	Plain HTTP client	Cloudflare
setlist.fm	Entertainment	Easy· 1/10	Plain HTTP client	—
mtv.com	Entertainment	Medium· 3/10	Browser-impersonation HTTP (matched TLS)	Akamai Bot Manager
bandsintown.com	Entertainment	Medium· 3/10	Browser-impersonation HTTP (matched TLS)	Cloudflare
axs.com	Entertainment	Hard· 6/10	Headless browser	Cloudflare
stubhub.com	Entertainment	Easy· 1/10	Plain HTTP client	—
theonion.com	Entertainment	Easy· 1/10	Plain HTTP client	—
popsugar.com	Entertainment	Easy· 1/10	Plain HTTP client	—
boxofficemojo.com	Entertainment	Easy· 1/10	Plain HTTP client	—
usmagazine.com	Entertainment	Easy· 1/10	Plain HTTP client	—
ranker.com	Entertainment	Hard· 7/10	Headless browser	Cloudflare Turnstile + 1 more· 1 CAPTCHA
tvguide.com	Entertainment	Easy· 1/10	Plain HTTP client	Cloudflare
paypal.com	Finance & markets	Easy· 1/10	Plain HTTP client	Cloudflare
stripe.com	Finance & markets	Easy· 1/10	Plain HTTP client	—
tradingview.com	Finance & markets	Easy· 1/10	Plain HTTP client	—
bloomberg.com	Finance & markets	Hard· 7/10 varies by IP	Headless browser	PerimeterX (HUMAN)
bankofamerica.com	Finance & markets	Hard· 6/10	Headless browser	Akamai Bot Manager
fidelity.com	Finance & markets	Medium· 3/10	Browser-impersonation HTTP (matched TLS)	Akamai Bot Manager
wise.com	Finance & markets	Medium· 3/10	Browser-impersonation HTTP (matched TLS)	Cloudflare
robinhood.com	Finance & markets	Easy· 1/10	Plain HTTP client	—
marketwatch.com	Finance & markets	Hard· 6/10	Headless browser	DataDome
investing.com	Finance & markets	Medium· 3/10	Browser-impersonation HTTP (matched TLS)	Cloudflare
schwab.com	Finance & markets	Easy· 1/10	Plain HTTP client	—
squareup.com	Finance & markets	Easy· 1/10	Plain HTTP client	Cloudflare