Cloudflare Blocks AI Crawlers by Default in 2026: What Changed and What to Do

Cloudflare sits in front of roughly a fifth of the web. On July 1, 2026 it announced the next step in a policy it started a year earlier: AI crawlers no longer get a blanket "block or allow" — they get sorted into categories, and two of the three now default to blocked on the pages that make a site money.

If you run any kind of scraper, data pipeline, or AI agent that touches the open web, this is worth understanding — not because it targets malicious scraping specifically, but because the new defaults don't distinguish "AI company training a model" from "developer running a legitimate research or monitoring job" unless you can prove which one you are.

What actually changed

Cloudflare's post — Content Independence Day: the next chapter — splits AI bot traffic into three categories, each with a precise definition:

• Search — "any behavior that collects or indexes your content, so it can answer questions about it later." The classic deal: an index gets built, the site gets referral traffic back. Stays allowed by default.
• Agent — "automated behavior that is acting, usually in real time, on a person's behalf, to get something done right now" — a ChatGPT-User-style fetch, or a browser-use agent like Gemini or Claude operating on someone's behalf mid-session.
• Training — a crawler taking content to train or fine-tune a model, where "your data is permanently absorbed into the underlying architecture" rather than referenced transiently.

Starting September 15, 2026, Agent and Training bots default to blocked on ad-supported pages — pages a site owner built to be seen and monetized by a human visitor. Search bots are unaffected. Crucially, this default applies to:

• domains new to Cloudflare after that date,
• new sites added by existing customers, and
• free-tier accounts that haven't touched the setting.

Existing paying customers keep whatever they've already configured — this is a new-signup default, not a retroactive block across Cloudflare's whole footprint. Multi-purpose crawlers (Googlebot, Bingbot, Applebot — all of which now do double duty as search and AI-training crawlers) get held to the most restrictive applicable rule across whatever functions they combine.

There's also a genuinely useful new piece of infrastructure for anyone who wants to express a data-use preference, not just get blocked or allowed as a category: an extended Content-Signal robots.txt directive, now included by default in Cloudflare's managed robots.txt:

Content-Signal: search=yes,ai-train=no,use=reference

The new use= parameter is the interesting addition — use=immediate ("interact, but store and reuse nothing"), use=reference (the default — "index, excerpt, and link back"), or use=full ("summarize and reproduce"). It's a declared preference, not an enforced block — enforcement happens at Cloudflare's edge, not via robots.txt compliance — but it's the clearest public signal yet of how site owners want their content used.

Why Cloudflare did this

The technical change is small; the numbers behind it explain the motivation. Cloudflare's same-day companion report, Agentic internet: one year of bot report data, lays out the trend it's reacting to:

• 52% of crawler requests are now for AI training, up from 22% in spring 2025.
• More than half of all internet traffic is now non-human.
• Some of the most heavily crawled content categories have seen human traffic decline as much as 40% in under a year.
• Google still drives ~88% of referral traffic and holds roughly 2x more information than any single AI competitor — the "Google Zero" scenario (a world where search sends publishers little to no traffic) is what publishers are reportedly bracing for.

This builds on Cloudflare's first Content Independence Day (July 1, 2025), when it made "block AI bots" the default for new domains and launched Pay Per Crawl — an HTTP 402 Payment Required mechanic where an unpaid crawler gets a 402 with a crawler-price header, can counter-offer or accept, and a successful paid fetch returns 200 with crawler-charged. Cloudflare acts as merchant of record and pays publishers monthly. One year in, Cloudflare says it has evolved that into a broader Pay Per Use marketplace (launch partners Ceramic.ai and You.com), though neither Cloudflare's post nor the coverage discloses pricing or Cloudflare's cut.

You'll also see a widely-cited figure — "416 billion AI bot requests blocked" — attached to this story in some coverage. That number is real, but it's from a December 2025 interview with Cloudflare CEO Matthew Prince, describing the first six months after the 2025 launch — not a statistic from the July 2026 announcement itself. Worth knowing so you don't mix up which policy generation produced which number.

The part that matters if you're not the target

Cloudflare's framing is aimed at AI companies training models and agents fetching pages on autopilot. But the enforcement mechanism doesn't read intent — it reads identity, and proving identity is now a permissioned process, not a header you can set yourself:

• Verified Bot program — bot operators enrol and commit to "honest self-representation" of their identity and purpose. This is the primary route to being recognized as a specific, trusted category instead of lumped in as unverified traffic.
• BotBase — a new public, searchable directory of known bots and agents, which site owners can reference directly in their own custom security rules.
• Forwarded header self-declaration (Forwarded: for="openai";use="reference", per RFC 7239) — lets an intermediary declare who it's acting for and why, enabling "transitive trust" through proxies or agent frameworks.

None of this is a static allowlist a third-party scraper can just adopt. If your traffic isn't individually verified and you're hitting a new-to-Cloudflare or untouched free-tier site after September 15, the honest read is: your legitimate research, price-monitoring, or dataset-building job risks the same default block as an actual AI training crawler, unless you're either respecting the site's declared Content-Signal and staying within it, or running through infrastructure that already handles the identity layer for you.

For the deeper mechanics of how Cloudflare (and DataDome, PerimeterX/HUMAN) fingerprint and block requests in the first place — IP reputation, TLS/HTTP fingerprinting, and the JS sensor layer — see scraping sites that block bots; this post is about the policy layer sitting on top of that.

How to keep collecting public data

Ranked by what's realistic for most teams:

1. Respect the site's declared Content-Signal where you can read it

If a site publishes ai-train=no or use=immediate, that's an explicit statement of intent from the owner — a good-faith scraper should honor it, both because it's the right thing to do and because it's the clearest evidence you're not the traffic this policy is aimed at.

2. Register as a Verified Bot, if you operate at real scale

Worth it if you run a large, identifiable crawling operation and want first-class recognition instead of relying on generic unblocking. It's a genuine commitment (accurate self-representation, ongoing compliance), not a checkbox — most small teams and one-off research jobs won't clear the bar for it to be worth the process.

3. A managed unblocking API (Crawlora)

For the common case — you need public data from sites that increasingly default to blocking unverified traffic, and you don't want to build or maintain the proxy rotation, fingerprinting, and identity-negotiation layer yourself — Crawlora's web scraping API and /web/scrape endpoint handle the access layer behind one key:

curl -X POST "https://api.crawlora.net/api/v1/web/scrape" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown"], "render": "auto"}'

import requests

resp = requests.post(
    "https://api.crawlora.net/api/v1/web/scrape",
    headers={"x-api-key": "YOUR_API_KEY"},
    json={"url": "https://example.com", "formats": ["markdown"], "render": "auto"},
)
print(resp.json()["data"]["markdown"][:500])

render: "auto" escalates from a plain fetch to a real browser only when a site demands it — the same escalation pattern that gets through Cloudflare, DataDome, and PerimeterX today. This doesn't get you Verified Bot status; it's the practical alternative for teams collecting public data at a scale too small to justify their own verification program, but real enough that "just try again from a different IP" stops working.

A note on compliance

None of this changes the underlying rules, it just raises the technical and identity bar for enforcing them. Whatever route you take:

• Collect only public data you're authorized to access — a site's Content-Signal and Terms of Service still apply regardless of whether you can technically get past the block.
• Treat scraped personal data (names, reviews, profiles) under GDPR/CCPA the same way you would from any other source — minimize what you store, have a lawful basis.
• If your use case is AI training specifically, this policy is aimed squarely at you — licensing or a declared, compliant path is the durable answer, not working around a block.

This is not legal advice — see is web scraping legal in 2026? for the broader public-vs-personal-data treatment.

Sources

Where this fits

This is the policy layer; for the mechanics of actually getting through Cloudflare, DataDome, and PerimeterX/HUMAN — the fingerprinting signals and the escalation pattern that works reliably — see scraping sites that block bots. For the scale of what's already blocked across the web, see the Anti-Bot Index, Crawlora's open dataset tracking bot-defense adoption across the top sites. To pick a broader toolkit, see how to choose a web scraping API.

Try it first, free: run any public URL through the Free Web Scraper, or check whether a site blocks bots with the Anti-Bot Checker — no signup. Test the endpoint in the Playground, and review pricing.