An AI crawler is an automated bot operated by an AI company — OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended — that fetches web pages either to train a model or to answer a live user query. Sites now detect and block these separately from traditional search-engine crawlers, often by user agent.
The category splits into two distinct behaviors that sites treat very differently. Training crawlers — GPTBot, ClaudeBot, Google-Extended, CCBot — fetch content to include in a model's training data, a one-time or periodic use with no direct traffic benefit back to the site. Retrieval crawlers — OAI-SearchBot, Claude-SearchBot, PerplexityBot — fetch a page in real time to answer a specific user question, closer to how a search engine crawls, and can drive a citation or click back to the source.
Because each identifies itself with its own user agent, robots.txt can block one category while allowing the other — a common pattern in 2026 is disallowing training bots while leaving retrieval bots (and traditional search crawlers) untouched, trading away training-data inclusion for AI-search visibility.
Cloudflare — sitting in front of a large share of the web — announced that starting September 15, 2026, new domains and untouched free-tier sites default to blocking AI Agent and Training crawlers on ad-supported pages, while Search crawlers stay allowed by default; existing paying customers keep their current configuration unless they opt in. The shift reflects scale: AI training traffic reportedly grew from roughly a fifth to over half of all crawler requests on Cloudflare's network within a year.
Not every crawler cooperates — some have been caught rotating user agents and IPs to evade no-crawl directives, which is why verification programs (Cloudflare's Verified Bot list, signed request headers) are emerging alongside robots.txt as a way to prove crawler identity rather than just claim it.
How Crawlora handles this
Crawlora's own scan tracks how AI-crawler blocking is spreading across the web, and the same identity-consistency work that gets a request past traditional anti-bot systems — coherent fingerprints, appropriate IP type — applies when the crawler being detected is an AI bot rather than a human-imitating scraper.
Glossary
FAQ
Yes — training bots (GPTBot, ClaudeBot, Google-Extended) and retrieval bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) use distinct user agents, so robots.txt can disallow one category while allowing the other.
The major, publicly-documented ones generally do. But robots.txt is voluntary, and some crawlers have been caught rotating user agents and IPs specifically to bypass disallow rules — which is why identity-verification programs are emerging alongside the file as a stronger signal.
Volume and value: AI training traffic has grown to a large share of all crawler requests on some networks, and site owners increasingly want to be compensated for or opted into that use, rather than have their content ingested by default the way search indexing traditionally worked.
Beyond AI Crawler, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.