AI readiness · llms.txt and robots.txt · Scan of October 11, 2026
A census of a top-ranked site list: 1,000,000 sites, one record each saying whether the site serves a real /llms.txt and /llms-full.txt, who produced the file, what it links to, and what its /robots.txt asks of the main AI crawlers. A homepage request from the same session separates a missing file from a site that answered nothing. Every call reads the stored index; nothing is fetched on request.
92,631
A file counts as present only when its body is a real llms.txt. A 200 response carrying an HTML page or a soft-404 is recorded as absent.
Scan of October 11, 2026. Rates use sites that gave an answer; sites that refused or failed are counted separately as unknown.
13.3%
of the sites that answered serve a valid llms.txt — 92,631 of 696,270. Another 102,270 sites refused the request or failed, so for them the answer is unknown, not no.
48.4%
of those files were written for the site. The rest were emitted by software: 28,319 come from one storefront template and 15,526 from SEO plugins, so a file's presence is weak evidence that anyone at the company chose to publish it.
20.2%
of sites with a readable robots.txt ask at least one AI training crawler to stay out, and 17.2% block GPTBot, the most-blocked of the 17 crawlers tracked. 19,240 sites block all 17.
6,874
sites do both at once: they publish llms.txt and block at least one AI training crawler — 7.4% of the sites with a file. Inviting AI readers and refusing AI training are separate decisions.
Each file gets one of five states. `present` and `absent` are answers. `blocked` means the site refused the request with a 401, 403, 429 or a bot challenge, and `error` means the connection failed or the server returned a 5xx: in both cases the file may exist, and the record says so instead of guessing. `skipped` means the homepage did not answer either, so the file was never requested. An adoption rate is therefore present over present plus absent — 92,631 over 696,270 here — and never present over all 1,000,000 sites, which would count every unreachable and refusing site as a no.
Of 1,000,000 sites, 798,540 answered a homepage request. The table gives the outcome for both files; a site lands in exactly one state per file.
| State | Meaning | /llms.txt | /llms-full.txt |
|---|---|---|---|
present | A valid file was served | 92,631 | 36,288 |
absent | No file: a 404 or 410, or a 200 carrying something that is not the file | 603,639 | 657,388 |
blocked | Request refused or challenged — unknown | 77,579 | 78,395 |
error | Connection failed or server error — unknown | 24,691 | 26,469 |
skipped | Homepage did not answer, file not requested | 201,460 | 201,460 |
Each valid file is classified by what produced it, from markers in the body. `custom` is the residue: no known template, plugin or builder signature was found, which is the closest the data gets to a file someone wrote. The buckets sum to the 92,631 valid files.
| source_kind | What it means | Sites | Share of valid files |
|---|---|---|---|
custom | No template or plugin signature — written for the site | 44,801 | 48.4% |
shopify_agent_template | A storefront platform's agent-instructions template | 28,319 | 30.6% |
seo_plugin | Emitted by an SEO plugin | 15,526 | 16.8% |
platform_default | A site builder's default file | 3,189 | 3.4% |
parked | A parked or placeholder domain's file | 796 | 0.9% |
Where the body names the tool that wrote it, the record carries it in `generator`. A file with no such marker has none, so these counts do not sum to the valid files.
| generator | Sites |
|---|---|
shopify | 29,584 |
yoast | 8,524 |
aioseo | 4,303 |
rankmath | 2,699 |
wix | 1,258 |
parked | 796 |
Each row is a filter on the search endpoint, so every count can be reproduced and narrowed. Shares are of the 92,631 valid files.
| Signal | Filter | Sites | Share |
|---|---|---|---|
| Contains at least one link | min_links=1 | 84,498 | 91.2% |
| Site also serves a valid /llms-full.txt | full_valid=true | 35,953 | 38.8% |
| Mentions the Model Context Protocol | mentions_mcp=true | 34,156 | 36.9% |
| Mentions OpenAPI or Swagger | mentions_openapi=true | 1,554 | 1.7% |
| Still holds template placeholder text | placeholder=true | 3,497 | 3.8% |
| Contains instruction-override text aimed at AI readers | injection=true | 3 | 0% |
27,661 of the 34,156 Model Context Protocol mentions (81%) come from the storefront template, which names it in its boilerplate; 4,052 are in hand-written files. The body of a file flagged for instruction-override text is withheld from the item endpoint.
543,319 sites served a readable robots.txt. Each crawler's stance is resolved the way a crawler would read the file: its own group if it has one, the wildcard group otherwise. A crawler counts as blocked when the root path is disallowed for it. One file can block many crawlers, so the rows overlap and do not sum to a total.
109,860 of 543,319
19,240 of 543,319
| Crawler | Sites blocking it | Share of readable robots.txt |
|---|---|---|
GPTBot | 93,467 | 17.2% |
ClaudeBot | 86,633 | 15.9% |
Amazonbot | 85,121 | 15.7% |
Google-Extended | 81,935 | 15.1% |
anthropic-ai | 80,609 | 14.8% |
Applebot-Extended | 79,425 | 14.6% |
meta-externalagent | 79,030 | 14.5% |
ChatGPT-User | 78,249 | 14.4% |
cohere-ai | 77,868 | 14.3% |
Diffbot | 76,389 | 14.1% |
PerplexityBot | 75,364 | 13.9% |
OAI-SearchBot | 69,167 | 12.7% |
Claude-SearchBot | 66,627 | 12.3% |
Perplexity-User | 66,559 | 12.3% |
Claude-User | 66,421 | 12.2% |
CCBot | 50,305 | 9.3% |
Bytespider | 47,081 | 8.7% |
robots.txt is a request to crawlers, not an access control, and this dataset records what the file asks, not whether any crawler complies. 6,019 sites also declare `ai-train=no` in a Content-Signal line and 1,900 declare `ai-train=yes`.
The twelve most common top-level domains in the list, each with its own rate: valid files over sites that answered for that TLD.
| TLD | Sites that answered | With llms.txt | Adoption |
|---|---|---|---|
.com | 306,776 | 50,987 | 16.6% |
.net | 27,487 | 1,852 | 6.7% |
.org | 29,238 | 2,249 | 7.7% |
.ru | 23,320 | 1,854 | 8% |
.uk | 21,956 | 1,854 | 8.4% |
.de | 22,339 | 2,294 | 10.3% |
.br | 13,556 | 2,277 | 16.8% |
.jp | 12,792 | 834 | 6.5% |
.nl | 11,408 | 877 | 7.7% |
.in | 8,825 | 1,941 | 22% |
.fr | 7,682 | 937 | 12.2% |
.io | 7,064 | 1,500 | 21.2% |
A TLD is where a name is registered, not where a company is based, and the list ranks sites by traffic rather than sampling them evenly. Read these as differences inside this list.
One record per site and scan. Search results omit the file bodies; the item endpoint returns them, capped in size.
domaintldrankis_infrastructurereachablehome_statushome_blockedhome_challengellms.statellms.validllms.source_kindllms.generatorllms.titlellms.summaryllms.linksllms.sectionsfull.statefull.validfull.bytesrobots.staterobots.default_stancerobots.crawlersrobots.train_blockedrobots.all_ai_blockedrobots.content_signalrobots.sitemapsrun_idrun_dateprobed_atCombine filters freely and page with `page` and `page_size`. Facets accept the same filters as search, so any count on this page can be reproduced for one TLD, one producer or one crawler.
Search — hand-written files that mention MCP
curl "https://api.crawlora.net/api/v1/datasets/llms/search?valid=true&source_kind=custom&mentions_mcp=true&page_size=50" \
-H "x-api-key: $CRAWLORA_API_KEY"Item — one site, with file bodies
curl "https://api.crawlora.net/api/v1/datasets/llms/items/example.com" \
-H "x-api-key: $CRAWLORA_API_KEY"Facets — crawlers blocked by sites that publish llms.txt
curl "https://api.crawlora.net/api/v1/datasets/llms/facets?facet=blocked_crawler&valid=true" \
-H "x-api-key: $CRAWLORA_API_KEY"In this scan of 1,000,000 top-ranked sites on October 11, 2026, 92,631 served a valid /llms.txt. That is 13.3% of the 696,270 sites that gave a clear answer. A further 102,270 sites refused the request or failed, and 201,460 did not answer a homepage request at all, so they are counted as unknown rather than as sites without a file.
48.4% look that way. 44,801 of the 92,631 valid files carry no template or plugin signature. 28,319 come from a storefront platform's agent-instructions template, 15,526 from SEO plugins, and the rest from site-builder defaults and parked domains. Use `source_kind=custom` to keep only the files that look hand-written.
Among the 543,319 sites with a readable robots.txt, 93,467 (17.2%) disallow the whole site for GPTBot and 86,633 (15.9%) for ClaudeBot, counting both rules that name the crawler and wildcard rules that cover it. 109,860 sites block at least one of the training crawlers tracked and 19,240 block all 17 crawlers. The file is a request; the dataset does not measure whether crawlers honour it.
/llms.txt is an index: a short Markdown file with a title, a summary and links to the pages an AI reader should use. /llms-full.txt is the same idea with the content inlined, so a model can read the documentation in one request. 36,288 sites serve a valid /llms-full.txt, and 35,953 serve both files.
Because the request was refused, and a refusal says nothing about the file. Sites behind a bot challenge or a rate limit answer 403 or 429 to every path, including ones that exist. Recording those as absent would understate adoption, so they are kept apart: `state=blocked` and `state=error` mean unknown. Filter with `state=present` or `state=absent` when you need sites with a definite answer.
Yes. A site owner can ask at privacy@crawlora.net. The domain is deleted from the dataset and is not scanned again. The dataset reads only three public files and the homepage of each site, and file bodies containing instruction-override text are withheld.
Track llms.txt adoption in a market, find the sites that publish an MCP server, or audit which AI crawlers a list of domains turns away — from one REST API or the same endpoints as MCP tools.