Endpoint Playground
Searches the AI-readiness census of the top 1M sites (dataset id enum value `llms`) — one record per site describing whether it serves a real `/llms.txt` or `/llms-full.txt` (who produced it, what it links to and mentions) and what its `/robots.txt` says to 17 AI crawlers, next to a homepage control request. Search results omit the file bodies; read one site with the item endpoint. `blocked` and `error` states mean unknown, never absent. Sort enum: `relevance`, `rank_asc`, `links_desc`, `bytes_desc`, `domain_asc`, `crawled_desc`. state and robots_state enum: `present`, `absent`, `blocked`, `error`, `skipped`. source_kind enum: `shopify_agent_template`, `seo_plugin`, `platform_default`, `parked`, `custom`. generator enum: `yoast`, `aioseo`, `rankmath`, `wix`, `shopify`, `parked`. robots_default enum: `allowed`, `partial`, `blocked`. content_signal_train enum: `yes`, `no`. Crawler enum for blocks_all, blocks_any and not_blocked: `GPTBot`, `OAI-SearchBot`, `ChatGPT-User`, `ClaudeBot`, `Claude-SearchBot`, `Claude-User`, `anthropic-ai`, `Google-Extended`, `PerplexityBot`, `Perplexity-User`, `CCBot`, `Bytespider`, `Applebot-Extended`, `Amazonbot`, `meta-externalagent`, `cohere-ai`, `Diffbot`. Site owners can ask for removal at privacy@crawlora.net. Billing: this search costs 1 credit per record returned, with a minimum of 2 (an empty result costs 2, a full page of 100 costs 100). Dataset Playground requests query indexed data and do not apply proxy routing.
14 Oct 2026: 20 items; 18 fields had a value in every checked item.
Sample value profiles: probed_at: 2026-10-11; rank: 3 to 175, median 120; origin: cloudflare.com and 19 other hosts.
curl "https://api.crawlora.net/api/v1/datasets/llms/search?state=present&source_kind=shopify_agent_template&generator=yoast&robots_state=present&robots_default=allowed&content_signal_train=yes&sort=relevance" \
-H "x-api-key: $CRAWLORA_API_KEY"| Parameter | Type | Required | Prefilled value | Description |
|---|---|---|---|---|
| q | string | No | Substring match on the site domain, max 256 characters | |
| text | string | No | Full-text match on the /llms.txt title and summary, max 256 characters | |
| state | string | No | present | /llms.txt state. Enum: present, absent, blocked, error, skipped. blocked and error mean unknown, not absent |
| valid | boolean | No | true keeps only sites serving a valid /llms.txt, false keeps the rest | |
| source_kind | string | No | shopify_agent_template | Producer of a valid /llms.txt. Enum: shopify_agent_template, seo_plugin, platform_default, parked, custom |
| generator | string | No | yoast | Generator detected in a valid /llms.txt. Enum: yoast, aioseo, rankmath, wix, shopify, parked |
| bot | string | No | AI crawler named inside a valid /llms.txt, e.g. GPTBot (see facet=bot for the values) | |
| min_links | integer | No | Minimum number of links in a valid /llms.txt, 0 or greater | |
| mentions_mcp | boolean | No | true keeps only valid /llms.txt files that mention the Model Context Protocol | |
| mentions_openapi | boolean | No | true keeps only valid /llms.txt files that mention OpenAPI or Swagger | |
| placeholder | boolean | No | true keeps only valid /llms.txt files that still hold template placeholder text | |
| injection | boolean | No | true keeps only valid /llms.txt files containing instruction-override text aimed at AI readers | |
| full_valid | boolean | No | true keeps only sites serving a valid /llms-full.txt | |
| robots_state | string | No | present | /robots.txt state. Enum: present, absent, blocked, error, skipped |
| robots_default | string | No | allowed | Default stance of robots.txt toward AI crawlers. Enum: allowed, partial, blocked |
| blocks_all | array | No | Repeatable AI crawler that robots.txt must block (AND). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| blocks_any | array | No | Repeatable AI crawler; robots.txt must block at least one (OR). Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| not_blocked | array | No | Repeatable AI crawler that robots.txt must NOT block. Enum: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot | |
| all_ai_blocked | boolean | No | true keeps only sites whose robots.txt blocks all 17 tracked AI crawlers | |
| min_train_blocked | integer | No | Minimum number of AI training crawlers (of 11 tracked) that robots.txt blocks, 0 or greater | |
| content_signal_train | string | No | yes | Content-Signal ai-train value declared in robots.txt. Enum: yes, no |
| tld | string | No | Exact top-level-domain filter, e.g. com, org, io | |
| category | string | No | Exact category label carried by the source list; the top-1M run has none, so this is empty for it | |
| reachable | boolean | No | true keeps only sites whose homepage control request answered, false only sites that did not | |
| home_blocked | boolean | No | true keeps only sites whose homepage was walled off from the scan (a block or challenge) | |
| is_infrastructure | boolean | No | false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but serve no public homepage; true keeps only those | |
| run_id | string | No | Scan run id; defaults to the latest run | |
| sort | string | No | relevance | Sort enum: relevance, rank_asc, links_desc, bytes_desc, domain_asc, crawled_desc |
| page | integer | No | Page number, defaults to 1 | |
| page_size | integer | No | Page size, defaults to 20 and maxes at 100; page * page_size must be <= 10000 |
{
"code": 200,
"msg": "OK",
"data": {
"dataset": "llms",
"items": [
{
"domain": "example.com",
"tld": "com",
"rank": 1204,
"reachable": true,
"llms": {
"state": "present",
"status": 200,
"bytes": 4120,
"valid": true,
"source_kind": "custom",
"title": "Example Docs",
"links": 18
},
"full": {
"state": "absent",
"status": 404
},
"robots": {
"state": "present",
"default_stance": "allowed",
"train_blocked": 3,
"all_ai_blocked": false
},
"run_id": "top1m-2026-10",
"probed_at": "2026-10-20T04:11:09Z"
}
],
"page": 1,
"page_size": 20,
"total": 1,
"sort": "rank_asc"
}
}The sample response, schema and code snippets below work without an account. Sending a live request needs an API key.