LLMs.txt Dataset Facets API

Endpoint facts

Method

GET

API path

/datasets/llms/facets

Auth

apiKey

Credits

1 credit/request

Charged on

Successful 2xx responses only

Platform

Datasets

Execution

Stored dataset records. No live scraping, browser session or proxy routing.

Response schema

#/definitions/datasets.llmsFacetResponseDoc

How this endpoint behaves

Returns distribution counts over the AI-readiness census, honoring the same filters as search — adoption by producer, generator, TLD or category, and which AI crawlers robots.txt blocks. Site owners can ask for their domain to be removed at privacy@crawlora.net; a removed domain is deleted from the dataset and not scanned again.

Dataset id enum:

• llms

Query parameters:

• facet: required facet to aggregate. Allowed values: state, source_kind, generator, bot, policy, tld, category, robots_state, robots_default, blocked_crawler, partial_crawler, content_signal_train, full_state, home_status.
• q: optional substring match on the site domain, max 256 characters.
• text: optional full-text match on the /llms.txt title and summary, max 256 characters.
• state: optional /llms.txt state. Allowed values: present, absent, blocked, error, skipped. blocked and error mean unknown, not absent; skipped means the site's homepage control did not answer, so the file was not requested.
• valid: optional filter; true keeps only sites serving a valid /llms.txt (state present), false keeps the rest.
• source_kind: optional producer of a valid /llms.txt. Allowed values: shopify_agent_template, seo_plugin, platform_default, parked, custom.
• generator: optional generator detected in a valid /llms.txt. Allowed values: yoast, aioseo, rankmath, wix, shopify, parked.
• bot: optional AI crawler named inside a valid /llms.txt, e.g. GPTBot. Use facet=bot for the values.
• min_links: optional minimum number of links in a valid /llms.txt, 0 or greater.
• mentions_mcp: optional filter; true keeps only valid files that mention the Model Context Protocol.
• mentions_openapi: optional filter; true keeps only valid files that mention OpenAPI or Swagger.
• placeholder: optional filter; true keeps only valid files that still hold template placeholder text.
• injection: optional filter; true keeps only valid files containing instruction-override text aimed at AI readers.
• full_valid: optional filter; true keeps only sites serving a valid /llms-full.txt.
• robots_state: optional /robots.txt state. Allowed values: present, absent, blocked, error, skipped.
• robots_default: optional default stance of robots.txt toward AI crawlers. Allowed values: allowed, partial, blocked.
• blocks_all: repeatable AI crawler that robots.txt must block (AND). Allowed values: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot.
• blocks_any: repeatable AI crawler; robots.txt must block at least one (OR). Same allowed values as blocks_all.
• not_blocked: repeatable AI crawler that robots.txt must not block. Same allowed values as blocks_all.
• all_ai_blocked: optional filter; true keeps only sites whose robots.txt blocks all 17 tracked AI crawlers.
• min_train_blocked: optional minimum number of AI training crawlers (of 11 tracked) that robots.txt blocks, 0 or greater.
• content_signal_train: optional Content-Signal ai-train value declared in robots.txt. Allowed values: yes, no. A robots.txt that writes anything else there is treated as declaring neither.
• tld: optional exact top-level-domain filter, e.g. com, org, io.
• category: optional exact category label carried by the source list. The top-1M run has none, so the filter matches nothing and facet=category is empty for it.
• reachable: optional filter; true keeps only sites whose homepage control request answered, false only sites that did not.
• home_blocked: optional filter; true keeps only sites whose homepage was walled off from the scan (a block or challenge).
• is_infrastructure: optional filter; false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but serve no public homepage; true keeps only those.
• run_id: optional scan run id; defaults to the latest run.

How to read the states:

• present: a valid file was served. Validity is judged from the body, not from the status code or content type.
• absent: the site answered 404/410, or answered 200 with something that is not a file (a soft-404 page, HTML, an empty body).
• blocked: the site refused the request (401, 403, 429, a bot challenge). This is unknown, not absent.
• error: a transport failure or a 5xx. This is unknown, not absent.
• skipped: the homepage control request did not answer, so the path was not requested.
• Adoption rates should use present over present plus absent.

Response behavior:

• Returns up to 50 buckets, each with the facet value and its count, ordered by count.
• Counts include blocked and error probes, which are unknown rather than absent. Filter state=present or state=absent to count only sites that were established either way.
• policy values are rate_limit, copyright, no_training, citation: terms detected in a valid /llms.txt.
• Invalid facet values and invalid filter values return the standard invalid params envelope.
• Returns an empty items array (not an error) when nothing matches or the dataset has no data yet.
• Does not trigger live scraping.

Sampled response

Request, 14 Oct 2026

GET /datasets/llms/facets?facet=source_kind

Items in items

5

Fields filled in every checked item

2

Shared guides

Related