LLMs.txt Dataset Search API

Endpoint facts

Method

GET

API path

/datasets/llms/search

Auth

apiKey

Credits

1 credit/record returned · minimum 2

Charged on

Successful 2xx responses only

Platform

Datasets

Execution

Stored dataset records. No live scraping, browser session or proxy routing.

Response schema

#/definitions/datasets.llmsSearchResponseDoc

How this endpoint behaves

Searches the AI-readiness census of the top 1M sites: one record per site describing whether it serves a real /llms.txt or /llms-full.txt (who produced it, what it links to and mentions) and what its /robots.txt says to 17 AI crawlers, next to a homepage control request from the same session. Search results omit the file bodies; read one site with the item endpoint. Site owners can ask for their domain to be removed at privacy@crawlora.net; a removed domain is deleted from the dataset and not scanned again.

Dataset id enum:

• llms

Query parameters:

• q: optional substring match on the site domain, max 256 characters.
• text: optional full-text match on the /llms.txt title and summary, max 256 characters.
• state: optional /llms.txt state. Allowed values: present, absent, blocked, error, skipped. blocked and error mean unknown, not absent; skipped means the site's homepage control did not answer, so the file was not requested.
• valid: optional filter; true keeps only sites serving a valid /llms.txt (state present), false keeps the rest.
• source_kind: optional producer of a valid /llms.txt. Allowed values: shopify_agent_template, seo_plugin, platform_default, parked, custom.
• generator: optional generator detected in a valid /llms.txt. Allowed values: yoast, aioseo, rankmath, wix, shopify, parked.
• bot: optional AI crawler named inside a valid /llms.txt, e.g. GPTBot. Use facet=bot for the values.
• min_links: optional minimum number of links in a valid /llms.txt, 0 or greater.
• mentions_mcp: optional filter; true keeps only valid files that mention the Model Context Protocol.
• mentions_openapi: optional filter; true keeps only valid files that mention OpenAPI or Swagger.
• placeholder: optional filter; true keeps only valid files that still hold template placeholder text.
• injection: optional filter; true keeps only valid files containing instruction-override text aimed at AI readers.
• full_valid: optional filter; true keeps only sites serving a valid /llms-full.txt.
• robots_state: optional /robots.txt state. Allowed values: present, absent, blocked, error, skipped.
• robots_default: optional default stance of robots.txt toward AI crawlers. Allowed values: allowed, partial, blocked.
• blocks_all: repeatable AI crawler that robots.txt must block (AND). Allowed values: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot.
• blocks_any: repeatable AI crawler; robots.txt must block at least one (OR). Same allowed values as blocks_all.
• not_blocked: repeatable AI crawler that robots.txt must not block. Same allowed values as blocks_all.
• all_ai_blocked: optional filter; true keeps only sites whose robots.txt blocks all 17 tracked AI crawlers.
• min_train_blocked: optional minimum number of AI training crawlers (of 11 tracked) that robots.txt blocks, 0 or greater.
• content_signal_train: optional Content-Signal ai-train value declared in robots.txt. Allowed values: yes, no. A robots.txt that writes anything else there is treated as declaring neither.
• tld: optional exact top-level-domain filter, e.g. com, org, io.
• category: optional exact category label carried by the source list. The top-1M run has none, so the filter matches nothing and facet=category is empty for it.
• reachable: optional filter; true keeps only sites whose homepage control request answered, false only sites that did not.
• home_blocked: optional filter; true keeps only sites whose homepage was walled off from the scan (a block or challenge).
• is_infrastructure: optional filter; false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but serve no public homepage; true keeps only those.
• run_id: optional scan run id; defaults to the latest run.
• sort: optional sort order. Allowed values: relevance, rank_asc, links_desc, bytes_desc, domain_asc, crawled_desc. Defaults to relevance with q or text, otherwise rank_asc.
• page: optional page number, defaults to 1.
• page_size: optional page size, defaults to 20, max 100; page * page_size must be <= 10000.

How to read the states:

• present: a valid file was served. Validity is judged from the body, not from the status code or content type.
• absent: the site answered 404/410, or answered 200 with something that is not a file (a soft-404 page, HTML, an empty body).
• blocked: the site refused the request (401, 403, 429, a bot challenge). This is unknown, not absent.
• error: a transport failure or a 5xx. This is unknown, not absent.
• skipped: the homepage control request did not answer, so the path was not requested.
• Adoption rates should use present over present plus absent.

Response behavior:

• Each item carries the site (domain, tld, rank, category, is_infrastructure), the homepage control (reachable, home_status, home_blocked, home_challenge), the /llms.txt result in llms and /llms-full.txt in full (state, status, bytes, kind, valid, source_kind, generator, title, summary, sections, links, MCP/OpenAPI mentions, named bots, policy terms), the /robots.txt result in robots (state, default_stance, per-crawler stance in crawlers, train_blocked, all_ai_blocked, content_signal, sitemaps), and provenance (run_id, run_date, probed_at).
• robots.crawlers covers: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Google-Extended, PerplexityBot, Perplexity-User, CCBot, Bytespider, Applebot-Extended, Amazonbot, meta-externalagent, cohere-ai, Diffbot.
• Invalid enum values and invalid ranges return the standard invalid params envelope.
• Returns an empty items array (not an error) when nothing matches or the dataset has no data yet.
• Does not trigger live scraping.

Billing

1 credit per record returned, minimum 2. A search that returns 20 records costs 20 credits, a full page of 100 costs 100, and an empty or one-record result costs 2. You are charged for what comes back, not for the page_size you ask for, but your balance must cover the requested page_size for the call to be accepted. The response header x-credit-cost shows what a call was charged.

Sampled response

Request, 14 Oct 2026

GET /datasets/llms/search?page_size=20&state=present

Items in items

20

Fields filled in every checked item

18

Shared guides

Related