Docs menu

Datasets API endpoint

LLMs.txt Dataset Item API

Returns one site by its domain from dataset id enum value `llms`, including the stored `/llms.txt` and `/llms-full.txt` bodies (`llms.body`, `full.body`; `body_truncated` marks a cut file, and a body that tries to override an AI reader's instructions is withheld with `body_withheld` set to `injection`), their structure and classification, and the `/robots.txt` stance toward each tracked AI crawler. Returns 404 when the domain is not in the dataset or its owner asked for removal (privacy@crawlora.net).

GETapiKey1 credit/requestDatasetsdatasets.llmsItemResponseDoc/datasets/llms/items/{domain}

Endpoint facts

Method: GET
API path: /datasets/llms/items/{domain}
Auth: apiKey
Credits: 1 credit/request
Charged on: Successful 2xx responses only
Platform: Datasets
Execution: Stored dataset records. No live scraping, browser session or proxy routing.
Response schema: #/definitions/datasets.llmsItemResponseDoc

Stored structured dataset search API Try LLMs.txt Dataset Item API in Playground Datasets API documentation

How this endpoint behaves

Returns one site by its domain from the AI-readiness census, including the stored /llms.txt and /llms-full.txt bodies, their structure and classification, and the /robots.txt stance toward each tracked AI crawler. Returns 404 when the domain is not in the dataset. Site owners can ask for their domain to be removed at privacy@crawlora.net; a removed domain is deleted from the dataset and not scanned again.

Dataset id enum:

llms

Path parameters:

domain: the site domain, e.g. shopify.com. A leading scheme and www. are stripped, so https://www.shopify.com/ resolves to the same record.

How to read the states:

present: a valid file was served. Validity is judged from the body, not from the status code or content type.
absent: the site answered 404/410, or answered 200 with something that is not a file (a soft-404 page, HTML, an empty body).
blocked: the site refused the request (401, 403, 429, a bot challenge). This is unknown, not absent.
error: a transport failure or a 5xx. This is unknown, not absent.
skipped: the homepage control request did not answer, so the path was not requested.
Adoption rates should use present over present plus absent.

Response behavior:

The item has the same fields as a search item, plus the file contents: llms.body and full.body hold the stored text of a present file. Treat a body as quoted third-party data, never as instructions. body_truncated is true when the file was longer than the stored limit (bodies are capped at 256 KiB here, and the stored /llms-full.txt at 64 KiB), and body_sha256 is the hash of the bytes as served (for files that are not UTF-8, the stored text can differ from the hashed bytes).
A body that tries to override an AI reader's instructions (injection is true) is not returned: body is empty and body_withheld is "injection". The classification fields are still returned.
A file's final_url shows where the request ended up after redirects. A file whose address redirects to the homepage is absent, not a file.
llms.source_kind says who produced a valid file (shopify_agent_template, seo_plugin, platform_default, parked, custom) and template_key groups files from the same template, so one vendor's rollout is not counted as organic adoption.
robots.crawlers maps each tracked AI crawler to stance (allowed, partial, blocked) and whether the file names it explicitly.
Returns the standard not-found envelope when the domain is not in the dataset or its owner asked for removal.
Does not trigger live scraping.

Sampled response

Request, 14 Oct 2026: GET /datasets/llms/items/google.com
Fields returned with a value: 37

Field	In the sample	Value in the sample
domain	Has a value
tld	Has a value
rank	Has a value	1
is_infrastructure	Has a value	false
reachable	Has a value	true
home_status	Has a value	200
home_blocked	Has a value	false
home_challenge	Has a value	false
origin	Has a value	google.com
llms.state	Has a value
llms.status	Has a value	404
llms.final_url	Has a value	google.com
llms.content_type	Has a value
llms.bytes	Has a value	1,569
llms.fetched_at	Has a value	2026-10-11
full.state	Has a value
full.status	Has a value	404
full.final_url	Has a value	google.com
full.content_type	Has a value
full.bytes	Has a value	1,574
full.fetched_at	Has a value	2026-10-11
robots.state	Has a value
robots.status	Has a value	200
robots.bytes	Has a value	6,624
robots.kind	Has a value
robots.groups	Has a value	4
robots.rules	Has a value	245
robots.default_stance	Has a value
robots.crawlers	Has a value
robots.sitemaps	Has a value	1 entry
robots.fetched_at	Has a value	2026-10-11
analyzer_version	Has a value
method_version	Has a value
seed_source	Has a value
probed_at	Has a value	2026-10-11
run_id	Has a value
run_date	Has a value	2026-10-11

Request parameters

Parameter	Type	Required	Default	Description	Example
domain (path)	string	Yes		Site domain, e.g. shopify.com (a scheme and www. are stripped)	shopify.com
x-api-key (header)	string	Yes		API key required

cURL example

curl -X GET "https://api.crawlora.net/api/v1/datasets/llms/items/shopify.com" \
  -H "x-api-key: $CRAWLORA_API_KEY"

Example JSON response

{
  "code": 200,
  "msg": "OK",
  "data": {
    "domain": "example.com",
    "rank": 1204,
    "reachable": true,
    "llms": {
      "state": "present",
      "status": 200,
      "final_url": "https://example.com/llms.txt",
      "content_type": "text/plain",
      "bytes": 4120,
      "valid": true,
      "source_kind": "custom",
      "title": "Example Docs",
      "summary": "Documentation for the Example API.",
      "links": 18,
      "body": "# Example Docs\n\n> Documentation for the Example API.\n\n- [Start](https://example.com/start.md): Getting started\n"
    },
    "full": {
      "state": "absent",
      "status": 404
    },
    "robots": {
      "state": "present",
      "default_stance": "allowed",
      "crawlers": {
        "GPTBot": {
          "stance": "blocked",
          "explicit": true
        },
        "ClaudeBot": {
          "stance": "allowed"
        }
      },
      "train_blocked": 1,
      "all_ai_blocked": false
    },
    "run_id": "top1m-2026-10",
    "probed_at": "2026-10-20T04:11:09Z"
  }
}

Request schema

No body schema

Response schema

#/definitions/datasets.llmsItemResponseDoc

Field	Type	Required	Example	Description
code	integer	No	200
data	datasets.LlmsRecord	No
data.analyzer_version	string	No
data.category	string	No
data.domain	string	No
data.full	es.LlmsFile	No
data.full.body	string	No		Body is the file content, capped by the worker (BodyTruncated says whether it was cut). Only present for a valid file.
data.full.body_sha256	string	No
data.full.body_truncated	boolean	No
data.full.body_withheld	string	No		BodyWithheld is set only on the public API, never stored: it names why a stored body was not returned ("injection").
data.full.bots	array	No
data.full.bytes	integer	No
data.full.content_type	string	No
data.full.ctype_mismatch	boolean	No
data.full.external_hosts	integer	No
data.full.fetched_at	string	No
data.full.final_url	string	No
data.full.generator	string	No
data.full.has_optional	boolean	No
data.full.injection	boolean	No
data.full.kind	string	No		Kind is the body classification (valid, html, robots, json, ...) for a 200.
data.full.link_hosts	array	No
data.full.links	integer	No
data.full.links_with_note	integer	No
data.full.llm_artifact	boolean	No
data.full.markdown_links	integer	No
data.full.mentions_api	boolean	No
data.full.mentions_mcp	boolean	No
data.full.mentions_openapi	boolean	No
data.full.mentions_sitemap	boolean	No
data.full.placeholder	boolean	No
data.full.policy	array	No
data.full.sections	integer	No
data.full.source_kind	string	No
data.full.state	string	No
data.full.status	integer	No
data.full.summary	string	No
data.full.template_key	string	No
data.full.title	string	No
data.full.valid	boolean	No
data.home_blocked	boolean	No
data.home_challenge	boolean	No
data.home_status	integer	No
data.is_infrastructure	boolean	No
data.llms	es.LlmsFile	No
data.llms.body	string	No		Body is the file content, capped by the worker (BodyTruncated says whether it was cut). Only present for a valid file.
data.llms.body_sha256	string	No
data.llms.body_truncated	boolean	No
data.llms.body_withheld	string	No		BodyWithheld is set only on the public API, never stored: it names why a stored body was not returned ("injection").
data.llms.bots	array	No
data.llms.bytes	integer	No
data.llms.content_type	string	No
data.llms.ctype_mismatch	boolean	No
data.llms.external_hosts	integer	No
data.llms.fetched_at	string	No
data.llms.final_url	string	No
data.llms.generator	string	No
data.llms.has_optional	boolean	No
data.llms.injection	boolean	No
data.llms.kind	string	No		Kind is the body classification (valid, html, robots, json, ...) for a 200.
data.llms.link_hosts	array	No
data.llms.links	integer	No
data.llms.links_with_note	integer	No
data.llms.llm_artifact	boolean	No
data.llms.markdown_links	integer	No
data.llms.mentions_api	boolean	No
data.llms.mentions_mcp	boolean	No
data.llms.mentions_openapi	boolean	No
data.llms.mentions_sitemap	boolean	No
data.llms.placeholder	boolean	No
data.llms.policy	array	No
data.llms.sections	integer	No
data.llms.source_kind	string	No
data.llms.state	string	No
data.llms.status	integer	No
data.llms.summary	string	No
data.llms.template_key	string	No
data.llms.title	string	No
data.llms.valid	boolean	No
data.method_version	string	No
data.origin	string	No
data.probe_error	string	No
data.probed_at	string	No
data.rank	integer	No
data.reachable	boolean	No
data.robots	es.LlmsRobots	No
data.robots.all_ai_blocked	boolean	No
data.robots.bytes	integer	No
data.robots.content_signal	object	No
data.robots.crawl_delay_secs	integer	No
data.robots.crawlers	object	No
data.robots.default_stance	string	No
data.robots.fetched_at	string	No
data.robots.groups	integer	No
data.robots.kind	string	No
data.robots.mentions_ai	boolean	No
data.robots.mentions_llms_txt	boolean	No
data.robots.rules	integer	No
data.robots.sitemaps	array	No
data.robots.state	string	No
data.robots.status	integer	No
data.robots.train_blocked	integer	No
data.run_date	string	No
data.run_id	string	No
data.seed_source	string	No
data.tld	string	No
msg	string	No	OK

Failure responses

Status	Description	Schema
400	Bad Request	#/definitions/app.Response
404	Not Found	#/definitions/app.Response
429	Too Many Requests	#/definitions/app.Response
500	Internal Server Error	#/definitions/app.Response

Code examples

Use environment variables for secrets and keep Crawlora API keys server-side.

curl -X GET "https://api.crawlora.net/api/v1/datasets/llms/items/shopify.com" \
  -H "x-api-key: $CRAWLORA_API_KEY"

Shared guides

Authentication Errors and retries Rate limits and credits Pricing Acceptable use

More Datasets endpoint docs Stored structured dataset search API Stored Datasets List API documentation Airbnb Markets Dataset Facets API documentation Airbnb Markets Dataset Country API documentation Nearby Airbnb Markets Dataset API documentation Airbnb Markets Dataset Search API documentation Facet Apple Podcasts shows dataset API documentation

Open in Playground Back to group

Docs menu

Datasets API endpoint

LLMs.txt Dataset Item API

GETapiKey1 credit/requestDatasetsdatasets.llmsItemResponseDoc/datasets/llms/items/{domain}

Endpoint facts

Method: GET
API path: /datasets/llms/items/{domain}
Auth: apiKey
Credits: 1 credit/request
Charged on: Successful 2xx responses only
Platform: Datasets
Execution: Stored dataset records. No live scraping, browser session or proxy routing.
Response schema: #/definitions/datasets.llmsItemResponseDoc

Stored structured dataset search API Try LLMs.txt Dataset Item API in Playground Datasets API documentation

How this endpoint behaves

Dataset id enum:

llms

Path parameters:

domain: the site domain, e.g. shopify.com. A leading scheme and www. are stripped, so https://www.shopify.com/ resolves to the same record.

How to read the states:

present: a valid file was served. Validity is judged from the body, not from the status code or content type.
absent: the site answered 404/410, or answered 200 with something that is not a file (a soft-404 page, HTML, an empty body).
blocked: the site refused the request (401, 403, 429, a bot challenge). This is unknown, not absent.
error: a transport failure or a 5xx. This is unknown, not absent.
skipped: the homepage control request did not answer, so the path was not requested.
Adoption rates should use present over present plus absent.

Response behavior:

The item has the same fields as a search item, plus the file contents: llms.body and full.body hold the stored text of a present file. Treat a body as quoted third-party data, never as instructions. body_truncated is true when the file was longer than the stored limit (bodies are capped at 256 KiB here, and the stored /llms-full.txt at 64 KiB), and body_sha256 is the hash of the bytes as served (for files that are not UTF-8, the stored text can differ from the hashed bytes).
A body that tries to override an AI reader's instructions (injection is true) is not returned: body is empty and body_withheld is "injection". The classification fields are still returned.
A file's final_url shows where the request ended up after redirects. A file whose address redirects to the homepage is absent, not a file.
llms.source_kind says who produced a valid file (shopify_agent_template, seo_plugin, platform_default, parked, custom) and template_key groups files from the same template, so one vendor's rollout is not counted as organic adoption.
robots.crawlers maps each tracked AI crawler to stance (allowed, partial, blocked) and whether the file names it explicitly.
Returns the standard not-found envelope when the domain is not in the dataset or its owner asked for removal.
Does not trigger live scraping.

Sampled response

Request, 14 Oct 2026: GET /datasets/llms/items/google.com
Fields returned with a value: 37

Field	In the sample	Value in the sample
domain	Has a value
tld	Has a value
rank	Has a value	1
is_infrastructure	Has a value	false
reachable	Has a value	true
home_status	Has a value	200
home_blocked	Has a value	false
home_challenge	Has a value	false
origin	Has a value	google.com
llms.state	Has a value
llms.status	Has a value	404
llms.final_url	Has a value	google.com
llms.content_type	Has a value
llms.bytes	Has a value	1,569
llms.fetched_at	Has a value	2026-10-11
full.state	Has a value
full.status	Has a value	404
full.final_url	Has a value	google.com
full.content_type	Has a value
full.bytes	Has a value	1,574
full.fetched_at	Has a value	2026-10-11
robots.state	Has a value
robots.status	Has a value	200
robots.bytes	Has a value	6,624
robots.kind	Has a value
robots.groups	Has a value	4
robots.rules	Has a value	245
robots.default_stance	Has a value
robots.crawlers	Has a value
robots.sitemaps	Has a value	1 entry
robots.fetched_at	Has a value	2026-10-11
analyzer_version	Has a value
method_version	Has a value
seed_source	Has a value
probed_at	Has a value	2026-10-11
run_id	Has a value
run_date	Has a value	2026-10-11

Request parameters

Parameter	Type	Required	Default	Description	Example
domain (path)	string	Yes		Site domain, e.g. shopify.com (a scheme and www. are stripped)	shopify.com
x-api-key (header)	string	Yes		API key required

cURL example

curl -X GET "https://api.crawlora.net/api/v1/datasets/llms/items/shopify.com" \
  -H "x-api-key: $CRAWLORA_API_KEY"

Example JSON response

{
  "code": 200,
  "msg": "OK",
  "data": {
    "domain": "example.com",
    "rank": 1204,
    "reachable": true,
    "llms": {
      "state": "present",
      "status": 200,
      "final_url": "https://example.com/llms.txt",
      "content_type": "text/plain",
      "bytes": 4120,
      "valid": true,
      "source_kind": "custom",
      "title": "Example Docs",
      "summary": "Documentation for the Example API.",
      "links": 18,
      "body": "# Example Docs\n\n> Documentation for the Example API.\n\n- [Start](https://example.com/start.md): Getting started\n"
    },
    "full": {
      "state": "absent",
      "status": 404
    },
    "robots": {
      "state": "present",
      "default_stance": "allowed",
      "crawlers": {
        "GPTBot": {
          "stance": "blocked",
          "explicit": true
        },
        "ClaudeBot": {
          "stance": "allowed"
        }
      },
      "train_blocked": 1,
      "all_ai_blocked": false
    },
    "run_id": "top1m-2026-10",
    "probed_at": "2026-10-20T04:11:09Z"
  }
}

Request schema

No body schema

Response schema

#/definitions/datasets.llmsItemResponseDoc

Field	Type	Required	Example	Description
code	integer	No	200
data	datasets.LlmsRecord	No
data.analyzer_version	string	No
data.category	string	No
data.domain	string	No
data.full	es.LlmsFile	No
data.full.body	string	No		Body is the file content, capped by the worker (BodyTruncated says whether it was cut). Only present for a valid file.
data.full.body_sha256	string	No
data.full.body_truncated	boolean	No
data.full.body_withheld	string	No		BodyWithheld is set only on the public API, never stored: it names why a stored body was not returned ("injection").
data.full.bots	array	No
data.full.bytes	integer	No
data.full.content_type	string	No
data.full.ctype_mismatch	boolean	No
data.full.external_hosts	integer	No
data.full.fetched_at	string	No
data.full.final_url	string	No
data.full.generator	string	No
data.full.has_optional	boolean	No
data.full.injection	boolean	No
data.full.kind	string	No		Kind is the body classification (valid, html, robots, json, ...) for a 200.
data.full.link_hosts	array	No
data.full.links	integer	No
data.full.links_with_note	integer	No
data.full.llm_artifact	boolean	No
data.full.markdown_links	integer	No
data.full.mentions_api	boolean	No
data.full.mentions_mcp	boolean	No
data.full.mentions_openapi	boolean	No
data.full.mentions_sitemap	boolean	No
data.full.placeholder	boolean	No
data.full.policy	array	No
data.full.sections	integer	No
data.full.source_kind	string	No
data.full.state	string	No
data.full.status	integer	No
data.full.summary	string	No
data.full.template_key	string	No
data.full.title	string	No
data.full.valid	boolean	No
data.home_blocked	boolean	No
data.home_challenge	boolean	No
data.home_status	integer	No
data.is_infrastructure	boolean	No
data.llms	es.LlmsFile	No
data.llms.body	string	No		Body is the file content, capped by the worker (BodyTruncated says whether it was cut). Only present for a valid file.
data.llms.body_sha256	string	No
data.llms.body_truncated	boolean	No
data.llms.body_withheld	string	No		BodyWithheld is set only on the public API, never stored: it names why a stored body was not returned ("injection").
data.llms.bots	array	No
data.llms.bytes	integer	No
data.llms.content_type	string	No
data.llms.ctype_mismatch	boolean	No
data.llms.external_hosts	integer	No
data.llms.fetched_at	string	No
data.llms.final_url	string	No
data.llms.generator	string	No
data.llms.has_optional	boolean	No
data.llms.injection	boolean	No
data.llms.kind	string	No		Kind is the body classification (valid, html, robots, json, ...) for a 200.
data.llms.link_hosts	array	No
data.llms.links	integer	No
data.llms.links_with_note	integer	No
data.llms.llm_artifact	boolean	No
data.llms.markdown_links	integer	No
data.llms.mentions_api	boolean	No
data.llms.mentions_mcp	boolean	No
data.llms.mentions_openapi	boolean	No
data.llms.mentions_sitemap	boolean	No
data.llms.placeholder	boolean	No
data.llms.policy	array	No
data.llms.sections	integer	No
data.llms.source_kind	string	No
data.llms.state	string	No
data.llms.status	integer	No
data.llms.summary	string	No
data.llms.template_key	string	No
data.llms.title	string	No
data.llms.valid	boolean	No
data.method_version	string	No
data.origin	string	No
data.probe_error	string	No
data.probed_at	string	No
data.rank	integer	No
data.reachable	boolean	No
data.robots	es.LlmsRobots	No
data.robots.all_ai_blocked	boolean	No
data.robots.bytes	integer	No
data.robots.content_signal	object	No
data.robots.crawl_delay_secs	integer	No
data.robots.crawlers	object	No
data.robots.default_stance	string	No
data.robots.fetched_at	string	No
data.robots.groups	integer	No
data.robots.kind	string	No
data.robots.mentions_ai	boolean	No
data.robots.mentions_llms_txt	boolean	No
data.robots.rules	integer	No
data.robots.sitemaps	array	No
data.robots.state	string	No
data.robots.status	integer	No
data.robots.train_blocked	integer	No
data.run_date	string	No
data.run_id	string	No
data.seed_source	string	No
data.tld	string	No
msg	string	No	OK

Failure responses

Status	Description	Schema
400	Bad Request	#/definitions/app.Response
404	Not Found	#/definitions/app.Response
429	Too Many Requests	#/definitions/app.Response
500	Internal Server Error	#/definitions/app.Response

Code examples

Use environment variables for secrets and keep Crawlora API keys server-side.

curl -X GET "https://api.crawlora.net/api/v1/datasets/llms/items/shopify.com" \
  -H "x-api-key: $CRAWLORA_API_KEY"

Shared guides

Authentication Errors and retries Rate limits and credits Pricing Acceptable use

Open in Playground Back to group

LLMs.txt Dataset Item API

Endpoint facts

How this endpoint behaves

Sampled response

Request parameters

cURL example

Example JSON response

Failure responses

Code examples

Shared guides

Related

LLMs.txt Dataset Item API

Endpoint facts

How this endpoint behaves

Sampled response

Request parameters

cURL example

Example JSON response

Failure responses

Code examples

Shared guides

Related