Endpoint Playground
Returns one site by its domain from dataset id enum value `llms`, including the stored `/llms.txt` and `/llms-full.txt` bodies (`llms.body`, `full.body`; `body_truncated` marks a cut file, and a body that tries to override an AI reader's instructions is withheld with `body_withheld` set to `injection`), their structure and classification, and the `/robots.txt` stance toward each tracked AI crawler. Returns 404 when the domain is not in the dataset or its owner asked for removal (privacy@crawlora.net). Dataset Playground requests query indexed data and do not apply proxy routing.
14 Oct 2026: Single record response; 37 fields had a value in every checked item.
Sample value profiles: llms.fetched_at: 2026-10-11; rank: 1; origin: google.com.
curl "https://api.crawlora.net/api/v1/datasets/llms/items/shopify.com" \
-H "x-api-key: $CRAWLORA_API_KEY"| Parameter | Type | Required | Prefilled value | Description |
|---|---|---|---|---|
| domain (path) | string | Yes | shopify.com | Site domain, e.g. shopify.com (a scheme and www. are stripped) |
{
"code": 200,
"msg": "OK",
"data": {
"domain": "example.com",
"rank": 1204,
"reachable": true,
"llms": {
"state": "present",
"status": 200,
"final_url": "https://example.com/llms.txt",
"content_type": "text/plain",
"bytes": 4120,
"valid": true,
"source_kind": "custom",
"title": "Example Docs",
"summary": "Documentation for the Example API.",
"links": 18,
"body": "# Example Docs\n\n> Documentation for the Example API.\n\n- [Start](https://example.com/start.md): Getting started\n"
},
"full": {
"state": "absent",
"status": 404
},
"robots": {
"state": "present",
"default_stance": "allowed",
"crawlers": {
"GPTBot": {
"stance": "blocked",
"explicit": true
},
"ClaudeBot": {
"stance": "allowed"
}
},
"train_blocked": 1,
"all_ai_blocked": false
},
"run_id": "top1m-2026-10",
"probed_at": "2026-10-20T04:11:09Z"
}
}The sample response, schema and code snippets below work without an account. Sending a live request needs an API key.