Retrieval API endpoint
Returns status for a crawl job owned by the authenticated principal. This control call does not reserve or settle credits.
/web/crawl/{job_id}job_id is the opaque ID returned by web-crawl. The response is scoped to the authenticated owner and includes status, progress, warnings, deadline, and cumulative reservation/charge/release usage when the accounting reader is configured. Reading status does not reserve or settle credits. While a job is queued, deadline_at is the latest possible finish time; when work starts it moves earlier to the start time plus the job's timeout_seconds. A job that waits more than 30 minutes without starting fails with a timeout error ("job was not started within the 30-minute queue limit"), completion_reason deadline, and no charge. A job ID owned by another principal or belonging to another job kind returns 404.
kind is crawl. state is one of queued, running, cancelling, succeeded, failed, or cancelled. completion_reason, when present, is one of frontier_exhausted, completed, page_limit, round_limit, deadline, budget, cancelled, upstream_failure, or internal_failure.
Progress fields are discovered, discovery_excluded, admitted, queued, in_flight, succeeded, failed, and skipped. A completed job can be partial when it reached a configured limit or a source failed. Results are retained for 24 hours by default. While settlement is pending, expires_at is omitted and result retention remains protected.
Failure reason values are invalid_url, unsupported_content_type, blocked, not_found, rate_limited, timeout, upstream_error, internal_error, out_of_scope, robots_disallowed, duplicate, limit_reached, unsupported_filter, unknown_source, idempotency_conflict, invalid_cursor, result_expired, full_text_unavailable, and budget_exhausted.
Warning code values are source_failed, source_not_indexed, discovery_truncated, ranking_window_limit, imprecise_date_excluded, content_unavailable, and cached_source_stale.
| Parameter | Type | Required | Default | Description | Example |
|---|---|---|---|---|---|
| job_id (path) | string | Yes | Owned crawl job ID | job-123 |
curl -X GET "https://api.crawlora.net/api/v1/web/crawl/job-123" \ -H "x-api-key: $CRAWLORA_API_KEY"
{
"code": 200,
"msg": "OK",
"data": {
"job_id": "opaque-job-id",
"kind": "crawl",
"state": "running",
"cancel_requested": false,
"partial": false,
"created_at": "2026-10-09T04:00:00Z",
"updated_at": "2026-10-09T04:00:15Z",
"deadline_at": "2026-10-09T04:05:00Z",
"progress": {
"discovered": 12,
"discovery_excluded": 2,
"admitted": 10,
"queued": 4,
"in_flight": 1,
"succeeded": 5,
"failed": 0,
"skipped": 0
},
"usage": {
"search_requests": 0,
"fetch_attempts": 6,
"fetched_urls": 5,
"browser_renders": 0,
"input_tokens": 0,
"output_tokens": 0,
"credits_reserved": 100,
"credits_charged": 25,
"credits_released": 0,
"settlement_pending": true
},
"warnings": []
}
}Request schema
No body schema
Response schema
#/definitions/crawlretrieval.crawlStatusEnvelope
| Field | Type | Required | Enum | Bounds | Example | Description |
|---|---|---|---|---|---|---|
| code | integer | No | 200 | |||
| data | crawlretrieval.crawlJobStatusDoc | No | ||||
| data.cancel_requested | boolean | No | ||||
| data.completion_reason | string | No | frontier_exhausted, completed, page_limit, round_limit, deadline, budget, cancelled, upstream_failure, internal_failure | |||
| data.created_at | string · date-time | No | ||||
| data.deadline_at | string · date-time | No | ||||
| data.error | crawlretrieval.crawlFailureDoc | No | crawlFailureDoc record, 4 fields: message, reason, retry_after_seconds, retryable. Described under Cancel a web crawl API. | |||
| data.expires_at | string · date-time | No | ||||
| data.job_id | string | No | ||||
| data.kind | string | No | crawl | |||
| data.partial | boolean | No | ||||
| data.progress | retrievaltypes.JobProgress | No | JobProgress record, 8 fields: admitted, discovered, discovery_excluded, failed, in_flight, queued, skipped, succeeded. Described under Cancel a web crawl API. | |||
| data.state | string | No | queued, running, cancelling, succeeded, failed, cancelled | |||
| data.updated_at | string · date-time | No | ||||
| data.usage | retrievaltypes.Usage | No | Usage record, 10 fields: browser_renders, credits_charged, credits_released, credits_reserved, fetch_attempts, fetched_urls, input_tokens, output_tokens, search_requests, settlement_pending. Described under Map a public website API. | |||
| data.warnings | array | No | ||||
| data.warnings[].code | string | No | source_failed, source_not_indexed, discovery_truncated, ranking_window_limit, imprecise_date_excluded, content_unavailable, cached_source_stale | |||
| data.warnings[].message | string | No | ||||
| data.warnings[].source_id | string | No | ||||
| msg | string | No | OK |
| Status | Description | Schema |
|---|---|---|
| 400 | Invalid job ID | #/definitions/app.Response |
| 401 | Authentication required | #/definitions/app.Response |
| 404 | Unknown, unowned, or non-crawl job | #/definitions/app.Response |
| 500 | Internal accounting or storage error | #/definitions/app.Response |
| 503 | Required backend unavailable | #/definitions/app.Response |
Use environment variables for secrets and keep Crawlora API keys server-side.
curl -X GET "https://api.crawlora.net/api/v1/web/crawl/job-123" \
-H "x-api-key: $CRAWLORA_API_KEY"