Submit a bounded web crawl API

Endpoint facts

Method

POST

API path

/web/crawl

Auth

apiKey

Credits

0 credits/request

Charged on

Successful 2xx responses only

Platform

Retrieval

Execution

Managed browser rendering where the target needs it

Request body schema

#/definitions/crawlretrieval.crawlRequestDoc

How this endpoint behaves

The endpoint returns 202 after it has persisted an owner-scoped job and its queue outbox record. Location is the configured public status URL and Retry-After suggests a three-second polling interval. An identical idempotency_key replays the original job and does not prepare another reservation.

Required body fields are idempotency_key (16–128 characters), url (absolute HTTP or HTTPS URL), and limits.max_credits (1–10,000). Optional sitemap is one of include, only, or skip and defaults to include. scope may opt into seed-host subdomains and restrict paths with anchored globs (* matches one path segment; ** matches multiple segments). Exclusions win. scope.query_policy is preserve or drop_tracking and defaults to preserve; tracking removal uses a pinned list and retains meaningful query parameters. fetch.render accepts auto or http and defaults to http; the effective browser-render cap is zero, so browser rendering and nonzero wait_for values are rejected.

limits.max_pages defaults to 100 and is bounded to 1–1,000 admitted URL tasks. Failed and skipped tasks consume a slot; retries reuse it. max_depth defaults to 2 and is bounded to 0–10; the seed has depth 0, and depth 0 performs no robots, sitemap, or link-discovery IO. timeout_seconds defaults to 300 and is bounded to 30–1,800; it counts from the moment work on the job starts, not from submission. max_total_bytes defaults to 20 MiB and is bounded to 1 KiB–100 MiB of decompressed metadata and document response bytes. max_browser_renders is fixed at 0. The trusted effective max_credits cap may be lower than 10,000.

The job uses a durable cursor and records one terminal outcome for each admitted URL. Sitemap and robots discovery are bounded and sitemap-only mode keeps the explicit seed while disabling page-link expansion. Source-profile changes stop further IO for an accepted job; already checkpointed results remain available.

A job waits at most 30 minutes in the queue before work starts. If it has not started by then it fails with a timeout error ("job was not started within the 30-minute queue limit"), nothing is charged, and the submission can be retried. deadline_at in the submission response is the latest possible finish time (30 minutes of queue wait plus timeout_seconds); once the job starts, deadline_at in the status response moves earlier to the start time plus timeout_seconds.

The response envelope contains job_id, state, replayed, status_url, results_url, and deadline_at. The state values are queued, running, cancelling, succeeded, failed, and cancelled.

Example request:

{
  "idempotency_key": "crawl-docs-example-0001",
  "url": "https://example.org/docs/",
  "scope": {
    "include_paths": ["/docs/**"],
    "query_policy": "preserve"
  },
  "sitemap": "include",
  "limits": {
    "max_pages": 100,
    "max_depth": 2,
    "max_credits": 100
  }
}

Shared guides

Related