Docs menu

Retrieval API endpoint

Map a public website API

Maps a public site into a bounded, approximate URL list from its sitemaps and the seed page's links. The first request needs `url`; options are `scope`, `sitemap` (`include`, `only`, `skip`), `max_urls` (1 to 5000, default 1000) and `page_size` (1 to 500, default 100). Continue with `cursor`, optionally with `page_size`; the first page's snapshot lasts 30 minutes.

POSTapiKey21 credits/requestRetrievalretrievalplatform.MapRequest/web/map

Endpoint facts

Method: POST
API path: /web/map
Auth: apiKey
Credits: 21 credits/request
Charged on: Successful 2xx responses only
Platform: Retrieval
Execution: Managed browser rendering where the target needs it
Request body schema: #/definitions/retrievalplatform.MapRequest

Try Map a public website API in Playground Retrieval API documentation

How this endpoint behaves

Builds an approximate, bounded URL inventory from public robots.txt, sitemap files, and links on the seed page. It follows only the seed host by default and applies public-target checks to every request and redirect. Requires the x-api-key header. A first-page result is frozen in an owner-bound snapshot for 30 minutes; continuation requests read that snapshot without new source fetches.

Request body:

First request requires url, an absolute HTTP or HTTPS URL. URL user credentials and non-public targets are rejected.
scope: optional. include_subdomains defaults to false; include_paths and exclude_paths each allow at most 20 unique path globs (256 characters each). Globs are anchored to the pathname; * matches within one path segment and ** across segments. Excludes take precedence. query_policy values are preserve and drop_tracking, default preserve.
sitemap: optional; values include, only, skip; default include.
max_urls: optional, 1–5000; default 1000 and bounded by effective capabilities.
page_size: optional, 1–500; default 100.
cursor: optional opaque continuation cursor. When present, only cursor and page_size may be supplied; the original URL, scope, sitemap mode, and maximum are frozen in the cursor-backed snapshot.

Response enum values:

coverage: approximate.
items[].discovered_via: sitemap, link, catalog (P1 currently emits sitemap or link).
warnings[].code: source_failed, source_not_indexed, discovery_truncated, ranking_window_limit, imprecise_date_excluded, content_unavailable, cached_source_stale.

Example first-page request:

{
  "url": "https://example.org/docs/",
  "scope": {"include_paths": ["/docs/**"], "query_policy": "preserve"},
  "sitemap": "include",
  "max_urls": 1000,
  "page_size": 100
}

Request parameters

Parameter	Type	Required	Default	Description	Example
request (body)	object	Yes		Map request

cURL example

curl -X POST "https://api.crawlora.net/api/v1/web/map" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"cursor":"string","max_urls":1,"page_size":1,"scope":{"exclude_paths":["string"],"include_paths":["string"],"include_subdomains":true,"query_policy":"preserve"},"sitemap":"string","url":"string"}'

Request body example

{
  "cursor": "string",
  "max_urls": 1,
  "page_size": 1,
  "scope": {
    "exclude_paths": [
      "string"
    ],
    "include_paths": [
      "string"
    ],
    "include_subdomains": true,
    "query_policy": "preserve"
  },
  "sitemap": "string",
  "url": "string"
}

Example JSON response

{
  "code": 200,
  "msg": "OK",
  "data": {
    "items": [
      {
        "url": "https://example.org/docs/start",
        "discovered_via": "sitemap",
        "observed_at": "2026-10-09T04:00:00Z"
      }
    ],
    "coverage": "approximate",
    "partial": false,
    "truncated": false,
    "has_more": false,
    "expires_at": "2026-10-09T04:30:00Z",
    "warnings": [],
    "usage": {
      "search_requests": 0,
      "fetch_attempts": 1,
      "fetched_urls": 1,
      "browser_renders": 0,
      "input_tokens": 0,
      "output_tokens": 0,
      "credits_reserved": 0,
      "credits_charged": 2,
      "credits_released": 0,
      "settlement_pending": false
    }
  }
}

Request schema

#/definitions/retrievalplatform.MapRequest

Field	Type	Required	Enum
cursor	string	No
max_urls	integer	No
page_size	integer	No
scope	retrievaltypes.URLScope	No
scope.exclude_paths	array	No
scope.include_paths	array	No
scope.include_subdomains	boolean	No
scope.query_policy	retrievaltypes.QueryPolicy	No	preserve, drop_tracking
sitemap	string	No
url	string	No

Response schema

Inline response schema

Field	Type	Required	Enum	Description
code	integer	No		Code is the HTTP status code or a custom code used to indicate the result of the request @example 200
data	retrievalplatform.MapResponse	No
data.coverage	string	No
data.expires_at	string	No
data.has_more	boolean	No
data.items	array	No
data.items[].discovered_via	string	No
data.items[].last_modified	string	No
data.items[].observed_at	string	No
data.items[].parent_url	string	No
data.items[].title	string	No
data.items[].url	string	No
data.next_cursor	string	No
data.partial	boolean	No
data.truncated	boolean	No
data.usage	retrievaltypes.Usage	No
data.usage.browser_renders	integer	No
data.usage.credits_charged	integer	No
data.usage.credits_released	integer	No
data.usage.credits_reserved	integer	No
data.usage.fetch_attempts	integer	No
data.usage.fetched_urls	integer	No
data.usage.input_tokens	integer	No
data.usage.output_tokens	integer	No
data.usage.search_requests	integer	No
data.usage.settlement_pending	boolean	No
data.warnings	array	No
data.warnings[].code	retrievaltypes.WarningCode	No	source_failed, source_not_indexed, discovery_truncated, ranking_window_limit, imprecise_date_excluded, content_unavailable, cached_source_stale
data.warnings[].message	string	No
data.warnings[].source_id	string	No
msg	unknown	No		Msg is the message that describes the result of the request @example "Request successful"

Failure responses

Status	Description	Schema
400	Bad Request	#/definitions/app.Response
401	Unauthorized	#/definitions/app.Response
402	Payment Required	#/definitions/app.Response
410	Gone	#/definitions/app.Response
429	Too Many Requests	#/definitions/app.Response
500	Internal Server Error	#/definitions/app.Response
503	Service Unavailable	#/definitions/app.Response

Code examples

Use environment variables for secrets and keep Crawlora API keys server-side.

curl -X POST "https://api.crawlora.net/api/v1/web/map" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"cursor":"string","max_urls":1,"page_size":1,"scope":{"exclude_paths":["string"],"include_paths":["string"],"include_subdomains":true,"query_policy":"preserve"},"sitemap":"string","url":"string"}'

Shared guides

Authentication Errors and retries Rate limits and credits Pricing Acceptable use

More Retrieval endpoint docs Search developer documentation API documentation Discover developer sources API documentation Get web retrieval capabilities API documentation Submit a bounded web crawl API documentation Read web crawl status API documentation Cancel a web crawl API documentation

Open in Playground Back to group

Docs menu

Retrieval API endpoint

Map a public website API

POSTapiKey21 credits/requestRetrievalretrievalplatform.MapRequest/web/map

Endpoint facts

Method: POST
API path: /web/map
Auth: apiKey
Credits: 21 credits/request
Charged on: Successful 2xx responses only
Platform: Retrieval
Execution: Managed browser rendering where the target needs it
Request body schema: #/definitions/retrievalplatform.MapRequest

Try Map a public website API in Playground Retrieval API documentation

How this endpoint behaves

Request body:

First request requires url, an absolute HTTP or HTTPS URL. URL user credentials and non-public targets are rejected.
scope: optional. include_subdomains defaults to false; include_paths and exclude_paths each allow at most 20 unique path globs (256 characters each). Globs are anchored to the pathname; * matches within one path segment and ** across segments. Excludes take precedence. query_policy values are preserve and drop_tracking, default preserve.
sitemap: optional; values include, only, skip; default include.
max_urls: optional, 1–5000; default 1000 and bounded by effective capabilities.
page_size: optional, 1–500; default 100.
cursor: optional opaque continuation cursor. When present, only cursor and page_size may be supplied; the original URL, scope, sitemap mode, and maximum are frozen in the cursor-backed snapshot.

Response enum values:

coverage: approximate.
items[].discovered_via: sitemap, link, catalog (P1 currently emits sitemap or link).
warnings[].code: source_failed, source_not_indexed, discovery_truncated, ranking_window_limit, imprecise_date_excluded, content_unavailable, cached_source_stale.

Example first-page request:

{
  "url": "https://example.org/docs/",
  "scope": {"include_paths": ["/docs/**"], "query_policy": "preserve"},
  "sitemap": "include",
  "max_urls": 1000,
  "page_size": 100
}

Request parameters

Parameter	Type	Required	Default	Description	Example
request (body)	object	Yes		Map request

cURL example

curl -X POST "https://api.crawlora.net/api/v1/web/map" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"cursor":"string","max_urls":1,"page_size":1,"scope":{"exclude_paths":["string"],"include_paths":["string"],"include_subdomains":true,"query_policy":"preserve"},"sitemap":"string","url":"string"}'

Request body example

{
  "cursor": "string",
  "max_urls": 1,
  "page_size": 1,
  "scope": {
    "exclude_paths": [
      "string"
    ],
    "include_paths": [
      "string"
    ],
    "include_subdomains": true,
    "query_policy": "preserve"
  },
  "sitemap": "string",
  "url": "string"
}

Example JSON response

{
  "code": 200,
  "msg": "OK",
  "data": {
    "items": [
      {
        "url": "https://example.org/docs/start",
        "discovered_via": "sitemap",
        "observed_at": "2026-10-09T04:00:00Z"
      }
    ],
    "coverage": "approximate",
    "partial": false,
    "truncated": false,
    "has_more": false,
    "expires_at": "2026-10-09T04:30:00Z",
    "warnings": [],
    "usage": {
      "search_requests": 0,
      "fetch_attempts": 1,
      "fetched_urls": 1,
      "browser_renders": 0,
      "input_tokens": 0,
      "output_tokens": 0,
      "credits_reserved": 0,
      "credits_charged": 2,
      "credits_released": 0,
      "settlement_pending": false
    }
  }
}

Request schema

#/definitions/retrievalplatform.MapRequest

Field	Type	Required	Enum
cursor	string	No
max_urls	integer	No
page_size	integer	No
scope	retrievaltypes.URLScope	No
scope.exclude_paths	array	No
scope.include_paths	array	No
scope.include_subdomains	boolean	No
scope.query_policy	retrievaltypes.QueryPolicy	No	preserve, drop_tracking
sitemap	string	No
url	string	No

Response schema

Inline response schema

Field	Type	Required	Enum	Description
code	integer	No		Code is the HTTP status code or a custom code used to indicate the result of the request @example 200
data	retrievalplatform.MapResponse	No
data.coverage	string	No
data.expires_at	string	No
data.has_more	boolean	No
data.items	array	No
data.items[].discovered_via	string	No
data.items[].last_modified	string	No
data.items[].observed_at	string	No
data.items[].parent_url	string	No
data.items[].title	string	No
data.items[].url	string	No
data.next_cursor	string	No
data.partial	boolean	No
data.truncated	boolean	No
data.usage	retrievaltypes.Usage	No
data.usage.browser_renders	integer	No
data.usage.credits_charged	integer	No
data.usage.credits_released	integer	No
data.usage.credits_reserved	integer	No
data.usage.fetch_attempts	integer	No
data.usage.fetched_urls	integer	No
data.usage.input_tokens	integer	No
data.usage.output_tokens	integer	No
data.usage.search_requests	integer	No
data.usage.settlement_pending	boolean	No
data.warnings	array	No
data.warnings[].code	retrievaltypes.WarningCode	No	source_failed, source_not_indexed, discovery_truncated, ranking_window_limit, imprecise_date_excluded, content_unavailable, cached_source_stale
data.warnings[].message	string	No
data.warnings[].source_id	string	No
msg	unknown	No		Msg is the message that describes the result of the request @example "Request successful"

Failure responses

Status	Description	Schema
400	Bad Request	#/definitions/app.Response
401	Unauthorized	#/definitions/app.Response
402	Payment Required	#/definitions/app.Response
410	Gone	#/definitions/app.Response
429	Too Many Requests	#/definitions/app.Response
500	Internal Server Error	#/definitions/app.Response
503	Service Unavailable	#/definitions/app.Response

Code examples

Use environment variables for secrets and keep Crawlora API keys server-side.

curl -X POST "https://api.crawlora.net/api/v1/web/map" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"cursor":"string","max_urls":1,"page_size":1,"scope":{"exclude_paths":["string"],"include_paths":["string"],"include_subdomains":true,"query_policy":"preserve"},"sitemap":"string","url":"string"}'

Shared guides

Authentication Errors and retries Rate limits and credits Pricing Acceptable use

Open in Playground Back to group

Map a public website API

Endpoint facts

How this endpoint behaves

Request parameters

cURL example

Request body example

Example JSON response

Failure responses

Code examples

Shared guides

Related

Map a public website API

Endpoint facts

How this endpoint behaves

Request parameters

cURL example

Request body example

Example JSON response

Failure responses

Code examples

Shared guides

Related