Error Handling

Handle invalid requests, authentication errors, rate limits, upstream failures, timeouts, challenged responses, and temporary execution issues.

Quickstart Rate limits

Error handling overview

Production public web data workflows need clear handling for invalid requests, auth errors, rate limits, upstream failures, timeouts, challenged responses, and temporary execution issues.

Error response format

Exact error bodies can vary by endpoint. This shape is illustrative until a specific endpoint documents a more precise response schema.

{
  "error": {
    "code": "upstream_unavailable",
    "message": "The upstream response could not be used.",
    "request_id": "req_..."
  }
}

Common HTTP status codes

Check individual endpoint docs for exact response behavior. Generated endpoint docs also include failure responses where metadata is available.

Status	Meaning	Retry?	Recommended action
400	Invalid request	No	Fix input, required fields, or schema mismatch.
401	Missing or invalid API key	No	Check x-api-key and server-side secret configuration.
403	Permission, plan, or policy issue	No	Check plan access and account state.
404	Endpoint or resource not found	No	Confirm endpoint path and path parameters.
422	Validation error when supported	No	Fix field values and enum choices.
429	Rate limit exceeded	After delay	Back off and respect rate-limit reset behavior.
500	Internal execution error	Yes	Retry with capped exponential backoff.
502	Upstream failure or unusable upstream response	Sometimes	Retry later, inspect endpoint notes, and log request context.
503	Temporary unavailable	Yes	Retry with backoff and reduce concurrency.
504	Timeout	Yes	Retry with backoff and review request size or endpoint notes.

Challenge-aware failures

Some upstream responses may be blocked, challenged, empty, redirected, or otherwise unusable. Crawlora aims to return clear failure context for supported endpoints instead of silently returning bad data.

Retry guidance

Retry temporary 5xx responses and timeouts with backoff
Respect 429 rate-limit responses
Do not retry invalid 400 requests without fixing input
Cap retry attempts
Avoid aggressive loops
Log request IDs when present

Production pattern examples

Retry wrappers

Use environment variables for secrets and keep Crawlora API keys server-side.

async function crawloraWithRetry(url: string, init: RequestInit, attempts = 3) {
  for (let attempt = 1; attempt <= attempts; attempt += 1) {
    const response = await fetch(url, init);
    if (response.ok) return response;
    if (![429, 500, 502, 503, 504].includes(response.status) || attempt === attempts) {
      throw new Error(`Crawlora request failed: ${response.status} ${await response.text()}`);
    }
    const delayMs = Math.min(1000 * 2 ** (attempt - 1), 8000);
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }
  throw new Error("Retry loop exited unexpectedly");
}

Debug checklist

Confirm endpoint path
Confirm x-api-key header
Validate request body
Check credits and plan
Check rate limits
Inspect response body
Log request ID when present
Try the same endpoint in Playground

Responsible public web data workflows

Crawlora is designed for responsible structured public web data workflows. Customers are responsible for using Crawlora in compliance with applicable laws, third-party rights, target-platform rules, and Crawlora terms.

Read Crawlora terms

FAQ

What does a Crawlora API error look like?

Use the endpoint docs first. If no precise schema is documented, expect an HTTP status plus a JSON or text body with failure context when available.

Should I retry failed requests?

Retry temporary 5xx, timeout, and some 429 responses with capped backoff. Do not retry invalid requests until the input is fixed.

What does a 429 mean?

The request exceeded a plan, endpoint, or throughput limit. Reduce concurrency and wait before retrying.

What happens when an upstream page is blocked or challenged?

Crawlora detects challenged, blocked, or unusable upstream responses for supported endpoints and surfaces failure context when the response cannot be used.

Do failed requests consume credits?

Credit behavior should be checked on pricing and endpoint docs because billing policy can vary by successful response and endpoint metadata.

How do I debug failed requests?

Check auth, endpoint path, input schema, plan limits, response body, request IDs, and Playground behavior.

Why did I get empty or partial data?

The platform may have returned no matching data, paginated data, or an unusable upstream response. Review endpoint response notes and schema docs.

Build predictable failure handling

Pair endpoint-specific schema docs with rate-limit handling and capped retries before production.

Read rate limits Test in Playground