Developer guides

Crawlora Python SDK

Connect Python scripts, notebooks, ETL jobs, and AI workflows to Crawlora's structured public web data APIs with the official Git-tagged beta SDK.

Official beta SDKGit installType stubsPipelinesNotebooks

Open GitHub repo Try Playground

Verified HTTP pattern

POST /google/search

Normalized JSON

Request

POST https://api.crawlora.net/api/v1/google/search
x-api-key: $CRAWLORA_API_KEY
Content-Type: application/json

{
  "country": "us",
  "keyword": "best CRM software",
  "language": "en",
  "limit": 10,
  "page": 1
}

Base URL

https://api.crawlora.net/api/v1

Auth header

x-api-key

Example endpoint

POST /google/search

The Python SDK is published from https://github.com/Crawlora-org/crawlora-python-sdk. Install the current promoted beta tag v1.5.0-sdk.3; PyPI publication is not enabled yet, so pin the Git tag explicitly.

Developer workflow

Install the SDK

Install the beta SDK from the tagged GitHub repository. The package imports as `crawlora`.

Install crawlora · bash

pip install "git+https://github.com/Crawlora-org/crawlora-python-sdk.git@v1.5.0-sdk.3"

Developer workflow

Environment variable

.env · bash

export CRAWLORA_API_KEY="your_api_key_here"

Developer workflow

SDK request

Use generated endpoint groups, keyword parameters, enum values, request options, and response aliases.

SDK client · python

import os
from crawlora import CrawloraClient

crawlora = CrawloraClient(api_key=os.environ["CRAWLORA_API_KEY"])
result = crawlora.bing.search(q="coffee shops", count=10)
print(result)

Developer workflow

Raw requests fallback

Use direct HTTP calls when you need a minimal dependency path or want to compare behavior with endpoint docs.

Python request · python

import os
import requests

API_KEY = os.environ["CRAWLORA_API_KEY"]
BASE_URL = "https://api.crawlora.net/api/v1"

def crawlora_request(path: str, payload: dict) -> dict:
    response = requests.post(
        f"{BASE_URL}{path}",
        headers={
            "x-api-key": API_KEY,
            "Content-Type": "application/json",
        },
        json=payload,
        timeout=60,
    )

    if not response.ok:
        raise RuntimeError(
            f"Crawlora request failed: {response.status_code} {response.text}"
        )

    return response.json()

if __name__ == "__main__":
    data = crawlora_request(
        "/google/search",
        {
            "country": "us",
            "keyword": "best CRM software",
            "language": "en",
            "limit": 10,
            "page": 1,
        },
    )

    print(data)

Developer workflow

Data pipeline example

Keep batch output simple by writing JSONL from structured SDK or HTTP results.

JSONL pipeline · python

import json

keywords = ["project management software", "crm for startups", "sales automation"]

with open("crawlora-search-results.jsonl", "w", encoding="utf-8") as output:
    for keyword in keywords:
        try:
            payload = crawlora_request(
                "/google/search",
                {"keyword": keyword, "country": "us", "language": "en", "limit": 10, "page": 1},
            )
            output.write(json.dumps({"keyword": keyword, "response": payload}) + "\n")
        except RuntimeError as exc:
            output.write(json.dumps({"keyword": keyword, "error": str(exc)}) + "\n")

Developer workflow

Notebook / AI workflow example

Normalize Crawlora output before passing it into a summarization, clustering, or retrieval step.

Structured agent input · python

def summarize_search_results(keyword: str) -> dict:
    data = crawlora_request(
        "/google/search",
        {"keyword": keyword, "country": "us", "language": "en", "limit": 10, "page": 1},
    )

    results = data.get("data", {}).get("result", [])
    return {
        "keyword": keyword,
        "items": [
            {
                "title": item.get("title"),
                "url": item.get("link"),
                "snippet": item.get("Snippet"),
            }
            for item in results
        ],
    }

# Pass this structured JSON into your LLM or agent layer.

Developer workflow

Error handling

Catch request exceptions, inspect 401, 429, and temporary 5xx responses, and log response context where available.

Status / code	Meaning	How to handle
400	Invalid request or missing required input.	Validate request bodies before calling Crawlora and surface useful messages to users.
401	Missing or invalid API key.	Check the `x-api-key` header and rotate the key from the console if needed.
402/403	Plan, permission, or billing issue where applicable.	Check plan access, credit state, and endpoint availability.
429	Rate limit exceeded.	Back off with jitter and reduce concurrency.
5xx	Temporary execution or upstream failure.	Retry safe jobs with exponential backoff and keep the failure visible.

Developer workflow

Production checklist

Store the key in an environment variable.
Set a timeout.
Avoid unbounded concurrency.
Handle 429 responses.
Store raw responses only when needed.
Sanitize user input.
Respect responsible-use rules.

Responsible public web data workflows

Use Crawlora for structured public web data workflows. Customers are responsible for compliance with applicable laws, third-party rights, platform rules, and Crawlora terms. Keep API keys server-side, validate inputs, and avoid collecting or storing unnecessary sensitive data.

Read Crawlora terms

Developer workflow

FAQ

Common questions for this Crawlora developer integration path.

Is there an official Crawlora Python SDK?

Yes. The official Python beta SDK is published at https://github.com/Crawlora-org/crawlora-python-sdk. Install it from an explicit Git tag such as v1.5.0-sdk.3.

Should I use requests or httpx?

Use requests for straightforward scripts. Use httpx if your project already needs async clients, connection pooling, or richer timeout controls.

Can I use Crawlora in notebooks?

Yes. Store your key outside the notebook when possible and keep result counts bounded.

Can I run Crawlora jobs in Airflow or cron?

Yes. Add retries with backoff, timeout controls, and logging around each endpoint call.

How do I handle rate limits?

Reduce concurrency and retry 429 responses with exponential backoff. Avoid tight retry loops.

Can I use Crawlora with LangChain?

Yes. Wrap a Crawlora HTTP call as a custom tool, loader, or retrieval ingestion step.

Where are response schemas documented?

Endpoint detail pages in the docs catalog show available examples, parameters, and schema references.

Install the Python SDK

Pin the current Git tag, set CRAWLORA_API_KEY, then inspect endpoint docs for platform-specific schemas.

Open GitHub repo Browse APIs

POST https://api.crawlora.net/api/v1/google/search x-api-key: $CRAWLORA_API_KEY Content-Type: application/json { "country": "us", "keyword": "best CRM software", "language": "en", "limit": 10, "page": 1 }

import os import requests API_KEY = os.environ["CRAWLORA_API_KEY"] BASE_URL = "https://api.crawlora.net/api/v1" def crawlora_request(path: str, payload: dict) -> dict: response = requests.post( f"{BASE_URL}{path}", headers={ "x-api-key": API_KEY, "Content-Type": "application/json", }, json=payload, timeout=60, ) if not response.ok: raise RuntimeError( f"Crawlora request failed: {response.status_code} {response.text}" ) return response.json() if __name__ == "__main__": data = crawlora_request( "/google/search", { "country": "us", "keyword": "best CRM software", "language": "en", "limit": 10, "page": 1, }, ) print(data)

import json keywords = ["project management software", "crm for startups", "sales automation"] with open("crawlora-search-results.jsonl", "w", encoding="utf-8") as output: for keyword in keywords: try: payload = crawlora_request( "/google/search", {"keyword": keyword, "country": "us", "language": "en", "limit": 10, "page": 1}, ) output.write(json.dumps({"keyword": keyword, "response": payload}) + "\n") except RuntimeError as exc: output.write(json.dumps({"keyword": keyword, "error": str(exc)}) + "\n")

def summarize_search_results(keyword: str) -> dict: data = crawlora_request( "/google/search", {"keyword": keyword, "country": "us", "language": "en", "limit": 10, "page": 1}, ) results = data.get("data", {}).get("result", []) return { "keyword": keyword, "items": [ { "title": item.get("title"), "url": item.get("link"), "snippet": item.get("Snippet"), } for item in results ], } # Pass this structured JSON into your LLM or agent layer.

Status / code

Meaning

How to handle

400

Invalid request or missing required input.

Validate request bodies before calling Crawlora and surface useful messages to users.

401

Missing or invalid API key.

Check the `x-api-key` header and rotate the key from the console if needed.

402/403

Plan, permission, or billing issue where applicable.

Check plan access, credit state, and endpoint availability.

429

Rate limit exceeded.

Back off with jitter and reduce concurrency.

5xx

Temporary execution or upstream failure.

Retry safe jobs with exponential backoff and keep the failure visible.

Crawlora Python SDK

Install the SDK

Install crawlora · bash

Environment variable

.env · bash

SDK request

SDK client · python

Raw requests fallback

Python request · python

Data pipeline example

JSONL pipeline · python

Notebook / AI workflow example

Structured agent input · python

Error handling

Production checklist

Responsible public web data workflows

Related developer links

Python SDK repo

Crawlora GitHub org

Docs

Playground

TypeScript guide

Go guide

LangChain guide

AI agents guide

FAQ

Install the Python SDK

Crawlora Python SDK

Install the SDK

Install crawlora · bash

Environment variable

.env · bash

SDK request

SDK client · python

Raw requests fallback

Python request · python

Data pipeline example

JSONL pipeline · python

Notebook / AI workflow example

Structured agent input · python

Error handling

Production checklist

Responsible public web data workflows

Related developer links

Python SDK repo

Crawlora GitHub org

Docs

Playground

TypeScript guide

Go guide

LangChain guide

AI agents guide

FAQ

Install the Python SDK