Developer guides
Use Crawlora APIs to provide structured public web data to LangChain workflows without relying on raw HTML as the primary input.
Verified HTTP pattern
POST /google/search
Request
POST https://api.crawlora.net/api/v1/google/search
x-api-key: $CRAWLORA_API_KEY
Content-Type: application/json
{
"country": "us",
"keyword": "best CRM software",
"language": "en",
"limit": 10,
"page": 1
}Base URL
https://api.crawlora.net/api/v1
Auth header
x-api-key
Example endpoint
POST /google/search
This repository does not contain an official LangChain package. Treat this guide as a custom tool and loader pattern.
Developer workflow
LangChain workflows often need external data. Crawlora can provide normalized JSON from supported public platforms, making it easier to summarize, classify, embed, or store results.
Developer workflow
Developer workflow
Adapt the wrapper to your installed LangChain version's current tool API.
import os
import requests
API_KEY = os.environ["CRAWLORA_API_KEY"]
BASE_URL = "https://api.crawlora.net/api/v1"
def crawlora_google_search(query: str) -> dict:
response = requests.post(
f"{BASE_URL}/google/search",
headers={"x-api-key": API_KEY, "Content-Type": "application/json"},
json={"keyword": query, "country": "us", "language": "en", "limit": 10, "page": 1},
timeout=60,
)
response.raise_for_status()
return response.json()
# Adapt this function to your installed LangChain version's tool wrapper.Developer workflow
Transform Crawlora JSON into simple document dictionaries before passing them into your retrieval or storage layer.
def crawlora_results_to_documents(payload: dict) -> list[dict]:
results = payload.get("data", {}).get("result", [])
return [
{
"page_content": item.get("Snippet") or item.get("title") or "",
"metadata": {
"title": item.get("title"),
"url": item.get("link"),
"position": item.get("position"),
"source": "crawlora_google_search",
},
}
for item in results
]Developer workflow
Developer workflow
Use Crawlora for structured public web data workflows. Customers are responsible for compliance with applicable laws, third-party rights, platform rules, and Crawlora terms. Keep API keys server-side, validate inputs, and avoid collecting or storing unnecessary sensitive data.
Read Crawlora termsDeveloper workflow
Use these pages to move between endpoint discovery, examples, pricing, and responsible-use guidance.
Developer workflow
Common questions for this Crawlora developer integration path.
This frontend repository does not contain an official Crawlora LangChain package. Use a custom tool or loader wrapper around the HTTP API.
Use a tool for agent-time decisions and a loader for scheduled ingestion or retrieval indexing.
Yes. Convert normalized result items into documents with clear metadata before embedding.
Yes, if the selected YouTube transcript endpoint fits your workflow. Keep result counts and token budgets bounded.
Bound result counts, cache repeated requests, and monitor credits on the pricing and console surfaces.
Back off on 429 responses, reduce concurrency, and avoid aggressive retry loops.
Crawlora returns platform-specific JSON for supported sources instead of relying on raw HTML extraction.
Start with Google Search or YouTube, normalize the response, then connect it to your agent or retrieval flow.