Developer guides
Use Crawlora APIs to provide structured public web data to LangChain workflows without relying on raw HTML as the primary input.
Verified HTTP pattern
GET /bing/search
Request
GET https://api.crawlora.net/api/v1/bing/search?q=best+CRM+software&country=us&count=10
x-api-key: $CRAWLORA_API_KEYBase URL
https://api.crawlora.net/api/v1
Auth header
x-api-key
Example endpoint
GET /bing/search
This repository does not contain an official LangChain package. Treat this guide as a custom tool and loader pattern.
Developer workflow
LangChain workflows often need external data. Crawlora can provide normalized JSON from supported public platforms, making it easier to summarize, classify, embed, or store results.
Developer workflow
Developer workflow
Adapt the wrapper to your installed LangChain version's current tool API.
import os
import requests
API_KEY = os.environ["CRAWLORA_API_KEY"]
BASE_URL = "https://api.crawlora.net/api/v1"
def crawlora_bing_search(query: str) -> dict:
response = requests.get(
f"{BASE_URL}/bing/search",
headers={"x-api-key": API_KEY},
params={"q": query, "country": "us"},
timeout=60,
)
response.raise_for_status()
return response.json()
# Adapt this function to your installed LangChain version's tool wrapper.Developer workflow
Transform Crawlora JSON into simple document dictionaries before passing them into your retrieval or storage layer.
def crawlora_results_to_documents(payload: dict) -> list[dict]:
results = payload.get("data", {}).get("results", [])
return [
{
"page_content": item.get("description") or item.get("title") or "",
"metadata": {
"title": item.get("title"),
"url": item.get("url"),
"position": item.get("position"),
"source": "crawlora_bing_search",
},
}
for item in results
]Developer workflow
Developer workflow
A LangChain tool runs when the agent decides it needs data mid-conversation; a document loader runs before the conversation to fill a vector store. Search endpoints (Bing, Brave, DuckDuckGo, Yahoo) are natural tools: the agent forms a keyword and gets back positions, titles, URLs and snippets it can reason over. Record endpoints (Google Maps places, Amazon products, SEC filings, YouTube transcripts) are natural loaders: each returns a document with an ID for metadata. Many pipelines use both: a loader to build the corpus, a tool for anything the corpus does not cover.
Developer workflow
A tool is only as good as its description. State the platform, the input the endpoint expects (a keyword and country for search; an ASIN for an Amazon product; a place ID for Google Maps), and the shape of the output in one or two sentences. Return a compact string or a small JSON object rather than the whole response, and include the fields the model will need to cite (URL, position, title). Bound the result count; ten search results is usually enough context and keeps token spend predictable.
Developer workflow
For retrieval, run the loader on a schedule: resolve the IDs you track, call the record endpoint per ID, convert each response into a Document with source, ID, fetched_at and any filterable fields in metadata, chunk long text (transcripts, filing sections) on natural boundaries, embed, and upsert into your vector store keyed on record ID plus chunk index. Because each endpoint has a fixed credit weight, a refresh costs the number of records, so re-pull only the endpoints whose data actually changes (prices, rankings) and leave the stable ones (filings, transcripts).
Developer workflow
Use Crawlora for structured public web data workflows. Customers are responsible for compliance with applicable laws, third-party rights, platform rules, and Crawlora terms. Keep API keys server-side, validate inputs, and avoid collecting or storing unnecessary sensitive data.
Read Crawlora termsDeveloper workflow
Use these pages to move between endpoint discovery, examples, pricing, and responsible-use guidance.
Developer workflow
Common questions for this Crawlora developer integration path.
This frontend repository does not contain an official Crawlora LangChain package. Use a custom tool or loader wrapper around the HTTP API.
Use a tool for agent-time decisions and a loader for scheduled ingestion or retrieval indexing.
Yes. Convert normalized result items into documents with clear metadata before embedding.
Yes, if the selected YouTube transcript endpoint fits your workflow. Keep result counts and token budgets bounded.
Bound result counts, cache repeated requests, and monitor credits on the pricing and console surfaces.
Back off on 429 responses, reduce concurrency, and avoid aggressive retry loops.
Crawlora returns platform-specific JSON for supported sources instead of relying on raw HTML extraction.
Search endpoints (Bing, Brave, DuckDuckGo, Yahoo) make good tools for live lookups. Record endpoints (Google Maps, Amazon, SEC, YouTube transcripts) make good loaders for batch ingestion into a vector store. The HTTP call is the same; only the wrapper differs.
Name the platform, the input the endpoint expects (keyword and country for search, ASIN for Amazon, place ID for Maps), and the output fields in one or two sentences. Return compact citable fields and cap the result count inside the tool.
Yes. The hosted MCP server exposes every documented endpoint as an MCP tool under the same API key, for agent runtimes that support MCP. For scheduled ingestion, use the HTTP API in a loader.
Start with Bing Search or YouTube, normalize the response, then connect it to your agent or retrieval flow.