Developer guides
Use Crawlora APIs to feed structured public web data into LlamaIndex readers, tools, and retrieval pipelines instead of parsing raw HTML.
Verified HTTP pattern
POST /google/search
Request
POST https://api.crawlora.net/api/v1/google/search
x-api-key: $CRAWLORA_API_KEY
Content-Type: application/json
{
"country": "us",
"keyword": "best CRM software",
"language": "en",
"limit": 10,
"page": 1
}Base URL
https://api.crawlora.net/api/v1
Auth header
x-api-key
Example endpoint
POST /google/search
This repository does not contain an official LlamaIndex package. Treat this guide as a custom reader and tool pattern around the HTTP API.
Developer workflow
LlamaIndex builds retrieval and agent workflows over your data. Crawlora can provide normalized JSON from supported public platforms, making it easier to convert into Documents, embed, and query than raw HTML. The same agent-native structured data also backs Crawlora's hosted MCP tools.
Developer workflow
Developer workflow
Convert Crawlora results into LlamaIndex Documents with clear metadata before indexing.
import requests
from llama_index.core import Document
BASE_URL = "https://api.crawlora.net/api/v1"
def crawlora_documents(query: str, limit: int = 10) -> list[Document]:
resp = requests.post(
f"{BASE_URL}/google/search",
headers={"x-api-key": "YOUR_API_KEY", "Content-Type": "application/json"},
json={"keyword": query, "country": "us", "language": "en", "limit": limit, "page": 1},
timeout=60,
)
resp.raise_for_status()
rows = resp.json().get("data", {}).get("result", [])
return [
Document(
text=row.get("Snippet", ""),
metadata={"url": row.get("link"), "title": row.get("title"), "position": row.get("position"), "source": "google_search"},
)
for row in rows
]Developer workflow
Build an index from the Documents, then query it in your RAG or agent flow.
from llama_index.core import VectorStoreIndex
docs = crawlora_documents("retrieval augmented generation")
index = VectorStoreIndex.from_documents(docs)
answer = index.as_query_engine().query("Summarize the latest on RAG")Developer workflow
Developer workflow
A reader runs ahead of time and turns many records into Documents for an index; a tool runs at query time and answers one question. Search endpoints (Google, Bing, Brave, DuckDuckGo) suit both: as a reader they seed an index with snippets and URLs, as a tool they let an agent look something up live. Record endpoints (Google Maps places, Amazon products, YouTube transcripts, SEC filings) suit readers: each response is a document with natural metadata (place ID, ASIN, video ID, accession number). Transcript and filing-section endpoints return long text and should be chunked before indexing.
Developer workflow
Put the fields you will filter on into Document metadata rather than into the text: the source platform, the record ID, the position for search results, the fetched-at timestamp, and the query or region that produced the record. LlamaIndex metadata filters can then scope a retrieval to one platform or one time window without re-embedding. Keep the raw response alongside the Document so you can rebuild the index if the chunking strategy changes.
Developer workflow
Most web data changes slowly except where it does not: prices and rankings move daily, transcripts and filings almost never. Schedule refreshes per endpoint rather than per index. Re-pull search results and product records on a cadence, dedupe by record ID against the existing index, and only re-embed Documents whose text changed. Because every endpoint has a fixed credit weight, the refresh cost is predictable per record. For agent-time freshness, wrap the same call as a tool and let the agent decide when to look something up live.
Developer workflow
Use Crawlora for structured public web data workflows. Customers are responsible for compliance with applicable laws, third-party rights, platform rules, and Crawlora terms. Keep API keys server-side, validate inputs, and avoid collecting or storing unnecessary sensitive data.
Read Crawlora termsDeveloper workflow
Use these pages to move between endpoint discovery, examples, pricing, and responsible-use guidance.
Developer workflow
Common questions for this Crawlora developer integration path.
This frontend repository does not contain an official Crawlora LlamaIndex package. Use a custom reader or tool wrapper around the HTTP API.
Use a reader for scheduled ingestion and indexing, and a tool for agent-time queries.
Yes. Convert normalized result items into Documents with clear metadata before embedding.
Yes, if the selected transcript endpoint fits your workflow. Keep result counts and token budgets bounded.
Bound result counts, cache repeated requests, and monitor credits on the pricing and console surfaces.
Back off on 429 responses, reduce concurrency, and avoid aggressive retry loops.
Record endpoints: Google Maps places, Amazon products, YouTube transcripts, SEC filing sections. Each response maps to one Document with the record ID in metadata. Search endpoints work as readers for seeding and as tools for live lookups.
Refresh per endpoint on its own cadence, dedupe on record ID, and re-embed only Documents whose text changed. Prices and rankings change daily; transcripts and filings rarely do. Fixed credit weights make the refresh cost the number of records you re-pull.
Yes, if your agent runtime supports MCP. The hosted server exposes every documented endpoint as a tool under the same API key, which removes the wrapper code for agent-time calls; loaders for batch ingestion still use the HTTP API.
Start with Google Search or YouTube transcripts, convert to Documents, then index and query.