Docs menu

Crawlora Docs

Build an AI Agent Web Data Workflow

Expose selected Crawlora endpoints as tool calls so agents can retrieve normalized JSON before summarizing or ranking results.

Overview

Expose selected Crawlora endpoints as tool calls so agents can retrieve normalized JSON before summarizing or ranking results.

What you will build

  • Agent receives task
  • Planner selects Crawlora tool or endpoint
  • Crawlora returns normalized JSON
  • Agent summarizes, ranks, or compares
  • Store result or produce report

APIs used

Only endpoints that exist in the generated endpoint metadata are linked here. Missing optional endpoints are intentionally omitted.

POSTGoogleapiKey3 credits/request

Google search API

/google/search

Returns normalized Google web search results. Results are fetched through Rayobrowse-rendered Chrome with availability fanout and stale-cache fallback when available. The endpoint returns 502 when Google serves a challenge page or unusable HTML. Rate limit is enforced at 1 request per second, and if the limit is exceeded a 429 status code is returned with rate limit headers.

POSTGoogle MapapiKey5 credits/request

Google Maps search API

/google/map/search

Returns results from Google Maps based on search options. Rate limit is enforced at 1 request per second.

GETYouTubeapiKey3 credits/request

Retrieve transcript for a YouTube video

/youtube/transcript/{id}

Returns transcript segments for a YouTube video using YouTube's native player captions. Set `format=text`, `format=srt`, or `format=vtt` to receive plain-text output instead of the standard response envelope.

GETTikTokapiKey8 credits/request

Search TikTok videos

/tiktok/search

Searches TikTok videos by keyword with cursor-based pagination.

GETAppStoreapiKey1 credit/request

Search the App Store

/appstore/search

Returns App Store search results for a term. Set `ids_only=true` to return only app IDs.

GETGooglePlayapiKey1 credit/request

Search Google Play

/googleplay/search

Returns Google Play search results for a term.

GETAmazonapiKey2 credits/request

Retrieve Amazon product details

/amazon/product/{asin}

Returns normalized product details for an Amazon ASIN on `amazon.com`, including pricing, availability, overview data, inline review samples, and descriptive content.

GETProductHuntapiKey3 credits/request

Search for products, users, or launches on Product Hunt

/producthunt/search

Performs a full-text Product Hunt search and returns matching products, users, or launches.

Data model

FieldNotes
task_idWorkflow field; map to exact endpoint response fields from endpoint docs.
tool_nameWorkflow field; map to exact endpoint response fields from endpoint docs.
endpointWorkflow field; map to exact endpoint response fields from endpoint docs.
inputWorkflow field; map to exact endpoint response fields from endpoint docs.
result_summaryWorkflow field; map to exact endpoint response fields from endpoint docs.
source_idsWorkflow field; map to exact endpoint response fields from endpoint docs.
request_idWorkflow field; map to exact endpoint response fields from endpoint docs.
credits_usedInclude when response or usage data makes it available.
created_atWorkflow field; map to exact endpoint response fields from endpoint docs.

Step-by-step workflow

  1. 1.Agent receives task
  2. 2.Planner selects Crawlora tool or endpoint
  3. 3.Crawlora returns normalized JSON
  4. 4.Agent summarizes, ranks, or compares
  5. 5.Store result or produce report

Example request

This example uses the real Google search API endpoint. Exact request fields come from the endpoint metadata.

Recipe request

Use environment variables for secrets and keep Crawlora API keys server-side.

curl -X POST "https://api.crawlora.net/api/v1/google/search" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"country":"us","keyword":"chatgpt","language":"en","limit":10,"page":1}'

Example response

Use endpoint detail pages for exact response schemas. This recipe does not invent response fields.

Generated example response

{
  "code": 200,
  "msg": "OK",
  "data": {
    "result": [
      {
        "position": 1,
        "title": "ChatGPT",
        "website_name": "ChatGPT",
        "icon": "",
        "link": "https://chatgpt.com/",
        "Snippet": "ChatGPT helps you get answers and create."
      }
    ]
  }
}

Illustrative tool schema

This is an illustrative agent tool schema, not an exact MCP schema. Use the MCP docs when a specific tool schema is available.

Illustrative tool schema

{
  "name": "crawlora_google_search",
  "description": "Search public Google results through Crawlora and return structured results.",
  "parameters": {
    "type": "object",
    "properties": {
      "query": { "type": "string" },
      "country": { "type": "string" },
      "language": { "type": "string" }
    },
    "required": ["query"]
  }
}

Storage/output suggestion

Store tool inputs, response summaries, and source identifiers so agent outputs can be audited.

Error handling

  • Validate required inputs before calling Crawlora
  • Retry 429 and temporary 5xx responses with capped backoff
  • Log endpoint, input, timestamp, and request ID when present
  • Treat empty results as a state your application can handle
  • Open /docs/errors for production retry guidance

Rate-limit and credit planning

Estimate usage by multiplying requests by endpoint credit cost. The table below only shows real credit costs available from the billing constants.

EndpointCredit costDocs
Google search API3 credits/request/docs/Google/google-search
Google Maps search API5 credits/request/docs/Google%20Map/google-map-search
Retrieve transcript for a YouTube video3 credits/request/docs/YouTube/youtube-transcript
Search TikTok videos8 credits/request/docs/TikTok/tiktok-search
Search the App Store1 credit/request/docs/AppStore/appstore-search
Search Google Play1 credit/request/docs/GooglePlay/googleplay-search
Retrieve Amazon product details2 credits/request/docs/Amazon/amazon-product
Search for products, users, or launches on Product Hunt3 credits/request/docs/ProductHunt/producthunt-search

Production checklist

  • Keep API keys server-side
  • Use request timeouts
  • Back off on rate limits
  • Store raw responses or source IDs for auditability
  • Monitor credits and failures
  • Avoid unnecessary refreshes
  • Review responsible-use requirements

Responsible public web data workflows

Crawlora is designed for responsible structured public web data workflows. Customers are responsible for using Crawlora in compliance with applicable laws, third-party rights, target-platform rules, and Crawlora terms.

Read Crawlora terms

Related APIs and pages

Build this workflow with real endpoint docs

Use this recipe for workflow shape, then rely on endpoint reference pages for exact paths, request schemas, response schemas, and credit costs.