Datasets API endpoint
Use Crawlora's Tech Stack Dataset Item API to search or inspect stored structured datasets as JSON. This page includes request parameters, cURL examples, response schema, validation behavior, credit cost, and a Playground link for testing before integration. Dataset endpoints read indexed records and do not apply proxy routing.
/datasets/techstack/items/{domain}Returns one site by its domain from dataset id enum value `techstack`, including every detected technology (name, categories, confidence, version, evidence) plus the CMS / e-commerce / CDN / web-server / server-language rollups. Also flags whether the domain looks like backend CDN/DNS/cloud-vendor infrastructure rather than a real, human-navigable site (see `is_infrastructure`). Returns 404 when the domain is not in the index. Developers commonly use this endpoint for repeatable dataset search, filtering, facets, local business enrichment, analytics, exports, and internal tools that need structured records beyond the limited manual refinement available in the Google Maps app. Authentication uses the x-api-key header, usage is metered with the credit cost shown on this page, and the request does not trigger live scraping or proxy routing.
Request parameters are generated from the active endpoint catalog. Dataset parameters filter, page, facet, or locate stored structured records; they do not configure a live scraper or proxy path.
| Parameter | Type | Required | Default | Description | Example |
|---|---|---|---|---|---|
| domain (path) | string | Yes | Site domain, e.g. shopify.com (a scheme and www. are stripped) | ||
| x-api-key (header) | string | Yes | API key required |
curl -X GET "https://api.crawlora.net/api/v1/datasets/techstack/items/%3Cdomain%3E" \ -H "x-api-key: $CRAWLORA_API_KEY"
Send your scraping API key in the x-api-key header. Use the console API Keys page to rotate or select the active key.
Endpoint usage is metered in credits. The plan prices, included credits, limits, and overage rates below match the active backend billing configuration.
| Plan | Price | Included credits | Daily cap | Rate limit | Overage |
|---|---|---|---|---|---|
| Free | $0/mo | 2,000 | 500 daily credits | 5/min | No overage |
| Starter | $9/mo | 20,000 | 5,000 daily credits | 15/min | $0.75/1,000 overage credits when enabled |
| Growth | $29/mo | 100,000 | 25,000 daily credits | 45/min | $0.45/1,000 overage credits when enabled |
| Pro | $79/mo | 400,000 | No daily cap | 120/min | $0.30/1,000 overage credits |
| Business | $199/mo | 1,200,000 | No daily cap | 300/min | $0.20/1,000 overage credits |
| Enterprise | $499/mo | 5,000,000 | No daily cap | 1,000/min | $0.12/1,000 overage credits |
This endpoint reads stored indexed dataset records. It does not execute a live upstream Google Maps request, browser session, or proxy-routed scraping job.
Some targets require real browser execution because the data is loaded through JavaScript, dynamic rendering, or interaction-like browser behavior.
For supported endpoints, Crawlora can route requests through a managed browser cluster. This allows Crawlora to execute JavaScript, load dynamic content, apply browser-level request behavior, and normalize the rendered result into JSON.
You do not need to operate your own Playwright, Puppeteer, Chrome, proxy, queue, or retry infrastructure.
- The item carries `domain`, `tld`, `rank`, `is_infrastructure` (`true` when the domain looks like backend CDN/DNS/cloud-vendor infrastructure — e.g. `cloudfront.net`, `gtld-servers.net` — rather than a real, human-navigable site; a best-effort heuristic classification, so `false`/absent is not a guarantee the domain is a real site), `reachable`, `status` (HTTP status of the last probe attempt, when one was made), `technologies` (each with `name`, `categories`, `confidence`, `version`, `evidence`), `technology_names`, `categories`, `tech_count`, the `cms`/`ecommerce`/`cdn`/`web_server`/`server_language` rollups, `render_tier`, `detector_version`, and the crawl provenance (`run_id`, `crawled_at`). - When the last probe attempt didn't fully pass, the item also carries `failure_reason` (a short classification, e.g. `"blocked/challenge markers found"`, `"bad status 503"`, `"request failed"`), `probe_error` (the first line of the underlying transport error, when there was one), and `block_matched` (which block/challenge marker(s) were found, when `failure_reason` is a block/challenge classification). This can appear even when `reachable` is `true` — a blocked page returning a non-5xx status still counts as reachable while carrying a failure reason. Absent for a site that fully passed. - When the crawl looked for extra non-homepage pages (opt-in per run), the item also carries `secondary_pages_discovered` (count of candidate pages found on the homepage nav, regardless of fetch outcome), `secondary_pages_fetched` (count that were actually fetched successfully and contributed content — always <= discovered), `secondary_page_categories` (which categories were successfully fetched and contributed, e.g. `login`, `docs`, `cart`), `secondary_pages` (the full per-page audit trail — every discovered candidate, each entry `{category, url, fetched}`; `fetched: false` means the candidate was found on the homepage nav but the fetch itself failed, e.g. blocked or timed out), and `secondary_only_technology_names` (technology names detected only via a fetched secondary page, not the homepage itself — a subset of `technology_names`). Absent when no secondary pages were discovered for that run. - Returns the standard not-found envelope when the domain is not in the index. - Does not trigger live scraping; for a live, on-demand fingerprint of any URL use `/web/techstack`. Example response: ```json { "code": 200, "msg": "OK", "data": { "domain": "example-store.com", "tld": "com", "rank": 5231, "reachable": true, "technologies": [ { "name": "Shopify", "categories": ["Ecommerce"], "confidence": "high", "evidence": "markup: cdn.shopify.com" }, { "name": "Google Analytics", "categories": ["Analytics"], "confidence": "high", "evidence": "script: googletagmanager.com/gtag/js" }, { "name": "Cloudflare", "categories": ["CDN"], "confidence": "high", "evidence": "header: cf-ray" } ], "technology_names": ["Shopify", "Google Analytics", "Cloudflare", "Afterpay"], "categories": ["Analytics", "CDN", "Ecommerce", "Payment"], "tech_count": 4, "ecommerce": "Shopify", "cdn": "Cloudflare", "secondary_pages_discovered": 2, "secondary_pages_fetched": 1, "secondary_page_categories": ["cart"], "secondary_pages": [ { "category": "cart", "url": "https://example-store.com/cart", "fetched": true }, { "category": "docs", "url": "https://example-store.com/docs", "fetched": false } ], "secondary_only_technology_names": ["Afterpay"], "render_tier": "http", "detector_version": "2026.07.4", "run_id": "top1m-2026-07-08" } } ```
Crawlora does not silently return invalid dataset search results when filters, pagination, coordinates, or stored record lookups cannot be satisfied.
| Status | Common failure case |
|---|---|
| 400 | Invalid input, missing required parameter, invalid enum, bad coordinate pair, or result window beyond the dataset limit |
| 404 | Requested stored dataset item is not present |
| 429 | Plan or endpoint rate limit exceeded |
| 500 | Internal dataset query or storage error |
When possible, Crawlora returns structured error context so your integration can adjust filters, page size, location inputs, or lookup identifiers.
| Status | Description | Schema |
|---|---|---|
| 400 | Bad Request | #/definitions/app.Response |
| 404 | Not Found | #/definitions/app.Response |
| 429 | Too Many Requests | #/definitions/app.Response |
| 500 | Internal Server Error | #/definitions/app.Response |
{
"code": 200,
"msg": "OK",
"data": {
"domain": "example-store.com",
"tld": "com",
"rank": 5231,
"reachable": true,
"technologies": [
{
"name": "Shopify",
"categories": [
"Ecommerce"
],
"confidence": "high",
"evidence": "markup: cdn.shopify.com"
},
{
"name": "Google Analytics",
"categories": [
"Analytics"
],
"confidence": "high",
"evidence": "script: googletagmanager.com/gtag/js"
},
{
"name": "Cloudflare",
"categories": [
"CDN"
],
"confidence": "high",
"evidence": "header: cf-ray"
}
],
"technology_names": [
"Shopify",
"Google Analytics",
"Cloudflare",
"Afterpay"
],
"categories": [
"Analytics",
"CDN",
"Ecommerce",
"Payment"
],
"tech_count": 4,
"ecommerce": "Shopify",
"cdn": "Cloudflare",
"secondary_pages_discovered": 2,
"secondary_pages_fetched": 1,
"secondary_page_categories": [
"cart"
],
"secondary_pages": [
{
"category": "cart",
"url": "https://example-store.com/cart",
"fetched": true
},
{
"category": "docs",
"url": "https://example-store.com/docs",
"fetched": false
}
],
"secondary_only_technology_names": [
"Afterpay"
],
"render_tier": "http",
"detector_version": "2026.07.4",
"run_id": "top1m-2026-07-08"
}
}Request schema
No body schema
Response schema
#/definitions/datasets.techstackItemResponseDoc
| Field | Type | Required | Enum | Bounds | Example | Description |
|---|---|---|---|---|---|---|
| code | integer | No | 200 | |||
| data | es.TechstackRecord | No | ||||
| data.analytics | array | No | ||||
| data.block_matched | array | No | block/challenge marker strings that matched, when FailureReason is a block/challenge classification | |||
| data.categories | array | No | distinct, sorted; category facets | |||
| data.category | string | No | optional seed category (usually empty) | |||
| data.cdn | string | No | ||||
| data.cms | string | No | single-value rollups (denormalized from Technologies for cheap facets/filters) | |||
| data.detector_version | string | No | ||||
| data.domain | string | No | identity | |||
| data.ecommerce | string | No | ||||
| data.failure_reason | string | No | failure diagnostics: populated whenever a probe attempt didn't fully pass, even if the site still counts as reachable (e.g. a blocked page returning a non-5xx status). Unset for a fully passing site. | |||
| data.final_url | string | No | URL that produced the adopted body (after www. fallback / redirects) | |||
| data.has_captcha | boolean | No | ||||
| data.is_infrastructure | boolean | No | IsInfrastructure flags a backend CDN/DNS/cloud-vendor/registry hostname (e.g. cloudfront.net, gtld-servers.net) rather than a real, human-navigable website. Best-effort heuristic: a false negative (unflagged) is not proof a domain is a real site. | |||
| data.method_version | string | No | ||||
| data.probe_error | string | No | raw transport error's first line (e.g. "context deadline exceeded", "no such host"), when there was one | |||
| data.probed_at | string | No | RFC3339 — when last probed | |||
| data.rank | integer | No | Tranco rank when seeded from the top-1M list | |||
| data.reachable | boolean | No | ||||
| data.render_tier | string | No | provenance / reproducibility | |||
| data.run_date | string | No | ||||
| data.run_id | string | No | ||||
| data.scan_method | string | No | ||||
| data.schema_version | integer | No | ||||
| data.scheme | string | No | ||||
| data.secondary_only_technology_names | array | No | technology names detected ONLY via a fetched secondary page, not the homepage alone; a subset of TechnologyNames | |||
| data.secondary_page_categories | array | No | which categories were successfully fetched and contributed content, sorted (careers|login|blog|status|signup|docs|contact|trust|cart) | |||
| data.secondary_pages | array | No | full audit trail: every discovered candidate, fetched or not; detail view, stored not indexed | |||
| data.secondary_pages[].category | string | No | ||||
| data.secondary_pages[].fetched | boolean | No | ||||
| data.secondary_pages[].url | string | No | ||||
| data.secondary_pages_discovered | integer | No | secondary-page telemetry: populated when the crawl looked for extra non-homepage pages (opt-in per run) and discovered at least one on the homepage nav for this domain. "Discovered" and "fetched" are deliberately distinct: a candidate can be found on the homepage nav but still fail to fetch (e.g. blocked or timed out) — that failure is recorded in SecondaryPages (Fetched=false), not silently dropped. | |||
| data.secondary_pages_fetched | integer | No | count that were ACTUALLY fetched successfully and contributed content; <= SecondaryPagesDiscovered | |||
| data.seed_source | string | No | tranco | internal-harvest | |||
| data.server_language | string | No | ||||
| data.source_url | string | No | the exact seed URL probed | |||
| data.status | integer | No | ||||
| data.tech_count | integer | No | ||||
| data.technologies | array | No | detections (the payload) | |||
| data.technologies[].categories | array | No | ||||
| data.technologies[].confidence | string | No | ||||
| data.technologies[].evidence | string | No | ||||
| data.technologies[].name | string | No | ||||
| data.technologies[].version | string | No | ||||
| data.technology_names | array | No | flattened for AND/NOT term filters + facets | |||
| data.tld | string | No | ||||
| data.web_server | string | No | ||||
| msg | string | No | OK |
Use environment variables for secrets and keep Crawlora API keys server-side.
curl -X GET "https://api.crawlora.net/api/v1/datasets/techstack/items/%3Cdomain%3E" \
-H "x-api-key: $CRAWLORA_API_KEY"Crawlora is designed for responsible structured public web data workflows. Customers are responsible for using Crawlora in compliance with applicable laws, third-party rights, target-platform rules, and Crawlora terms.
Read Crawlora terms