Datasets API endpoint
Use Crawlora's Tech Stack Dataset Search API to search or inspect stored structured datasets as JSON. This page includes request parameters, cURL examples, response schema, validation behavior, credit cost, and a Playground link for testing before integration. Dataset endpoints read indexed records and do not apply proxy routing.
/datasets/techstack/searchSearches the website tech-stack index (dataset id enum value `techstack`) — one record per site listing the web technologies it is built with (frameworks, CMS, e-commerce, analytics, CDNs, servers, and more), BuiltWith / Wappalyzer-style. The reverse-index filters are the point: repeat `technology` to require several at once (AND), `any_of` to match at least one (OR), and `not` to exclude — e.g. sites on `Shopify` and `Klaviyo` but not `Recharge`. Sort enum: `relevance`, `rank_asc`, `tech_count_desc`, `domain_asc`, `crawled_desc`. render_tier enum: `http`, `browser`. Developers commonly use this endpoint for repeatable dataset search, filtering, facets, local business enrichment, analytics, exports, and internal tools that need structured records beyond the limited manual refinement available in the Google Maps app. Authentication uses the x-api-key header, usage is metered with the credit cost shown on this page, and the request does not trigger live scraping or proxy routing.
Request parameters are generated from the active endpoint catalog. Dataset parameters filter, page, facet, or locate stored structured records; they do not configure a live scraper or proxy path.
| Parameter | Type | Required | Default | Description | Example |
|---|---|---|---|---|---|
| q | string | No | Substring match on the site domain, max 256 characters | ||
| technology | array | No | Repeatable exact technology name the site MUST use (AND), e.g. technology=Shopify&technology=Klaviyo | ||
| any_of | array | No | Repeatable exact technology name; the site must use at least one (OR) | ||
| not | array | No | Repeatable exact technology name the site must NOT use (excludes) | ||
| category | string | No | Exact category filter, e.g. Ecommerce, CMS, Analytics, Payment, CDN | ||
| cms | string | No | Exact CMS filter, e.g. WordPress, Shopify, Webflow | ||
| ecommerce | string | No | Exact e-commerce platform filter, e.g. Shopify, WooCommerce, Magento | ||
| cdn | string | No | Exact CDN / hosting filter, e.g. Cloudflare, Fastly, Vercel | ||
| web_server | string | No | Exact web-server filter, e.g. nginx, Apache, IIS | ||
| server_language | string | No | Exact server language / framework filter, e.g. PHP, ASP.NET, Ruby on Rails | ||
| tld | string | No | Exact top-level-domain filter, e.g. com, org, io | ||
| render_tier | string | No | Fetch-tier filter. Enum: http, browser | ||
| seed_source | string | No | Source filter for where the domain was discovered, e.g. tranco | ||
| has_captcha | boolean | No | true keeps only sites with a detected CAPTCHA | ||
| reachable | boolean | No | true keeps only sites whose homepage was fetched, false only sites that could not be fetched | ||
| min_tech_count | integer | No | Minimum number of detected technologies, 0 or greater | ||
| run_id | string | No | the latest run | Scan run id; defaults to the latest run | |
| sort | string | No | Sort enum: relevance, rank_asc, tech_count_desc, domain_asc, crawled_desc | ||
| page | integer | No | 1 | Page number, defaults to 1 | |
| page_size | integer | No | 20 and maxes at 100 | Page size, defaults to 20 and maxes at 100; page * page_size must be <= 10000 | |
| x-api-key (header) | string | Yes | API key required |
curl -X GET "https://api.crawlora.net/api/v1/datasets/techstack/search?q=coffee&has_captcha=true&reachable=true&page=1" \ -H "x-api-key: $CRAWLORA_API_KEY"
Send your scraping API key in the x-api-key header. Use the console API Keys page to rotate or select the active key.
Endpoint usage is metered in credits. The plan prices, included credits, limits, and overage rates below match the active backend billing configuration.
| Plan | Price | Included credits | Daily cap | Rate limit | Overage |
|---|---|---|---|---|---|
| Free | $0/mo | 2,000 | 500 daily credits | 5/min | No overage |
| Starter | $9/mo | 20,000 | 5,000 daily credits | 15/min | $0.75/1,000 overage credits when enabled |
| Growth | $29/mo | 100,000 | 25,000 daily credits | 45/min | $0.45/1,000 overage credits when enabled |
| Pro | $79/mo | 400,000 | No daily cap | 120/min | $0.30/1,000 overage credits |
| Business | $199/mo | 1,200,000 | No daily cap | 300/min | $0.20/1,000 overage credits |
| Enterprise | $499/mo | 5,000,000 | No daily cap | 1,000/min | $0.12/1,000 overage credits |
This endpoint reads stored indexed dataset records. It does not execute a live upstream Google Maps request, browser session, or proxy-routed scraping job.
Some targets require real browser execution because the data is loaded through JavaScript, dynamic rendering, or interaction-like browser behavior.
For supported endpoints, Crawlora can route requests through a managed browser cluster. This allows Crawlora to execute JavaScript, load dynamic content, apply browser-level request behavior, and normalize the rendered result into JSON.
You do not need to operate your own Playwright, Puppeteer, Chrome, proxy, queue, or retry infrastructure.
- Each item carries `domain`, `tld`, `rank`, `reachable`, `technologies` (each with `name`, `categories`, `confidence`, `version`, `evidence`), `technology_names`, `categories`, `tech_count`, the `cms`/`ecommerce`/`cdn`/`web_server`/`server_language` rollups, `render_tier`, and `detector_version`. - The maximum result window is `10000`; `page * page_size` must not exceed `10000`. - Invalid enum values return the standard invalid params envelope. - Returns an empty `items` array (not an error) when nothing matches or the dataset has no data yet. - Does not trigger live scraping. Example response: ```json { "code": 200, "msg": "OK", "data": { "dataset": "techstack", "items": [ { "domain": "example-store.com", "tld": "com", "rank": 5231, "reachable": true, "technologies": [ { "name": "Shopify", "categories": ["Ecommerce"], "confidence": "high", "evidence": "markup: cdn.shopify.com" }, { "name": "Cloudflare", "categories": ["CDN"], "confidence": "high", "evidence": "header: cf-ray" } ], "technology_names": ["Shopify", "Cloudflare"], "categories": ["CDN", "Ecommerce"], "tech_count": 2, "ecommerce": "Shopify", "cdn": "Cloudflare", "render_tier": "http", "detector_version": "2026.07.4" } ], "page": 1, "page_size": 20, "total": 1, "sort": "rank_asc" } } ```
Crawlora does not silently return invalid dataset search results when filters, pagination, coordinates, or stored record lookups cannot be satisfied.
| Status | Common failure case |
|---|---|
| 400 | Invalid input, missing required parameter, invalid enum, bad coordinate pair, or result window beyond the dataset limit |
| 404 | Requested stored dataset item is not present |
| 429 | Plan or endpoint rate limit exceeded |
| 500 | Internal dataset query or storage error |
When possible, Crawlora returns structured error context so your integration can adjust filters, page size, location inputs, or lookup identifiers.
| Status | Description | Schema |
|---|---|---|
| 400 | Bad Request | #/definitions/app.Response |
| 429 | Too Many Requests | #/definitions/app.Response |
| 500 | Internal Server Error | #/definitions/app.Response |
{
"code": 200,
"msg": "OK",
"data": {
"dataset": "techstack",
"items": [
{
"domain": "example-store.com",
"tld": "com",
"rank": 5231,
"reachable": true,
"technologies": [
{
"name": "Shopify",
"categories": [
"Ecommerce"
],
"confidence": "high",
"evidence": "markup: cdn.shopify.com"
},
{
"name": "Cloudflare",
"categories": [
"CDN"
],
"confidence": "high",
"evidence": "header: cf-ray"
}
],
"technology_names": [
"Shopify",
"Cloudflare"
],
"categories": [
"CDN",
"Ecommerce"
],
"tech_count": 2,
"ecommerce": "Shopify",
"cdn": "Cloudflare",
"render_tier": "http",
"detector_version": "2026.07.4"
}
],
"page": 1,
"page_size": 20,
"total": 1,
"sort": "rank_asc"
}
}Request schema
No body schema
Response schema
#/definitions/datasets.techstackSearchResponseDoc
| Field | Type | Required | Enum | Bounds | Example | Description |
|---|---|---|---|---|---|---|
| code | integer | No | 200 | |||
| data | datasets.TechstackSearchResponse | No | ||||
| data.dataset | string | No | ||||
| data.items | array | No | ||||
| data.items[].analytics | array | No | ||||
| data.items[].categories | array | No | distinct, sorted; category facets | |||
| data.items[].category | string | No | optional seed category (usually empty) | |||
| data.items[].cdn | string | No | ||||
| data.items[].cms | string | No | single-value rollups (denormalized from Technologies for cheap facets/filters) | |||
| data.items[].detector_version | string | No | ||||
| data.items[].domain | string | No | identity | |||
| data.items[].ecommerce | string | No | ||||
| data.items[].final_url | string | No | URL that produced the adopted body (after www. fallback / redirects) | |||
| data.items[].has_captcha | boolean | No | ||||
| data.items[].method_version | string | No | ||||
| data.items[].probed_at | string | No | RFC3339 — when last probed | |||
| data.items[].rank | integer | No | Tranco rank when seeded from the top-1M list | |||
| data.items[].reachable | boolean | No | ||||
| data.items[].render_tier | string | No | provenance / reproducibility | |||
| data.items[].run_date | string | No | ||||
| data.items[].run_id | string | No | ||||
| data.items[].scan_method | string | No | ||||
| data.items[].schema_version | integer | No | ||||
| data.items[].scheme | string | No | ||||
| data.items[].seed_source | string | No | tranco | internal-harvest | |||
| data.items[].server_language | string | No | ||||
| data.items[].source_url | string | No | the exact seed URL probed | |||
| data.items[].status | integer | No | ||||
| data.items[].tech_count | integer | No | ||||
| data.items[].technologies | array | No | detections (the payload) | |||
| data.items[].technologies[].categories | array | No | ||||
| data.items[].technologies[].confidence | string | No | ||||
| data.items[].technologies[].evidence | string | No | ||||
| data.items[].technologies[].name | string | No | ||||
| data.items[].technologies[].version | string | No | ||||
| data.items[].technology_names | array | No | flattened for AND/NOT term filters + facets | |||
| data.items[].tld | string | No | ||||
| data.items[].web_server | string | No | ||||
| data.page | integer | No | ||||
| data.page_size | integer | No | ||||
| data.sort | string | No | ||||
| data.total | integer | No | ||||
| msg | string | No | OK |
Use environment variables for secrets and keep Crawlora API keys server-side.
curl -X GET "https://api.crawlora.net/api/v1/datasets/techstack/search?q=coffee&has_captcha=true&reachable=true&page=1" \
-H "x-api-key: $CRAWLORA_API_KEY"Crawlora is designed for responsible structured public web data workflows. Customers are responsible for using Crawlora in compliance with applicable laws, third-party rights, target-platform rules, and Crawlora terms.
Read Crawlora terms