Datasets API endpoint
Searches the website tech-stack index (dataset id enum value `techstack`) — one record per site listing the web technologies it is built with (frameworks, CMS, e-commerce, analytics, CDNs, servers, and more), BuiltWith / Wappalyzer-style. The reverse-index filters are the point: repeat `technology` to require several at once (AND), `any_of` to match at least one (OR), and `not` to exclude — e.g. sites on `Shopify` and `Klaviyo` but not `Recharge`. Sort enum: `relevance`, `rank_asc`, `tech_count_desc`, `domain_asc`, `crawled_desc`. render_tier enum: `http`, `browser`. Billing: this search costs 1 credit per record returned, with a minimum of 2 (an empty result costs 2, a full page of 100 costs 100).
/datasets/techstack/searchSearches the website tech-stack index — one record per site listing the web technologies it is built with (JavaScript frameworks, CMS, e-commerce platforms, analytics and tag managers, CDNs, web servers, server languages and more), BuiltWith / Wappalyzer-style. The reverse-index filters are the point: find every site that uses a given technology, or a combination of them.
Dataset id enum:
techstackQuery parameters:
q: optional substring match on the site domain, max 256 characters.technology: repeatable exact technology name the site must use (AND across all values), e.g. technology=Shopify&technology=Klaviyo. Use the names returned by search.any_of: repeatable exact technology name; the site must use at least one (OR).not: repeatable exact technology name the site must not use (excludes).category: optional exact category filter, e.g. Ecommerce, CMS, Analytics, Payment, CDN.cms: optional exact CMS filter, e.g. WordPress, Shopify, Webflow.ecommerce: optional exact e-commerce platform filter, e.g. Shopify, WooCommerce, Magento.cdn: optional exact CDN / hosting filter, e.g. Cloudflare, Fastly, Vercel.web_server: optional exact web-server filter, e.g. nginx, Apache, IIS.server_language: optional exact server language / framework filter, e.g. PHP, ASP.NET, Ruby on Rails.tld: optional exact top-level-domain filter, e.g. com, org, io.render_tier: optional fetch-tier filter. Allowed values: http, browser.seed_source: optional filter for where the domain was discovered, e.g. tranco.has_captcha: optional filter; true keeps only sites with a detected CAPTCHA.is_infrastructure: optional filter; false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but were never meant to serve a public homepage, keeping only real, human-navigable sites; true keeps only those backend hostnames.reachable: optional filter; true keeps only sites whose homepage was fetched, false only sites that could not be fetched.min_tech_count: optional minimum number of detected technologies, 0 or greater.run_id: optional scan run id; defaults to the latest run.sort: optional sort order. Allowed values: relevance, rank_asc, tech_count_desc, domain_asc, crawled_desc. Defaults to relevance with q, otherwise rank_asc.page: optional page number, defaults to 1.page_size: optional page size, defaults to 20 and maxes at 100.Response behavior:
domain, tld, rank, is_infrastructure (true when the domain looks like backend CDN/DNS/cloud-vendor infrastructure rather than a real, human-navigable site; a best-effort heuristic), reachable, technologies (each with name, categories, confidence, version, evidence), technology_names, categories, tech_count, the cms/ecommerce/cdn/web_server/server_language rollups, render_tier, and detector_version.secondary_pages_discovered (count of candidates found on the homepage nav, regardless of fetch outcome), secondary_pages_fetched (count actually fetched successfully), secondary_page_categories (which categories were successfully fetched, e.g. login, docs, cart), secondary_pages (the full per-page audit trail, each entry {category, url, fetched}), and secondary_only_technology_names (technologies detected only via a fetched secondary page, not the homepage itself — a subset of technology_names). Absent when no secondary pages were discovered for that run.10000; page * page_size must not exceed 10000.items array (not an error) when nothing matches or the dataset has no data yet.1 credit per record returned, minimum 2. A search that returns 20 records costs 20 credits, a full page of 100 costs 100, and an empty or one-record result costs 2. You are charged for what comes back, not for the page_size you ask for, but your balance must cover the requested page_size for the call to be accepted. The response header x-credit-cost shows what a call was charged.
| Field | In the sample | Value in the sample |
|---|---|---|
| dataset | Has a value | |
| items | Empty | |
| page | Has a value | |
| page_size | Has a value | |
| total | Has a value | 0 |
| sort | Has a value |
| Parameter | Type | Required | Default | Description | Example |
|---|---|---|---|---|---|
| q | string | No | Substring match on the site domain, max 256 characters | ||
| technology | array<string> | No | Repeatable exact technology name the site MUST use (AND), e.g. technology=Shopify&technology=Klaviyo | ||
| any_of | array<string> | No | Repeatable exact technology name; the site must use at least one (OR) | ||
| not | array<string> | No | Repeatable exact technology name the site must NOT use (excludes) | ||
| category | string | No | Exact category filter, e.g. Ecommerce, CMS, Analytics, Payment, CDN | ||
| cms | string | No | Exact CMS filter, e.g. WordPress, Shopify, Webflow | ||
| ecommerce | string | No | Exact e-commerce platform filter, e.g. Shopify, WooCommerce, Magento | ||
| cdn | string | No | Exact CDN / hosting filter, e.g. Cloudflare, Fastly, Vercel | ||
| web_server | string | No | Exact web-server filter, e.g. nginx, Apache, IIS | ||
| server_language | string | No | Exact server language / framework filter, e.g. PHP, ASP.NET, Ruby on Rails | ||
| tld | string | No | Exact top-level-domain filter, e.g. com, org, io | ||
| render_tier | string | No | Fetch-tier filter. Enum: http, browser Allowed values: http, browser. | ||
| seed_source | string | No | Source filter for where the domain was discovered, e.g. tranco | ||
| has_captcha | boolean | No | true keeps only sites with a detected CAPTCHA | ||
| is_infrastructure | boolean | No | false (the common case) excludes backend CDN/DNS/cloud-vendor hostnames that rank highly but were never meant to serve a public homepage, keeping only real, human-navigable sites; true keeps only those backend hostnames | ||
| reachable | boolean | No | true keeps only sites whose homepage was fetched, false only sites that could not be fetched | ||
| min_tech_count | integer | No | Minimum number of detected technologies, 0 or greater | ||
| run_id | string | No | the latest run | Scan run id; defaults to the latest run | |
| sort | string | No | Sort enum: relevance, rank_asc, tech_count_desc, domain_asc, crawled_desc Allowed values: relevance, rank_asc, tech_count_desc, domain_asc, crawled_desc. | ||
| page | integer | No | 1 | Page number, defaults to 1 | |
| page_size | integer | No | 20 and maxes at 100 | Page size, defaults to 20 and maxes at 100; page * page_size must be <= 10000 | |
| x-api-key (header) | string | Yes | API key required |
curl -X GET "https://api.crawlora.net/api/v1/datasets/techstack/search?q=coffee&render_tier=http&has_captcha=true&is_infrastructure=true&reachable=true&sort=relevance&page=1" \ -H "x-api-key: $CRAWLORA_API_KEY"
{
"code": 200,
"msg": "OK",
"data": {
"dataset": "techstack",
"items": [
{
"domain": "example-store.com",
"tld": "com",
"rank": 5231,
"reachable": true,
"technologies": [
{
"name": "Shopify",
"categories": [
"Ecommerce"
],
"confidence": "high",
"evidence": "markup: cdn.shopify.com"
},
{
"name": "Cloudflare",
"categories": [
"CDN"
],
"confidence": "high",
"evidence": "header: cf-ray"
}
],
"technology_names": [
"Shopify",
"Cloudflare"
],
"categories": [
"CDN",
"Ecommerce"
],
"tech_count": 2,
"ecommerce": "Shopify",
"cdn": "Cloudflare",
"render_tier": "http",
"detector_version": "2026.07.4"
}
],
"page": 1,
"page_size": 20,
"total": 1,
"sort": "rank_asc"
}
}Request schema
No body schema
Response schema
#/definitions/datasets.techstackSearchResponseDoc
| Field | Type | Required | Enum | Bounds | Example | Description |
|---|---|---|---|---|---|---|
| code | integer | No | 200 | |||
| data | datasets.TechstackSearchResponse | No | ||||
| data.dataset | string | No | Identifier of the dataset queried (`techstack`). | |||
| data.items | array | No | Matching website technology records on this page. | |||
| data.items[].analytics | array | No | Analytics, tag manager and tracking pixel technologies detected on the site. | |||
| data.items[].block_matched | array | No | block/challenge marker strings that matched, when FailureReason is a block/challenge classification | |||
| data.items[].categories | array | No | distinct, sorted; category facets | |||
| data.items[].category | string | No | optional seed category (usually empty) | |||
| data.items[].cdn | string | No | Main CDN, hosting or platform provider detected on the site. | |||
| data.items[].cms | string | No | single-value rollups (denormalized from Technologies for cheap facets/filters) | |||
| data.items[].detector_version | string | No | Version of the technology detection rules used for this record. | |||
| data.items[].domain | string | No | identity | |||
| data.items[].ecommerce | string | No | Ecommerce platform detected on the site. | |||
| data.items[].failure_reason | string | No | failure diagnostics: populated whenever a probe attempt didn't fully pass, even if the site still counts as reachable (e.g. a blocked page returning a non-5xx status). Unset for a fully passing site. | |||
| data.items[].final_url | string | No | URL that produced the adopted body (after www. fallback / redirects) | |||
| data.items[].has_captcha | boolean | No | Whether a CAPTCHA technology was detected on the site. | |||
| data.items[].is_infrastructure | boolean | No | IsInfrastructure flags a backend CDN/DNS/cloud-vendor/registry hostname (e.g. cloudfront.net, gtld-servers.net) rather than a real, human-navigable website. Best-effort heuristic: a false negative (unflagged) is not proof a domain is a real site. | |||
| data.items[].method_version | string | No | Version of the scan procedure used to collect this record. | |||
| data.items[].probe_error | string | No | raw transport error's first line (e.g. "context deadline exceeded", "no such host"), when there was one | |||
| data.items[].probed_at | string | No | RFC3339 — when last probed | |||
| data.items[].rank | integer | No | Tranco rank when seeded from the top-1M list | |||
| data.items[].reachable | boolean | No | Whether the site answered the probe. | |||
| data.items[].render_tier | string | No | provenance / reproducibility | |||
| data.items[].run_date | string | No | Time the scan run started, in UTC (RFC 3339). | |||
| data.items[].run_id | string | No | Identifier of the scan run that produced this record. | |||
| data.items[].scan_method | string | No | Method used to scan the site, for example `http_scan`. | |||
| data.items[].schema_version | integer | No | Version of the record format. | |||
| data.items[].scheme | string | No | URL scheme the site was reached with, `https` or `http`. | |||
| data.items[].secondary_only_technology_names | array | No | technology names detected ONLY via a fetched secondary page, not the homepage alone; a subset of TechnologyNames | |||
| data.items[].secondary_page_categories | array | No | which categories were successfully fetched and contributed content, sorted (careers|login|blog|status|signup|docs|contact|trust|cart) | |||
| data.items[].secondary_pages | array | No | full audit trail: every discovered candidate, fetched or not; detail view, stored not indexed | |||
| data.items[].secondary_pages[].category | string | No | Kind of page found on the homepage navigation: `careers`, `login`, `blog`, `status`, `signup`, `docs`, `contact`, `trust` or `cart`. | |||
| data.items[].secondary_pages[].fetched | boolean | No | Whether the page was loaded successfully. False when it was blocked or timed out. | |||
| data.items[].secondary_pages[].url | string | No | Address of the page found on the homepage navigation. | |||
| data.items[].secondary_pages_discovered | integer | No | secondary-page telemetry: populated when the crawl looked for extra non-homepage pages (opt-in per run) and discovered at least one on the homepage nav for this domain. "Discovered" and "fetched" are deliberately distinct: a candidate can be found on the homepage nav but still fail to fetch (e.g. blocked or timed out) — that failure is recorded in SecondaryPages (Fetched=false), not silently dropped. | |||
| data.items[].secondary_pages_fetched | integer | No | count that were ACTUALLY fetched successfully and contributed content; <= SecondaryPagesDiscovered | |||
| data.items[].seed_source | string | No | tranco | internal-harvest | |||
| data.items[].server_language | string | No | Server-side language or web framework detected on the site. | |||
| data.items[].source_url | string | No | the exact seed URL probed | |||
| data.items[].status | integer | No | HTTP status code of the last probe attempt. | |||
| data.items[].tech_count | integer | No | Number of distinct technologies detected on the site. | |||
| data.items[].technologies | array | No | detections (the payload) | |||
| data.items[].technologies[].categories | array | No | Categories the technology belongs to, such as `CDN` or `Analytics`. | |||
| data.items[].technologies[].confidence | string | No | How sure the detection is: `high`, `medium` or `low`. | |||
| data.items[].technologies[].evidence | string | No | Signal the technology was recognised from, such as `header: cf-ray`. | |||
| data.items[].technologies[].name | string | No | Technology name, such as `Shopify` or `Cloudflare`. | |||
| data.items[].technologies[].version | string | No | Version of the technology, when the site exposes it. | |||
| data.items[].technology_names | array | No | flattened for AND/NOT term filters + facets | |||
| data.items[].tld | string | No | Top-level domain of the site, such as `com`. | |||
| data.items[].web_server | string | No | Web server software detected on the site. | |||
| data.page | integer | No | Page number returned. | |||
| data.page_size | integer | No | Number of items per page. | |||
| data.sort | string | No | Sort order applied. | |||
| data.total | integer | No | Total number of records matching the filters. | |||
| msg | string | No | OK |
| Status | Description | Schema |
|---|---|---|
| 400 | Bad Request | #/definitions/app.Response |
| 429 | Too Many Requests | #/definitions/app.Response |
| 500 | Internal Server Error | #/definitions/app.Response |
Use environment variables for secrets and keep Crawlora API keys server-side.
curl -X GET "https://api.crawlora.net/api/v1/datasets/techstack/search?q=coffee&render_tier=http&has_captcha=true&is_infrastructure=true&reachable=true&sort=relevance&page=1" \
-H "x-api-key: $CRAWLORA_API_KEY"