Data layer for AI agents
Structured, ready-made web data your AI agents and pipelines query over one REST API — or straight from an MCP client: a 4.3M-app App Store + Google Play catalog, the daily app charts for both stores, 7.9M app reviews with sentiment, a Google Maps local-business dataset, and live web studies like the Anti-Bot and Dead-Web indexes. Pull clean JSON, pay on success — no crawl to run.
4.3M
apps indexed, plus 24 more datasets — all one REST API away.
1.4M
iOS apps
2.9M
Android apps
998K
sites scanned
Normalized JSON · pay on success · 2,000 free credits / month.
| Dataset | What it covers | Records | Updated | REST endpoint |
|---|---|---|---|---|
| App Store + Google Play apps | Every app's category, rating, ratings count, price, install scale and popularity | 4.3M apps | Continuously enriched | datasets/apps/search |
| App Store + Google Play charts | The daily Top Free, Top Paid and Top Grossing rankings on the iOS App Store and Google Play | 90 charts daily | Refreshed daily | datasets/apps-charts/search |
| App Store + Google Play reviews | Recent user reviews of the top apps on the iOS App Store and Google Play | 7.9M reviews | Periodically refreshed | datasets/apps-reviews/search |
| Google Maps businesses | Scraped Google Maps business records held in a search index | Local business listings | Continuously expanded | datasets/google-map-businesses/search |
| Airbnb markets | Aggregate Airbnb short-term-rental market statistics from a near-complete public-listing supply census | 7.9M listings | Snapshot 2026-07-03 | datasets/airbnb-markets/search |
| GitHub developer profiles | Enriched public profiles for individual GitHub developers | 973K profiles | Snapshot 2026-07-07 | datasets/github-users/search |
| X (Twitter) user profiles | Public X (Twitter) profiles | 471K profiles | Snapshot 2026-07-21 | datasets/x-users/search |
| SEC EDGAR companies | Every SEC EDGAR filer as one record | 977K filers | Snapshot 2026-07-21 | datasets/sec-companies/search |
| Jobs — companies & postings | Discovered company career-site boards across 14 ATS providers, plus the live postings crawled from them and normalized into one JSON shape | 20K boards | Snapshot 2026-07-21 | datasets/jobs/search |
| Steam games | The top 10,000 Steam games by SteamSpy owner estimate | 10K top games | Snapshot 2026-07-11 | datasets/steam-games/search |
| Chrome Web Store items | Extensions, themes and legacy apps with displayed users, ratings, versions, declared permissions, privacy disclosures and change history | 966 pilot records | Pilot snapshot 2026-07-14 | datasets/chrome-extensions/search |
| Steam charts | Steam's official store charts as a daily time series | 220 rows daily | Refreshed daily | datasets/steam-charts/search |
| US housing markets | Redfin market statistics for every US metro, county, city and ZIP | 52K regions | Snapshot 2026-05 | datasets/housing-markets/search |
| Website tech stack | The technologies behind every site | 979K sites | Snapshot 2026-07-21 | datasets/techstack/search |
| Journalist contacts | Journalist and reporter records | 44K+ journalists | Continuously expanded | datasets/journalists/search |
| Numbeo cost of living | Cost of living, quality of life, crime & safety, health care, pollution, traffic and property investment | 1,079 cities | Snapshot July 21, 2026 | datasets/numbeo-cities/search |
| Most-Played Steam Games | Steam's official most-played chart | Top 100 live | Live | — |
| Local Business on the Web | How online local businesses really are: 40% of 12.4M US/UK/Canada Google Maps listings have no website, the most common listed “website” is a Facebook page, and coverage runs from 93% (banks) to 33% (farms) | 12.4M listings analyzed | Snapshot 2026-06-10 | — |
| TikTok Creators Index | The TikTok creator pyramid: among 3.26M established creators, only 1.6% have 1M+ followers, 4% are verified, and 4% list a business contact | 3.26M creators analyzed | Snapshot 2026-06-21 | — |
| Airbnb Market Index | Short-term-rental market data across 10 major Airbnb markets: estimated occupancy runs 22%–67%, nightly rates $69–$464, and professional multi-listing operators range 4%–24% | 10 markets analyzed | Snapshot 2026-06-23 | — |
| Anti-Bot Index | Which of the world's top sites run anti-bot / WAF protection, and which vendor | 998K sites scanned | Snapshot June 14, 2026 | — |
| Dead-Web Index | How much of the most popular web is actually dead, blocked, or parked | Top global domains, probed | Probed live | — |
| AI-Crawler Blocking Index | Which AI crawlers each site blocks in robots.txt | Top 1M sites · robots.txt | Snapshot 2026-06-20 | — |
| State of Web Scraping & Anti-Bot 2026 | The flagship annual report | Top 1M–10M · synthesis | June 2026 | — |
| Search vs. Store | Do people download the AI apps they Google? Joins Google Trends search demand, US App Store rank and SimilarWeb web traffic for 40 major AI apps | 40 AI apps · 3 signals | Snapshot 2026-07-01 | — |
The app and Google Maps datasets return JSON from the REST endpoints above; the Anti-Bot and Dead-Web indices are interactive studies. Counts are the latest indexed totals and grow as the crawl runs.
A few cuts from the two most data-rich datasets — hover any chart to isolate a series. The full breakdowns live on each dataset's page.
The app catalog skews Android
Google Play is ~2.1× the App Store — 2.9M Android apps to 1.4M iOS.
2,939,228 apps
1,409,323 apps
Most apps are invisible
61% of Android apps have under 1,000 installs; only 171 have crossed a billion.
| Install bucket | Apps |
|---|---|
| < 1K | 1,792,550 |
| 1K–10K | 554,968 |
| 10K–100K | 328,183 |
| 100K–1M | 147,183 |
| 1M–10M | 52,684 |
| 10M–100M | 12,223 |
| 100M–1B | 1,280 |
| 1B+ | 171 |
Most of the web doesn’t fight back
Of 998,497 top sites scanned for the Anti-Bot Index, 437,857 run a managed wall — but only 79,835 (8%) actively challenged a bot.
| Scanned → Reachable | 818,614 (36.3%) |
| Reachable → Has a wall | 437,857 (19.4%) |
| Reachable → No wall | 380,757 (16.9%) |
| Has a wall → Passively present | 358,022 (15.9%) |
| Scanned → Unreachable | 179,883 (8%) |
| Has a wall → Actively challenged | 79,835 (3.5%) |
Stored, structured, and deduplicated — query the index directly without a live crawl. Filter, facet, sort, and page through normalized records over the REST API.
App intelligence
4.3M apps
1.4M iOS · 2.9M Android
Every app's category, rating, ratings count, price, install scale and popularity — deduplicated to one record per app per store, across the iOS App Store and Google Play.
Key fields
titledevelopercategoryscoreratings_countprice / freeinstalls (Android)popularitycountries_available (iOS)App rankings
90 charts daily
15 countries · Top Free / Paid / Grossing
The daily Top Free, Top Paid and Top Grossing rankings on the iOS App Store and Google Play — one record per chart, snapshot and rank, so it's a time series you can diff for movers, not just today's leaderboard.
Key fields
storechart_typecountrysnapshot_daterankapp_idtitledeveloperReview sentiment
7.9M reviews
recent reviews · sentiment · helpfulness
Recent user reviews of the top apps on the iOS App Store and Google Play — score, title, text, helpful count and date, one record per review. Recent-review sentiment runs harsher than lifetime ratings, which is exactly the signal.
Key fields
storeapp_idcountryscoretitletextthumbs_upreviewed_atLocal business
Local business listings
search · facets · nearby · detail
Scraped Google Maps business records held in a search index — query by keyword and location, facet by category, find places near a point, or fetch one business by place_id. Reads the stored index, so there's no live scraping or proxy routing.
Key fields
namecategoryaddressratingreview_countcoordinatesplace_idTravel & hospitality
7.9M listings
60+ countries · aggregate-only
Aggregate Airbnb short-term-rental market statistics from a near-complete public-listing supply census — listing supply, ratings and nightly-price bands rolled up by country, metro and geo cell. Aggregate-only: no individual listings or hosts, thin cells suppressed.
Key fields
countrymarketlistingsavg_ratingmedian nightly pricesuperhost sharegeo densityDeveloper intelligence
973K profiles
search · facets · nearby · item
Enriched public profiles for individual GitHub developers — followers, company, interest domains, recent activity and clean reverse-geocoded location, one record per login. Query the stored index directly, no live crawl.
Key fields
loginnamecompanyfollowerspublic_reposinfluence_tierrank_scoregeo.countrydomainsactive_90dSocial intelligence
471K profiles
search · facets · item
Public X (Twitter) profiles — followers, following, bio, external link, post count, blue check and account age, one record per username. A seeded index of established accounts, read logged-out from public pages; query the stored index directly, no live crawl.
Key fields
usernamenamebiofollowersfollowingfollower_following_ratiopostsis_blue_verifiedcreated_atexternal_urlCompany intelligence
977K filers
search · item · facets · financials · insider
Every SEC EDGAR filer as one record — profile, tickers, industry, filing history, XBRL financial-statement history and insider-transaction rollups — plus institutional 13F holdings by manager. Built from SEC public data, queryable directly, no live crawl.
Key fields
ciknametickersexchangesic_descriptionstate_of_incorporationentity_typeforms_filedlatest_annual_revenuerevenue_bandinsider_txn_count_90dHiring intelligence
20K boards
search · companies · facets · nearby · item
Discovered company career-site boards across 14 ATS providers, plus the live postings crawled from them and normalized into one JSON shape — company, department, location, employment type, remote flag. Query the stored index directly, no live crawl.
Key fields
providercompanytitledepartmentlocationemployment_typeremoteis_openposted_atstatusPC gaming intelligence
10K top games
search · facets · item
The top 10,000 Steam games by SteamSpy owner estimate — price, review tier, owner band, genres, platforms and release year, one enriched record per app. Query the stored index directly, no live crawl.
Key fields
app_idnameprice_centsis_freereview_tierowners_midpointgenresrelease_yearplatform_linuxmetacriticExtension intelligence
966 pilot records
search · metrics · facets · history
Extensions, themes and legacy apps with displayed users, ratings, versions, declared permissions, privacy disclosures and change history — plus chart-ready aggregate metrics. The current public study is a quality-checked 1,000-ID pilot before the full-catalog backfill.
Key fields
idnameitem_typeusersratingmanifest_versionpermissionshas_broad_host_accesscollects_datastatusPC gaming rankings
220 rows daily
concurrent · most played · US top sellers
Steam's official store charts as a daily time series — live concurrent players and most-played by 24h peak (global), plus the US top sellers with price. One record per chart, day and rank, so you can diff for movers, not just read today's leaderboard.
Key fields
chartcountrysnapshot_daterankapp_idnameconcurrent_in_gamepeak_in_gamefinal_price_centsReal-estate intelligence
52K regions
search · facets · item · monthly since 2012
Redfin market statistics for every US metro, county, city and ZIP — median sale and list price, inventory, days on market and competitiveness, monthly back to 2012 — joined to Census income for price-to-income, salary-to-buy and the affordability gap. Aggregate market data only; reads a stored index, no live scraping.
Key fields
region_typeregionstate_codeperiod_beginmedian_sale_pricehomes_soldmedian_household_incomeprice_to_incomesalary_to_buyTechnology intelligence
979K sites
search · facets · item
The technologies behind every site — CMS, ecommerce platform, CDN, analytics, web server and ~200 others — fingerprinted from HTML + response headers across the Tranco top-1M, one record per domain. Query the stored index directly, no live crawl.
Key fields
domaintechnology_namescategoriescmsecommercecdnweb_serverserver_languagehas_captchareachablePR & media intelligence
44K+ journalists
227 outlets · 7.2% have an email
Journalist and reporter records — outlet, title, beat topics and whatever public contact info (work email or social handle) that outlet lists on its own staff page — crawled directly from news outlets' own public pages, one record per journalist.
Key fields
outlet_idoutletverticalnametitletopicscontact_emailsocial_linkscontact_typeCost-of-living intelligence
1,079 cities
125 countries · seven Numbeo indices
Cost of living, quality of life, crime & safety, health care, pollution, traffic and property investment — all seven Numbeo indices merged into one record per city and per country, on the New York = 100 comparable scale. Aggregate index data only; reads a stored index, no live scraping.
Key fields
citycountrycost_of_living_indexquality_of_life_indexsafety_indexcrime_indexhealth_care_indexpollution_indextraffic_indexcoverageTwo living indices that turn large-scale crawls into browsable, citable studies — with per-entity pages, charts, and machine-readable tables.
Web data study
Top 100 live
Steam's official most-played chart — the top 100 games ranked by live concurrent players right now, with 24-hour peak. Reads Steam's charts through Crawlora's /steam/charts endpoint.
See the live Most-Played chartWeb data study
12.4M listings analyzed
How online local businesses really are: 40% of 12.4M US/UK/Canada Google Maps listings have no website, the most common listed “website” is a Facebook page, and coverage runs from 93% (banks) to 33% (farms).
Explore the Local Business IndexWeb data study
3.26M creators analyzed
The TikTok creator pyramid: among 3.26M established creators, only 1.6% have 1M+ followers, 4% are verified, and 4% list a business contact — rising to 23% of millionaires.
Explore the TikTok Creators IndexWeb data study
10 markets analyzed
Short-term-rental market data across 10 major Airbnb markets: estimated occupancy runs 22%–67%, nightly rates $69–$464, and professional multi-listing operators range 4%–24% — from a representative map-grid sample.
Explore the Airbnb Market IndexWeb data study
998K sites scanned
Which of the world's top sites run anti-bot / WAF protection, and which vendor — from a full top-1,000,000 scan, broken down by site category and Tranco rank band.
Explore the Anti-Bot IndexWeb data study
Top global domains, probed
How much of the most popular web is actually dead, blocked, or parked — live reachability and HTTP status across the world's top domains. Search any domain for its verdict.
Explore the Dead-Web IndexWeb data study
Top 1M sites · robots.txt
Which AI crawlers each site blocks in robots.txt — a full top-1,000,000 scan, broken down by crawler, website category, TLD and Tranco rank. Same domain universe as the Anti-Bot Index.
Explore the AI-Crawler IndexWeb data study
Top 1M–10M · synthesis
The flagship annual report — how the open web defends itself, synthesizing the Anti-Bot Adoption, AI-Crawler Blocking and Dead-Web indexes into one citable read. Measures the web's defensive posture, not bot traffic.
Read the 2026 reportWeb data study
40 AI apps · 3 signals
Do people download the AI apps they Google? Joins Google Trends search demand, US App Store rank and SimilarWeb web traffic for 40 major AI apps — search and store barely align; only ChatGPT tops both. CSV + JSON, CC BY 4.0.
Explore Search vs. StoreThe same datasets, framed for a specific job — the buyer, the workflow, and the API in one place.
Local lead generation
Turn 12.4M Google Maps businesses into a lead list with verified emails — discovery plus contact enrichment, billed on success.
ExploreApp intelligence
Monitor 4.1M apps across the App Store and Google Play — rankings, ratings, reviews and availability — through one API.
ExploreCreator intelligence
Discover 3.3M TikTok creators by niche, follower tier, engagement and verification — aggregate-only, GDPR-aware.
ExploreEvery dataset is a documented REST endpoint. Authenticate with an x-api-key header, filter and sort with query params, and get normalized JSON — no HTML parsing, no proxies to manage. Queries read the stored index, and you’re billed pay on success: charged for results, not failed requests.
Search the app catalog
curl "https://api.crawlora.net/api/v1/datasets/apps/search?store=android&sort=popularity_desc&size=20" \
-H "x-api-key: $CRAWLORA_API_KEY"Swap in datasets/google-map-businesses/search for local businesses.
Crawlora runs 300+ live endpoints across search, maps, marketplaces, social, finance and more — most public web data can become a structured dataset. Browse what’s live, or tell us what you need.
Crawlora maintains a family of web-data datasets: an App Store + Google Play app catalog (4,348,551 apps — 1,409,323 iOS and 2,939,228 Android), the daily App Store + Google Play top-chart rankings (90 charts across 15 countries), 7,915,917 app reviews with sentiment, a Google Maps local-business dataset, a near-complete Airbnb markets dataset (7.9M listings across 60+ countries), and live web studies such as the Anti-Bot Index (998,497 sites scanned) and the Dead-Web Index. The apps, charts, reviews, Maps and Airbnb datasets are queryable as JSON over the REST API; the indices are interactive studies you can browse and cite.
Query them through Crawlora's REST API with an x-api-key header. `datasets/apps/search` searches the app catalog (filter by store, category, rating, price, sort by popularity) and `datasets/google-map-businesses/search` searches stored Google Maps businesses with keyword, location, facets and nearby queries. Both return normalized JSON and bill pay-on-success. See the Datasets API docs to start.
The app catalog is continuously enriched (latest snapshot July 21, 2026) and the Google Maps dataset is continuously expanded. The Anti-Bot Index is a periodic full top-1M scan (latest June 14, 2026); the Dead-Web Index probes domains live on request.
Crawlora's catalog has indexed 4,348,551 apps across both stores — 1,409,323 on the iOS App Store and 2,939,228 on Google Play — each deduplicated to one record per store with category, rating, price, install and popularity fields.
Dataset queries are billed pay-on-success on credit-based plans (you're charged for results, not failed requests), with 2,000 free credits a month and no card to start. See pricing for plan limits.
Query 4.3M apps, Google Maps businesses and the live web indices through one REST API — clean JSON, no parsing, pay on success.