How We Scraped 132.2 Million Google Business Listings

Crawlora's Google Maps dataset holds 132.2 million business listings across 25 countries as of its August 25, 2026 snapshot — the same number behind our census of the global web-presence gap. Getting there was never about writing a better scraper for one search box. Google Maps hard-caps every search at roughly 100 to 200 results, so a single query — however well-disguised — never gets past a few hundred rows, no matter how it's phrased or how many times it's re-run. Scraping "a city's plumbers" and scraping "every business Google Maps knows about" are different engineering problems, and the gap between them is almost entirely architecture: how many tasks you seed before a single result exists, how a durable queue survives crashes and reruns, and what happens when a shared resource silently starves one corner of the crawl for weeks. This is that architecture, including the parts that broke.

The wall: why one search never gets past a few hundred results

Search "restaurants in Austin" or "plumbers in Ohio" on Google Maps and the results feed caps out at roughly 100 to 200 places, no matter how the query is worded or how many times it's re-run. Google ties results to the current map viewport rather than to true offset-based pagination, so there's no "page 6" to request. Our own beginner's guide to scraping Google Maps mentions this cap almost as an aside, in an FAQ about splitting a search into tiles. At real scale, that aside is the entire design problem: everything below exists because of one constant, unmovable ceiling on a single query.

Seed the world, not a search box

The instinct is to wait for a query and go fetch it. At scale, that's backwards — the crawl seeds root tasks for every place a business could plausibly be, upfront, before it knows whether anything is actually there. The seed space is a gazetteer of cities and towns crossed with a taxonomy of business-type keywords ("coffee shop," "auto repair," "dentist," and hundreds more).

For the United States alone, 7,557 city and town seeds crossed with an 834-term business-type taxonomy produced 6,302,538 confirmed root search tasks — before a single result came back. That's the part that surprises people: the task count explodes at the seeding stage, long before there's any data to show for it. The crawl is designed for the coverage it eventually needs, not for whatever a handful of manual searches happen to turn up.

When a cell is still full, cut it smaller

A flat grid of city-by-keyword searches still hits the same ~120-result wall in any dense area — central Manhattan or a "restaurant" search in a big metro will saturate a root task instantly. The fix is recursive: a root task that comes back full spawns child tasks that split its search into smaller geographic tiles, and a tile that's still full splits again into narrower sub-searches. A search that returns 40 results stops there; one that returns 120 keeps subdividing until it doesn't.

The effect is that dense areas automatically accumulate many fine-grained child tasks while sparse ones — a rural county, a niche category — never subdivide past their first search. Nobody hand-draws the grid or decides in advance how deep any one area needs to go; the cap itself is what triggers the next level of subdivision.

Skip the browser

Most scraping guides — including our own tutorial-level one — reach for a headless browser by default, because Google Maps looks like a heavy JavaScript app. At real scale that default is the wrong one. The search results and place details shown on the page are both fetched by the page's own client-side calls to Google's internal data endpoints — plain HTTP requests that happen to carry a browser's TLS and header fingerprint. Reproduce that fingerprint with an HTTP client and call the same endpoint directly, and there's no page left to render at all.

A single-process benchmark against that endpoint, scanning concurrency from 1 to 16 in-flight requests, showed why this matters:

Queries per second0.86 q/s → 11.36 q/s

1

0.86 q/s

2

1.83 q/s

4

3.52 q/s

8

6.07 q/s

16

11.36 q/s

That's a near-linear climb with a flat latency curve and zero failures — the signature of a bottleneck that isn't there. Rendering, CAPTCHAs, and layout parsing simply don't enter into this path. Everything that actually limits production throughput lives downstream of the fetch: how fast the fleet can write results and claim the next task. That shows up three separate times later in this post.

A durable queue, not a script

A crawl that runs for weeks across tens of millions of tasks can't be a script that dies with the process. The task store is a persistent state machine — every task is pending, leased, done, retry, or dead — decoupled from a small, capped "hot" dispatch buffer that's continuously refilled from that durable store. A worker that crashes mid-task just leaves it leased until a timeout expires and another worker reclaims it; a paused fleet resumes mid-backlog without replaying anything already done; and the backlog can sit at tens of millions of open tasks without ever needing to fit in memory.

This part of the design isn't specific to Google Maps — we've written about the generic version of it before: a durable queue plus disposable workers, applied to a flat list of URLs. Everything else in this post is what's different when the "list" isn't flat — it's a search space that only reveals its true size as you crawl it.

Three lanes so nobody starves

The crawl runs three kinds of work — discovery (the searches above), place-detail hydration, and reverse geocoding — and for a while they all pulled from the same queue. That was a mistake, and it produced the first real incident: a one-time backlog of detail-page re-fetches ran through the same pool as fresh discovery and consumed the large majority of fleet capacity, nearly stalling new discovery entirely.

The fix was giving each kind of work its own independent queue and its own worker pool, so one kind can never crowd out another. After the split — same fleet size, no new hardware — discovery throughput went from roughly 9 to 94 tasks per second, overnight. The lesson wasn't "add capacity." It was "stop making three different jobs compete for one line."

Resolve location for free

Every place needs its administrative geography resolved — state, county, city — from raw coordinates. Hitting a public geocoding service for each of 130+ million places would hit that service's rate limits long before the crawl got anywhere close to its current size. Instead, the crawl resolves geography offline, against a baked place-name gazetteer grid held in memory, and only falls back to a real geocoding service — cached, rate-limited — for the sliver of coordinates the offline grid can't resolve on its own. This is one of the few places in the whole pipeline where the cheap option and the correct option at scale are the same option.

Identity and freshness, honestly

The same business routinely gets discovered more than once: a city-level search, a cuisine keyword, a tile subdivision, an overflow page, and a neighboring city's radius overlap can all surface the same restaurant independently. All five need to converge on one record, not five, so the place's own stable identifier — not the search that found it — is the dedup key. Before rewriting a record, the crawl hashes its stable fields and skips the write entirely if nothing has changed since the last pass within a rolling window, which cuts write volume without losing freshness where it actually matters.

Freshness itself deserves an honest number instead of a marketing one. A crawl this size is never uniformly fresh — new regions and categories keep getting added while others wait their turn to be re-crawled. At one mid-2026 checkpoint, just under half of the served dataset had been touched that year; the rest was still standing on a bulk load from two years earlier. That's not a flaw being papered over here — it's the normal state of a living crawl, and any dataset this size that claims otherwise probably hasn't actually looked.

What actually breaks at 100 million rows

Three incidents, in the order they were found:

A concurrency setting that did nothing. A worker fell behind and the obvious fix was raising its internal concurrency. It changed nothing — CPU utilization sat in the single digits while the backlog kept growing. The real constraint was a single serialized claim step per worker: one process can only pull one task off the durable store at a time, no matter how big its internal pool is. The fix was more independent workers, each running its own claim loop, not a bigger pool inside one worker. Watching CPU sit idle while a backlog grows is the tell that concurrency isn't the lever.

A rate-limit ceiling misdiagnosed as blocking. A recurring throughput ceiling looked like anti-bot pushback for weeks — the obvious response is more IPs, more backoff, more caution. Forty-eight hours of metrics proved it was something else entirely: a fixed connection-limit cap on an internal routing layer, saturating at the same hard number regardless of load. Raising that cap fixed the ceiling immediately. No amount of IP rotation would have touched it, because it was never Google saying no.

A fairness bug that made a whole country go dark. One country's crawl looked completely dead for weeks — near-zero new records some days — while its open-task count sat in the hundreds of thousands, which ruled out "nothing left to do." It wasn't blocked, either. A leftover backlog of over 14 million untouched tasks from a much larger, earlier run had clogged a shared first-come-first-served claim queue, and the small country's tasks could never surface ahead of it. The fix was giving each region its own claim lane instead of one shared queue for everything — a fairness problem wearing a blocking problem's clothes.

Prioritize by gap, not by size

Left alone, a crawl like this greedily re-polishes whatever's already dense — big US metros get re-crawled again and again while smaller markets wait indefinitely. The fix is scoring candidate regions by coverage deficit — how far a cell's known place count sits below what's expected for its size and category mix — rather than by raw population or search volume, and dispatching worker time against that deficit. It's a small scheduling change with an outsized effect: it's the difference between a crawl that keeps getting deeper in the same five cities and one that actually gets wider.

The honest numbers today

Metric	Figure	What it shows
Public dataset size	132.2M listings	August 25, 2026 snapshot, 25 countries
Google's per-search result cap	~100–200 places	Why a single query can't scale past a few hundred rows
One country's root-task seed count	6,302,538	U.S. seed wave — before any results returned
Single-process fetch benchmark	11.36 q/s, 100% success	Proof the browser was never the bottleneck
Discovery throughput, before vs. after fleet split	9 → 94 tasks/sec	Same hardware — the fix was queue separation, not scale
Fairness-bug clog size	14M+ leftover tasks	From one earlier run, starving a smaller country's queue

Do this yourself, at a sane scale

Everything above exists to solve a problem of hundreds of millions of rows. Most people don't have that problem — they need a few thousand businesses in a category and a metro area, not a standing crawl of the entire planet. For that, none of the seeding, tiling, queueing, or fleet-splitting above is necessary: a Google Maps search API already does the tiling and dedup behind one endpoint. Our tutorial-level guide covers the five-minute version of the same idea — search by query and location, then enrich a specific result with a place-details call — minus the fleet.

Query the same Google Maps dataset

132.2M+ business listings across 25 countries, filterable by category, location, rating, and website presence over one REST API — no crawl to build or maintain.

Google Maps API docsSee the dataset findingsPricing