How To Scrape 10 Million Websites in 10 Minutes

Ten million websites in ten minutes works out to about 16,700 requests every second, held for the full ten minutes. No single machine does that, and no cleverer individual request gets you there. The only thing that does is a fleet of cheap, identical workers pulling from one shared queue — where total throughput rises with the number of workers, and any worker can disappear without losing work.

This post builds that system end to end: the decoupled architecture, the language choice that makes the concurrency cheap, the handful of per-worker settings that decide whether adding a box actually adds throughput, and the failure-handling that keeps it running while it does. The worker is a small open-source binary you can run on your own laptop, so every step is reproducible rather than hand-waved.

It's a throughput problem, not a request problem

Start from the arithmetic, because it sets everything else: 10,000,000 URLs ÷ 600 seconds ≈ 16,667 requests/second. A single well-tuned box holding ~500 concurrent, I/O-bound probes — each taking a second or two — lands somewhere around a few thousand requests a second. So the target is roughly four to ten boxes, if throughput scales cleanly as you add them. The entire job of the design below is to earn that "if": to make the fleet scale close to linearly, and to keep it alive while it runs.

The architecture: a decoupled pipeline

Everything hangs off one idea — decouple the stages so they never block each other, and keep all the durable state in one place. Workers become disposable; the queue becomes the system of record.

• Seed loader. Streams the URL list into the queue in chunks — never load 10M rows into memory. Make it idempotent (SADD into a Redis set, or push to a list) so a re-run resumes instead of double-enqueuing.
• Work queue (Redis). The heart of it. It decouples producers from consumers, absorbs bursts (backpressure), and makes the whole system at-least-once: a worker pops a job, and if it dies mid-flight the job is simply re-popped by another worker. Add a SET or Bloom filter for visited-URL dedup, and a dead-letter queue for jobs that exhaust their retries.
• Worker fleet. Stateless pods running identical code. Each one buffers a small batch (~100–200 jobs), processes them with internal concurrency, bulk-writes results, and syncs its stats periodically and on shutdown. Scaling is just "add pods." A crash is just "the others keep going."
• Results sink. Any store that takes batched writes (Elasticsearch, Postgres COPY, Parquet files) keyed on an idempotent id like domain|run_id, so a re-processed job overwrites in place instead of duplicating.

Notice there's no central "coordinator" doing real-time scheduling. The queue is the coordination. That's what lets the fleet scale horizontally without a brittle brain in the middle.

Pick the language for the bottleneck

The axis that decides this isn't raw single-request speed. Fetching is I/O-bound — you spend the whole time waiting on the network — so what matters is how cheaply a runtime holds thousands of requests in flight at once, how well it uses every core, and how small it ships. People reach for "which uses less memory?" first, so we'll start there — and the answer isn't the one the memes predict.

• Go is the path of least resistance here — though not for the reason usually given. Goroutines are green threads (~2–8 KB of stack each, multiplexed onto OS threads with a built-in network poller), so you write plain synchronous-looking code and the scheduler parks them on I/O without tying up a thread; 100k+ in-flight requests per box is unremarkable. Its real wins are true multi-core for free (no GIL, no process fleet) and deployment — a single static binary in a ~10–20 MB scratch image with millisecond cold start, which makes "add a pod" cheap. What is not an automatic win is memory; see the measurement below.
• Python is more competitive than its reputation. asyncio + aiohttp give real I/O concurrency and, as it turns out, a genuinely lean memory footprint. Its costs sit elsewhere: it's one event loop per core (cooperative — one stray blocking call stalls the whole loop), so using all your cores means running a fleet of processes, and the image is interpreter-plus-dependencies (hundreds of MB) with a slow cold start. On the GIL: the no-GIL free-threaded build became officially supported (still optional) in Python 3.14, but it mainly helps CPU-bound multithreading — for I/O fan-out the GIL was never the limiter, so it changes little for a fetcher.

So which actually uses less memory? We measured it

The folk wisdom is "goroutines are tiny, so Go sips RAM." That's true for a bare goroutine and false for a connection. We held 10,000 concurrent HTTP requests open against a local server and recorded peak resident memory (plain HTTP, no TLS — a clean footprint test, not a full scrape):

Out of the box, Go used ~1.6× the memory of Python, not less. Each Go connection carries a grown goroutine stack plus net/http's read and write buffers, and the garbage collector keeps roughly twice the live heap resident by default — that last part is the "resident memory doubles" you'll meet again in the tuning section. The fix is the same lever: cap the GC (GOMEMLIMIT, or a lower GOGC) and Go drops to parity with Python.

So memory is not the reason to pick Go here — untuned, it's a reason against. Pick Go for the CPU and multi-core efficiency, the tiny static image, and the low base footprint (8 MB idle here, versus Python's 28 MB); just don't ship it with the GC left wide open.

The honest escape hatch: if the expensive stage is extraction — pandas, BeautifulSoup, an LLM call — Python's ecosystem can dominate, and the right move is to split the pipeline (Go fetchers feeding Python extractors) rather than force one language to do both. Match the language to whichever stage is actually the bottleneck.

And here's the part that's genuinely changed. This decision used to be settled by "write what the team already knows," because ramping up on Go for one high-throughput component cost weeks — so a Python shop stayed Python even where a leaner runtime fit the job. LLM-assisted coding has largely dissolved that ramp. A model will scaffold the goroutine pool, the tuned http.Transport, the channel-and-semaphore backpressure, and the Dockerfile idiomatically, even for a team that's never shipped Go. Right-tool-for-the-job is now a realistic default instead of a luxury.

The worker: one cheap, honest probe

The unit of work is deliberately small. Here's the baseline, running crawlora-deadweb — the open Go probe we'll use as the worker — on a single machine before we distribute it:

cat domains.txt | crawlora-deadweb --json --concurrency 50 > out.ndjson

What's inside one good worker is just two things done carefully:

• High concurrency, because the work is I/O-bound. A bounded pool of goroutines (or asyncio tasks) keeps hundreds of requests in flight per box. The CPU is mostly idle; the network is the job.
• An honest, cheap verdict per URL. DNS → TCP connect → one GET /, then label the result and move on. Crucially, don't burn the budget retrying hosts that are genuinely gone: no DNS or a refused connection means dead — drop it; a 403/429 means alive but refusing this client — a completely different problem that a retry won't fix either. Telling those apart is most of what keeps a large run from wasting hours on corpses.

The five settings that make a worker fast

A fleet only scales linearly if each box is tuned; otherwise you add boxes and they all pile into the same shared bottleneck. Five knobs carry almost all of it:

If a large crawl is mysteriously slow, the answer is DNS far more often than people expect. Cache aggressively, spread lookups across multiple resolvers, and run a local caching resolver on each node.

Scale out: add boxes, watch the bottleneck

With a tuned worker, scaling is mostly a replica count. In a real run, bumping the fetch fleet from 4 to 10 pods drained the queue in hours instead of a day. The shape you're aiming for:

Tuned fleet — throughput tracks workers1.6k → 16kUntuned — a shared bottleneck caps it1.5k → 5.1k

1

1.6k

1.5k

2

3.2k

2.8k

4

6.4k

4.2k

6

9.6k

4.8k

8

12.8k

5k

10

16k

5.1k

On Kubernetes the hygiene is small but load-bearing: set imagePullPolicy: Always so a re-pushed :latest actually rolls, give pods real requests/limits, and optionally drive an HPA off queue depth rather than CPU. And don't over-shard: throughput is bounded by the slowest shared resource, so if the curve is bending flat, adding boxes scales the bottleneck, not the work — fix DNS or the sink first.

Keep it up: failure is the normal case

At ten million requests, rare events happen constantly — a worker will be OOMKilled, a deploy will roll mid-run, a host will start timing out. High availability isn't a feature you bolt on; it's the set of defaults that make each of those a non-event:

The throughline: stateless workers plus a durable queue means there's no single point of failure. The queue is the only thing that has to survive, and it's the one thing you replicate.

The part that actually breaks: resources

In practice a system at this scale rarely fails on logic. It fails on memory — a restart loop of OOMKills (exit 137). The cause is almost always the same: the runtime isn't cgroup-aware, so inside a container it sees the node's cores and RAM instead of the pod's limit. For Go before 1.25, GOMAXPROCS defaults to the node's core count (over-parallelizing GC and TLS handshakes against a fractional CPU cap), and there's no memory ceiling, so the heap can nearly double past the limit before GC runs.

The fixes, in order of how much they bought us:

The lesson generalizes beyond Go: whatever the runtime, size its parallelism and memory to the container's quota, not the host's. (For Python: size the process/worker pool to the CPU quota, not os.cpu_count(), and watch RSS per worker.)

Know when you're actually done

Two counting traps bite everyone at the finish line:

And don't scale the fleet to zero the instant the queue is empty. Workers buffer ahead, so killing them at queue == 0 SIGTERMs them mid-buffer and drops in-flight work — an under-count. Wait until processed ≈ target (confirmed against the sink), and only then scale down.

Be a good citizen

None of this is permission to hammer the web. Keep per-host rate caps, respect robots.txt and terms of service, send unauthenticated GETs only, and identify your bot. A list of ten million distinct hosts spreads load naturally; ten million pages on one site needs real per-host throttling. And don't run a wide fan-out from a single bare cloud IP — connecting to many distinct hosts from one unmanaged address looks like a network scan and earns abuse reports. Use controlled egress or a managed pool.

The worker is open source

crawlora-deadweb is the small Go probe used as the worker above — DNS, TCP, one honest GET, then a verdict. Run it on a list of your own, or read its classify package to see how the unit of work is built. The same escalate-only-as-far-as-forced pattern is what Crawlora runs as a hosted API.

Get the worker on GitHubSee the hosted version