Web Scraping with Crawlee in 2026

Most scrapers start as a single script and end as an unmaintained pile of retry loops, ad hoc concurrency limits, and a requests call that silently returns empty HTML the day a target ships client-side rendering. Crawlee is a crawling framework that owns the parts every scraper eventually needs — a request queue, auto-scaling concurrency, session and proxy rotation, storage, and (since its Python v1.0 launch) a crawler that decides per page whether a plain HTTP fetch is enough or a real browser is required. This guide installs it, runs both crawler types, and is honest about the one layer it still doesn't touch.

What a crawling framework buys you over a script

A hand-rolled scraper reinvents the same handful of problems on every project: how many requests run at once before the target starts dropping connections, what happens when one request 429s, where scraped records get written, and how a crawl resumes after it dies halfway through 40,000 URLs. Crawlee's Python v1.0 launch packages all of that behind one interface — a RequestQueue that survives a crash and resumes where it left off, auto-scaling concurrency that backs off under load instead of a fixed worker count you have to guess correctly, and a Dataset/KeyValueStore abstraction that works the same whether you're storing to disk, memory, or Postgres. The team also shipped a SitemapRequestLoader (load a crawl's starting URLs straight from a sitemap instead of hand-parsing XML) and built-in robots.txt compliance via a respect_robots_txt_file flag, so enqueue_links() skips disallowed paths automatically instead of you remembering to check.

None of this is unique in isolation — Scrapy has covered queue management and concurrency for over a decade (see our Scrapy tutorial for that framework's take on the same problem). What's new in Crawlee is that the same request-handling API works across a static-HTML parser and a full Playwright browser, which is the part that actually saves rewrite time when a target's rendering requirements change under you.

Install it

python -m pip install 'crawlee[all]'

The [all] extra pulls in every crawler backend (BeautifulSoup, Parsel, Playwright, curl-impersonate) so you aren't chasing a missing-extra ImportError mid-project; scope it down to crawlee[beautifulsoup,playwright] once you know which crawlers you're actually shipping. If you'd rather scaffold a full project (config, requirements.txt, an entrypoint) instead of a bare script, the CLI does that too:

uvx 'crawlee[cli]' create my-crawler

A Playwright-backed crawl also needs the browser binaries themselves, the same install step web scraping with Playwright covers: playwright install.

Run a static-page crawler

BeautifulSoupCrawler fetches HTML over plain HTTP and hands you a parsed soup object — no browser, fastest option, right for any page whose content is present in the initial server response:

import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

async def main() -> None:
    crawler = BeautifulSoupCrawler(max_requests_per_crawl=50)

    @crawler.router.default_handler
    async def request_handler(context: BeautifulSoupCrawlingContext) -> None:
        context.log.info(f"Processing {context.request.url} ...")
        data = {
            "url": context.request.url,
            "title": context.soup.title.string if context.soup.title else None,
        }
        await context.push_data(data)
        await context.enqueue_links()

    await crawler.run(["https://crawlee.dev"])

if __name__ == "__main__":
    asyncio.run(main())

@crawler.router.default_handler registers the function that runs for every request without a more specific route; context.enqueue_links() reads every <a href> on the page and adds matching ones back onto the same queue, so a five-line handler already crawls an entire site's link graph. push_data() appends the record to the crawler's default dataset — no separate database setup for a first pass.

Switch to a real browser without rewriting the handler

PlaywrightCrawler uses the identical router/request_handler shape — the only real difference is what the context gives you access to:

import asyncio
from crawlee.crawlers import PlaywrightCrawler, PlaywrightCrawlingContext

async def main() -> None:
    crawler = PlaywrightCrawler(max_requests_per_crawl=50)

    @crawler.router.default_handler
    async def request_handler(context: PlaywrightCrawlingContext) -> None:
        context.log.info(f"Processing {context.request.url} ...")
        data = {
            "url": context.request.url,
            "title": await context.page.title(),
        }
        await context.push_data(data)
        await context.enqueue_links()

    await crawler.run(["https://crawlee.dev"])

if __name__ == "__main__":
    asyncio.run(main())

context.soup becomes context.page — a full Playwright Page object, so anything you already know from web scraping with Playwright (waiting for a selector, clicking, evaluating JS) works unchanged. Everything else — the queue, the retries, enqueue_links(), the storage layer — is the exact same code path underneath, which is the actual point: migrating a crawl from static to JS-rendered isn't a rewrite, it's a class swap plus whatever page interaction the new target needs.

Let AdaptivePlaywrightCrawler decide for you

Choosing HTTP vs. browser per-site by hand is fine for one target and a chore across dozens with inconsistent rendering. AdaptivePlaywrightCrawler, new in the v1.0 release, runs the cheap HTTP path by default and only pays for a browser when it has to:

import asyncio
from crawlee.crawlers import AdaptivePlaywrightCrawler, AdaptivePlaywrightCrawlingContext

async def main() -> None:
    crawler = AdaptivePlaywrightCrawler.with_beautifulsoup_static_parser(
        max_requests_per_crawl=50,
    )

    @crawler.router.default_handler
    async def request_handler(context: AdaptivePlaywrightCrawlingContext) -> None:
        context.log.info(f"Processing {context.request.url} ...")
        data = {"url": context.request.url, "title": context.soup.title.string}
        await context.push_data(data)
        await context.enqueue_links()

    await crawler.run(["https://crawlee.dev"])

if __name__ == "__main__":
    asyncio.run(main())

This is the feature that actually matters if you're maintaining more than a handful of scrapers: you stop hard-coding "this domain needs Playwright, that one doesn't" and let the crawler re-verify its own assumption instead of silently going stale when a target changes its rendering.

What's actually new under the hood: ImpitHttpClient

Crawlee's HTTP crawlers (BeautifulSoupCrawler, ParselCrawler, and the HTTP path inside AdaptivePlaywrightCrawler) default to ImpitHttpClient as of v1.0 — a Rust-backed client with browser impersonation built in, not an opt-in extra. That's the same TLS/JA3-fingerprint problem web scraping with curl_cffi covers in detail: a plain Python HTTP stack negotiates TLS differently than a real browser and that mismatch alone triggers some anti-bot checks before a single header is inspected. Crawlee shipping impersonation as the default client means its static-page crawlers don't hit that specific wall out of the box, without you adding curl_cffi as a separate dependency.

Export the dataset

Every push_data() call lands in the crawler's default Dataset. Pull the whole thing out to a file once the crawl finishes:

await crawler.export_data(path="results.json", ensure_ascii=False)
# or, for CSV:
import csv
await crawler.export_data(path="results.csv", delimiter=",", quoting=csv.QUOTE_ALL)

export_data() accepts the same keyword arguments json.dump and csv.DictWriter do, so column quoting, delimiters, and Unicode handling are configurable without a second pass over the data.

What Crawlee doesn't fix

Crawlee is orchestration, not detection evasion. ImpitHttpClient's impersonation raises the floor on the HTTP crawlers' TLS fingerprint, and PlaywrightCrawler renders real JavaScript — but neither of those defeats a hard anti-bot deployment's other layers on its own:

• IP reputation. A crawler with perfect concurrency management and a convincing TLS handshake, run from a datacenter IP with no browsing history, is still a datacenter IP to a vendor that scores IP reputation. See proxies for web scraping, explained for what actually moves that signal — Crawlee's ProxyConfiguration rotates the proxies you give it, it doesn't source trustworthy ones for you.
• Browser fingerprinting beyond JavaScript execution. Plain PlaywrightCrawler drives stock Chromium, which fails the same WebGL and headless-detection checks our stealth-browser benchmark measured directly. If a target fingerprints the browser itself rather than just requiring JS to run, pair Crawlee's orchestration with an actual stealth engine — Camoufox is the one we found passes WebGL cleanly — rather than expecting PlaywrightCrawler alone to solve it.

Crawlee vs. the alternatives

• Multiple scrapers with inconsistent rendering needs across sites: AdaptivePlaywrightCrawler is the actual reason to reach for Crawlee over hand-picking per target.
• One static site, no JS, no anti-bot: BeautifulSoupCrawler works, but plain requests + BeautifulSoup (see our beautifulsoup tutorial) is lighter for a genuinely one-off script.
• Deep Scrapy investment already, or need its spider-middleware ecosystem: Scrapy's queue/concurrency model solves the same core problem — switching frameworks isn't worth it just for AdaptivePlaywrightCrawler alone.
• Target's block is specifically TLS/JA3, not rendering: Crawlee's default ImpitHttpClient already impersonates a browser's handshake, so you likely don't need curl_cffi as a separate layer on top.
• Target fingerprints the browser itself (WebGL, canvas): swap PlaywrightCrawler's browser for Camoufox rather than expecting Crawlee's orchestration to fix a detection layer it doesn't touch.

When orchestration still isn't enough: bridge to an API

For a genuinely hard target — rotating anti-bot vendors, CAPTCHAs, IP reputation that no proxy list keeps clean — maintaining a Crawlee fleet with residential proxies and stealth browsers becomes an ongoing job in itself. Crawlora exposes the same data as structured endpoints over plain HTTPS instead:

import requests

r = requests.get(
    "https://api.crawlora.net/api/v1/amazon/search",
    params={"k": "mechanical keyboard"},
    headers={"x-api-key": "YOUR_API_KEY"},
)
data = r.json()
for item in data["data"]:
    print(item["asin"], item["list_price"], item["link"])

Proxy rotation, fingerprint upkeep, and rendering happen behind the endpoint, and the response is normalized JSON instead of DOM you keep re-selecting after every site redesign. There's an official Python SDK if you'd rather not build the request by hand, and billing is pay-on-success: a blocked or failed fetch costs nothing (pricing). Web scraping vs API walks through when DIY orchestration is still the right call and when it isn't.

Wrap-up

Crawlee's real contribution isn't any single crawler class — BeautifulSoup and Playwright scraping both existed long before it — it's putting the request queue, retries, concurrency, storage, and (as of v1.0) the HTTP-vs-browser decision itself behind one consistent API, so a rendering-requirement change doesn't mean a rewrite. pip install 'crawlee[all]', pick a crawler class, crawler.run([...]): that covers the common case. Just remember it manages the crawl, not the target's trust in your IP or fingerprint — pair it with the right lower-level tool (curl_cffi, Camoufox, or a managed proxy) for whichever detection layer you're actually up against. Try the underlying endpoints in the playground with no code at all, or browse the docs for the full catalog.

Skip the proxy-and-fingerprint stack on hard targets

Structured endpoints, managed proxies and rendering, pay-on-success billing. 2,000 free credits/month, no card.

Try the PlaygroundSee the Web Scraping API

Related reading

• Web scraping with curl_cffi and curl-impersonate: the TLS-fingerprint layer ImpitHttpClient also addresses
• Web scraping with Camoufox: the browser-fingerprint layer plain PlaywrightCrawler doesn't cover
• Web scraping with Playwright: what context.page gives you inside a PlaywrightCrawler handler
• Scrapy tutorial: the other major Python crawling-framework option, and how its model compares
• BeautifulSoup tutorial: the parsing library BeautifulSoupCrawler wraps
• Proxies for web scraping, explained: the IP-reputation layer Crawlee's orchestration doesn't touch