Web Scraping with Crawl4AI in 2026

Most Python scraping frameworks hand you raw HTML and leave the "turn this into something an LLM can read" step to you. Crawl4AI inverts that: it's a self-hosted, async Python crawler (Apache-2.0, 50,000+ GitHub stars) built specifically to turn an arbitrary URL into clean Markdown, with CSS/XPath or LLM-based structured extraction layered on top when Markdown alone isn't enough. This guide installs it, runs a basic crawl, picks between its extraction strategies, and is honest about the two things it doesn't fix — a hard anti-bot deployment, and the operational care a crawler with a documented history of security advisories still needs.

What Crawl4AI actually does

pip install crawl4ai gets you a single Python package built around one core idea: point AsyncWebCrawler at a URL, get back Markdown formatted for an LLM to read — not the raw HTML a requests.get() would return, and not a hand-built DOM-to-text pipeline you maintain yourself. That Markdown-first output is the actual differentiator over a general crawling framework like Crawlee or Scrapy: both of those hand you a parsed object and leave content cleanup to your own code, where Crawl4AI's DefaultMarkdownGenerator and content filters do that step before the result ever reaches you.

It runs entirely on infrastructure you provide — your servers, your Playwright browser instances, your proxies if a target needs them — which is the same self-hosting tradeoff our Crawl4AI alternatives guide covers from the "should I self-host at all" angle. This post assumes you've already made that call and walks through actually using it.

Install it

pip install crawl4ai
crawl4ai-setup

crawl4ai-setup is the step that's easy to skip and then hit a confusing error for: it installs the Playwright browser binaries Crawl4AI's browser-based crawling needs, runs OS-level checks (missing Linux libraries, mainly), and confirms the environment is actually ready to crawl. If something's still off afterward, crawl4ai-doctor is a diagnostic command that checks your Python version, verifies the Playwright install, and flags environment-variable or library conflicts — rerun crawl4ai-setup once it tells you what to fix.

Run a basic crawl

import asyncio
from crawl4ai import AsyncWebCrawler, BrowserConfig, CrawlerRunConfig, CacheMode

async def main() -> None:
    browser_config = BrowserConfig(headless=True)
    run_config = CrawlerRunConfig(cache_mode=CacheMode.BYPASS)

    async with AsyncWebCrawler(config=browser_config) as crawler:
        result = await crawler.arun(
            url="https://quotes.toscrape.com/",
            config=run_config,
        )
        if result.success:
            print(result.markdown[:500])
        else:
            print(f"Crawl failed: {result.error_message}")

if __name__ == "__main__":
    asyncio.run(main())

BrowserConfig controls the Playwright browser itself (headless mode, viewport, proxy, user agent); CrawlerRunConfig controls everything about one specific crawl run (cache behavior, JS to execute, wait conditions, which extraction strategy to apply). Both get passed in once, not re-specified per call. CacheMode.BYPASS is worth setting explicitly rather than relying on the default — Crawl4AI caches fetched pages by URL, which is the right call for a crawl you're re-running during development but silently returns stale content if you forget it's on for a production job that expects fresh data every run.

The async with block handles browser startup and teardown automatically. For a long-running service that crawls many URLs without restarting a browser each time, call crawler.start() once and crawler.close() when you're actually done instead.

Pick an extraction strategy

Markdown alone is enough when you're feeding an LLM or a RAG pipeline. When you need structured fields back, Crawl4AI gives you two different tools depending on how regular the page's markup actually is.

JsonCssExtractionStrategy for pages with a stable, repeating structure — fast, no LLM call, and it fails loudly (empty fields) rather than silently hallucinating a value when the selector stops matching:

from crawl4ai import JsonCssExtractionStrategy

schema = {
    "name": "Quotes",
    "baseSelector": ".quote",
    "fields": [
        {"name": "text", "selector": ".text", "type": "text"},
        {"name": "author", "selector": ".author", "type": "text"},
    ],
}

run_config = CrawlerRunConfig(extraction_strategy=JsonCssExtractionStrategy(schema))

JsonXPathExtractionStrategy is the same idea with XPath selectors instead of CSS, for the cases where an XPath expression is the only clean way to reach a field (text matching, sibling-relative selectors CSS can't express).

LLMExtractionStrategy for everything a CSS/XPath schema can't cleanly describe — inconsistent markup across a page template, fields defined by meaning rather than position, or a target you don't want to write a selector schema for at all:

from crawl4ai import LLMExtractionStrategy, LLMConfig

strategy = LLMExtractionStrategy(
    llm_config=LLMConfig(provider="openai/gpt-4o-mini", api_token="env:OPENAI_API_KEY"),
    instruction="Extract the quote text and the author's name as JSON.",
)
run_config = CrawlerRunConfig(extraction_strategy=strategy)

The tradeoff is the obvious one: an LLM call per page costs money and latency a CSS selector doesn't, and introduces the usual LLM failure modes (a malformed field, an occasional hallucinated value) that a deterministic selector can't produce. Reach for it when the page genuinely doesn't have a stable schema, not as a default for every extraction.

Crawl a whole site, not one URL

crawl4ai.deep_crawling turns a single-page fetch into a site crawl without reaching for a separate framework:

from crawl4ai.deep_crawling import BFSDeepCrawlStrategy

run_config = CrawlerRunConfig(
    deep_crawl_strategy=BFSDeepCrawlStrategy(max_depth=2, max_pages=50),
)

BFSDeepCrawlStrategy crawls breadth-first (every link at depth 1 before any at depth 2); DFSDeepCrawlStrategy goes depth-first instead; BestFirstCrawlingStrategy scores discovered links and prioritizes the most relevant ones first rather than crawling in link-discovery order. Which one to pick depends on the shape of the site: BFS for a reasonably flat site where you want broad coverage fast, DFS when a specific branch is more likely to have what you want, best-first when you have a scoring signal (a keyword match, a URL pattern) worth prioritizing on.

Check this before running it as a service

Crawl4AI has a real, documented security history worth knowing rather than discovering later: CVE-2026-26216 (remote code execution via user-supplied hook code) and CVE-2026-26217 (path traversal), both affecting versions before 0.8.0 and fixed in the 0.8.7 security-hardening release. Pin to 0.8.7 or later — 0.9.5 is current as of this post — and don't run an old pin just because it still imports cleanly.

The other operational detail: Crawl4AI ships a Docker-based API server for running it as a standalone service rather than importing it as a library. As of v0.9.0, that server is secure-by-default — authentication is on automatically, where earlier versions required you to remember to turn it on yourself. If you're running an older Docker image, don't assume it's locked down; check the version and upgrade before exposing it past localhost.

What Crawl4AI doesn't fix

Crawl4AI's browser-based crawling runs on Playwright-driven Chromium under the hood — the same browser engine Crawlee's PlaywrightCrawler and a stock Playwright script both use. That means the same gaps apply:

• Browser fingerprinting. Stock Chromium fails the same WebGL and headless-detection checks our stealth-browser benchmark measured directly — Crawl4AI's Markdown-generation layer sits on top of the browser, it doesn't patch what the browser itself leaks. If a target fingerprints the browser rather than just requiring JS execution, pair Crawl4AI's crawl with Camoufox as the actual browser, or accept that a hard anti-bot target is outside what Crawl4AI alone solves.
• IP reputation. Markdown-ready output from a datacenter IP with no browsing history is still a datacenter IP to a vendor scoring IP reputation. See proxies for web scraping, explained for what actually moves that signal.
• CAPTCHAs and managed challenge pages. Crawl4AI has no built-in challenge solver — a Cloudflare Turnstile or similar interactive challenge stops the crawl the same way it would stop a stock Playwright script.

Crawl4AI vs. the alternatives

• Feeding an LLM or RAG pipeline from arbitrary URLs you don't control the markup of: Crawl4AI's Markdown-first output is the actual reason to reach for it over a generic parser.
• A fixed, known set of structured fields from a consistently-templated page: JsonCssExtractionStrategy covers it without an LLM call — see if the page's markup is stable enough first.
• Deep crawling investment already in Crawlee or Scrapy: both have mature queue/concurrency models; switching frameworks isn't worth it just for Crawl4AI's Markdown generator alone unless LLM-ready output is specifically what you're missing.
• Target fingerprints the browser itself (WebGL, canvas): Crawl4AI's stock Chromium doesn't clear that bar any better than plain Playwright does — swap in Camoufox, or expect to hit the same wall.
• Deciding whether to self-host at all versus a hosted API: that's the question our Crawl4AI alternatives page answers; this post assumes you've already decided to self-host.

When self-hosting still isn't enough: bridge to an API

Self-hosting Crawl4AI means you own the browser fleet, the proxy budget, and the Markdown/extraction pipeline — a fine trade when the target is cooperative and the volume is your own to manage. For a target that's actively hostile — rotating anti-bot vendors, CAPTCHAs, IP reputation no proxy list keeps clean — maintaining that infrastructure becomes an ongoing job rather than a one-time setup. Crawlora exposes the same kind of data as structured endpoints over plain HTTPS instead, with proxy rotation, fingerprint upkeep, and rendering handled behind the endpoint:

import requests

r = requests.get(
    "https://api.crawlora.net/api/v1/amazon/search",
    params={"k": "mechanical keyboard"},
    headers={"x-api-key": "YOUR_API_KEY"},
)
data = r.json()
for item in data["data"]:
    print(item["asin"], item["list_price"], item["link"])

That's normalized JSON for a known platform, not generic Markdown you'd configure per site — the two tools are actually solving different scopes, Crawl4AI for arbitrary URLs you control the extraction logic for, Crawlora for named platforms where a documented schema beats configuring your own. There's an official Python SDK if you'd rather not build the request by hand, and billing is pay-on-success: a blocked or failed fetch costs nothing (pricing). Web scraping vs API walks through that scope tradeoff in more depth.

Wrap-up

Crawl4AI's real pitch is narrower than "another scraping framework" — it's specifically the Markdown-first, LLM-pipeline-ready output, with CSS/XPath or LLM-based extraction layered on when Markdown alone doesn't give you the structured fields you need, and deep crawling built in rather than bolted on. pip install crawl4ai, crawl4ai-setup, AsyncWebCrawler().arun(url): that covers the common case. Pin a version past 0.8.0 for the patched CVEs, check your Docker image's vintage if you're running the API server, and remember it drives stock Chromium underneath — pair it with a real stealth engine or a hosted API for whatever detection layer the target actually checks. Try the underlying endpoints in the playground with no code at all, or browse the docs for the full catalog.

Skip the proxy-and-fingerprint stack on hard targets

Structured endpoints, managed proxies and rendering, pay-on-success billing. 2,000 free credits/month, no card.

Try the PlaygroundSee the Web Scraping API

Related reading

• Crawl4AI alternatives: the "should I self-host at all" question this post assumes you've already answered
• Web scraping with Crawlee: the orchestration-first framework alternative, if deep queue/concurrency management matters more than Markdown-first output
• Web scraping with Scrapling: adaptive-parser framework with its own LLM-adjacent angle, a different bet than Crawl4AI's Markdown generation
• Web scraping with Camoufox: the browser-fingerprint layer Crawl4AI's stock Chromium doesn't cover
• Our stealth-browser engine benchmark: the WebGL/headless-detection test this post's fingerprinting claim is sourced from
• BeautifulSoup tutorial: the lighter option for a genuinely one-off static-page script with no Markdown-pipeline need
• Proxies for web scraping, explained: the IP-reputation layer no self-hosted crawler touches on its own