Web Scraping with Playwright — A Practical Python Guide

Playwright is the strongest general-purpose browser automation library for web scraping in 2026: one Python API drives Chromium, Firefox, and WebKit, its locators auto-wait so scripts stop breaking on timing, and page.route lets you block heavy assets or read a site's own JSON responses instead of parsing HTML at all. This guide takes you from pip install to a working scraper against a sandbox site, then through the power features — resource blocking, response interception, parallel contexts — and finishes with the honest part most tutorials skip: how detectable headless Playwright is, and what running it at scale actually costs.

Why Playwright for scraping in 2026

If the page you need renders its data with JavaScript, requests plus BeautifulSoup never sees it — those tools read the raw HTML the server sends, not what a browser builds afterward. That stack is still the right default for static pages (start with our Python web scraping guide and the BeautifulSoup tutorial if that's your case). For everything else you need a headless browser, and Playwright has become the default choice for four concrete reasons:

• Auto-waiting locators. Every action on a Locator — click, read text, get attribute — waits for the element to be attached, visible, and stable before running, and retries until a timeout. The time.sleep(3) scattered through old Selenium scripts simply disappears.
• One API, three engines. p.chromium, p.firefox, and p.webkit share the same interface. If a site behaves differently under Chrome's engine, switching to Firefox is a one-word change.
• Browser contexts. A context is an isolated, incognito-style session inside one browser process — own cookies, own storage. Contexts are cheap to create and destroy, which is how Playwright does parallelism without launching a browser per task.
• Network interception. page.route sits between the browser and the network, so you can abort requests for images and fonts, or capture the JSON a site's own frontend fetches — often the cleanest data source on the page.

Versus Selenium, the honest comparison: Selenium has a larger legacy ecosystem, more language bindings, and Selenium Grid is battle-tested for distributed testing. But Playwright's auto-waiting, built-in network interception, and context model make scraper code shorter and less flaky — Selenium needs explicit WebDriverWait conditions and a proxy tool like selenium-wire for anything network-level. We wrote up the full comparison in Selenium web scraping; for new scraping projects, Playwright is the better start.

Versus requests + BeautifulSoup: a browser is roughly two orders of magnitude more expensive per page than a plain HTTP request. Use Playwright because you must (JavaScript rendering, user interaction, session flows), not because it's the shiny option.

Setup and your first scrape

Two commands. The first installs the library, the second downloads the browser binary — install only Chromium unless you know you need the others:

pip install playwright
playwright install chromium

Here is a complete, runnable scraper against books.toscrape.com, a sandbox built for practicing:

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://books.toscrape.com/")

    books = []
    for card in page.locator("article.product_pod").all():
        books.append({
            "title": card.locator("h3 a").get_attribute("title"),
            "price": card.locator(".price_color").inner_text(),
            "in_stock": "In stock" in card.locator(".availability").inner_text(),
        })

    print(f"scraped {len(books)} books")
    print(books[0])
    browser.close()

Run it and you get 20 dicts with title, price, and stock status. Note what is not in this script: no waits, no sleeps, no retry loop. page.goto waits for the page to load, and each locator action waits for its element. That's the auto-waiting doing its job.

One caveat from the official docs worth knowing on day one: the sync API is not thread-safe. One sync_playwright() instance per thread, or use the async API for concurrency (more on that below).

Locators done right

page.locator() takes CSS (or XPath, auto-detected), but Playwright's docs push you toward the user-facing locators — and for scrapers that advice pays off, because roles and visible text survive redesigns that shred CSS class chains:

# CSS — fine for stable, semantic markup
page.locator("article.product_pod")

# Role-based — survives class renames
page.get_by_role("link", name="next").click()

# Text-based — good for labels and buttons
page.get_by_text("Add to basket")

Locators are strict by default: if a locator matches two elements and you act on it, Playwright raises an error instead of silently picking the first. Narrow with .filter(has_text="..."), .first, or .nth(i) when you genuinely want one of many.

The workhorse pattern for scraping is: locate the repeating container, call .all(), then extract fields relative to each item — exactly what the first script did. Extend it with pagination and you have a real crawler:

books = []
while True:
    for card in page.locator("article.product_pod").all():
        books.append({
            "title": card.locator("h3 a").get_attribute("title"),
            "price": card.locator(".price_color").inner_text(),
        })
    next_link = page.locator("li.next a")
    if next_link.count() == 0:
        break
    next_link.click()

print(f"{len(books)} books across all pages")

count() is the idiomatic "does this exist?" check — it returns immediately instead of waiting for a timeout the way an action on a missing element would.

Power features for scrapers

Block images, fonts, and media with page.route

A scraper doesn't need to see the page, so don't pay to render it. Register a route handler before navigating and abort the request types you don't need:

BLOCKED_TYPES = {"image", "font", "media"}

def block_heavy(route):
    if route.request.resource_type in BLOCKED_TYPES:
        route.abort()
    else:
        route.continue_()

page.route("**/*", block_heavy)
page.goto("https://books.toscrape.com/")

On image-heavy pages this cuts transferred bytes by half or more and shaves seconds off every navigation — Scrapfly's benchmarks put the bandwidth saving at 2–5x on typical sites. You can also block by URL pattern (page.route("**/*.png", lambda r: r.abort())) for surgical cases. Two cautions: blocking stylesheets can break selectors that depend on visibility, and blocking analytics scripts on some anti-bot-protected sites makes you more suspicious, not less.

Intercept JSON responses instead of scraping the DOM

Modern sites often load data through background XHR/fetch calls and render it client-side. That means the structured data you want already exists as JSON on the wire — parsing the DOM afterward is scraping a lossy copy. Capture the response directly:

with page.expect_response("**/api/search*") as resp_info:
    page.get_by_role("button", name="Search").click()

data = resp_info.value.json()  # the site's own structured payload

For passive capture across a whole session, attach a listener instead:

captured = []

def on_response(response):
    if response.request.resource_type in ("xhr", "fetch") and "/api/" in response.url:
        captured.append(response.json())

page.on("response", on_response)
page.goto("https://example.com/catalog")

When it works, this is strictly better than DOM scraping: typed fields, no selector maintenance, and often extra fields the page never displays. Open DevTools' Network tab on your target site before writing any locators — you may not need them.

Parallel scraping with browser contexts

Don't launch a browser per URL. Launch one browser and open many contexts — each is an isolated session at a fraction of the cost:

import asyncio
from playwright.async_api import async_playwright

async def scrape_one(browser, sem, url):
    async with sem:
        context = await browser.new_context()
        page = await context.new_page()
        await page.goto(url)
        title = await page.locator("h1").inner_text()
        await context.close()
        return {"url": url, "title": title}

async def main(urls):
    sem = asyncio.Semaphore(4)  # 3–5 concurrent pages per machine is realistic
    async with async_playwright() as p:
        browser = await p.chromium.launch()
        results = await asyncio.gather(*(scrape_one(browser, sem, u) for u in urls))
        await browser.close()
    return results

The semaphore matters: each open page costs CPU and a few hundred MB of memory, and unbounded concurrency hammers the target site into rate limiting you. Throttle per-domain, not just globally.

Sync or async?

Use the sync API for scripts, one-off jobs, and anything a cron runs — it's easier to read and debug. Reach for the async API when you need concurrent pages in one process, or when your scraper lives inside an async app. The two APIs mirror each other method-for-method, so upgrading later is mechanical, not a rewrite.

Common failures and how to fix them

Can Playwright be detected? Yes — here's the honest picture

Out of the box, headless Playwright is detectable, and any tutorial that skips this is setting you up. The tells are well documented: navigator.webdriver is true, the CDP (Chrome DevTools Protocol) connection Playwright depends on leaves runtime artifacts that detection scripts probe for, and headless Chromium's browser fingerprint — fonts, WebGL renderer strings, missing plugin lists, default viewport — differs measurably from a real user's browser. Anti-bot vendors test against Playwright specifically, because everyone uses it.

Patches exist. playwright-stealth (a port of puppeteer-stealth) rewrites the obvious JavaScript properties, and patched builds go further. But this is an arms race you are on the wrong side of: detection vendors ship updates continuously, and every fix you apply is a signature they already collect. Expect CAPTCHAs, silent junk data, and IP blocks on protected targets — we cover the escalation ladder in scraping sites that block bots.

Then there's cost, which bites even when detection doesn't. A browser page needs a real CPU slice and hundreds of MB of RAM; a machine that pushes thousands of plain HTTP requests a minute manages a few dozen concurrent Playwright pages. Add residential proxies for protected sites and per-page economics get worse. Playwright is a superb tool — for the sites where it works, nothing in Python is more pleasant. But "run more browsers" is an expensive answer to scale.

When to stop running browsers

For protected, high-volume targets, the pragmatic move is to let someone else run the browsers, proxies, and unblocking — and consume structured JSON over a normal HTTP API. That's what Crawlora's web scraping API does: documented per-platform endpoints, normalized fields, no parser or fingerprint to maintain. An Amazon search is one plain request:

import requests

resp = requests.get(
    "https://api.crawlora.net/api/v1/amazon/search",
    params={"k": "wireless earbuds"},
    headers={"x-api-key": "YOUR_API_KEY"},
)
for item in resp.json()["data"][:3]:
    print(item["title"], item["price"], item["rating"])

Billing is pay-on-success — you're charged only for successful 2xx responses — and the free tier is 2,000 credits/month, no card, which is enough to test whether the endpoints cover your targets before writing any browser code. See pricing for how credits map to endpoints, or poke at live responses in the playground.

A sane decision rule for the whole stack:

• Static HTML, no JavaScript rendering needed — use requests + BeautifulSoup, not a browser.
• JavaScript-rendered pages, unprotected or lightly protected — Playwright, with asset blocking and response interception.
• The data arrives via a background API call — intercept the JSON; you may not need locators at all.
• Protected platforms (retail, search, social) at volume — a structured scraping API beats maintaining stealth patches and proxy pools.
• Mixed workload — most real projects run Playwright for the long tail and an API for the hard, high-volume targets.

Skip the browser farm for protected sites

Documented endpoints, normalized JSON, managed proxies and unblocking. Pay-on-success billing, and 2,000 credits/month, no card.

Try the PlaygroundRead the Docs

Related reading

• Web scraping with Python: the complete guide — the pillar: choosing between requests, BeautifulSoup, and browsers.
• Selenium web scraping — the full Playwright vs Selenium comparison for scrapers.
• BeautifulSoup tutorial — parsing HTML when you don't need a browser.
• Scraping sites that block bots — what detection escalation looks like and your options at each rung.