A headless browser is a real web browser — Chrome, Firefox, or WebKit — that runs without a visible window, executing JavaScript and rendering pages exactly like a normal browser so scripts can read the fully-rendered page instead of raw HTML.
A plain HTTP client (curl, Python requests, fetch) only downloads the HTML a server sends — it never runs the page's JavaScript. Modern sites built with React, Vue, or Next.js render most of their content client-side, so that raw HTML response is often close to empty. A headless browser loads the page in a real rendering engine, runs the scripts, and lets you read the DOM after everything has executed.
It's called "headless" because there's no GUI window — the same Chromium or Firefox engine runs, just without pixels drawn to a screen. That makes it scriptable and fast enough to run many instances on a server at once.
Headless browsers are the standard tool for scraping single-page apps, infinite-scroll feeds, or pages that only populate data after an XHR/fetch call resolves. Libraries like Playwright, Puppeteer, and Selenium (in headless mode) drive Chromium, Firefox, or WebKit programmatically — click buttons, wait for network idle, then extract the rendered DOM.
The tradeoff is cost: a browser process uses far more CPU and memory than a raw HTTP request, and it's slower per page. Most production scraping systems only spin up a browser when the target actually needs one, and fall back to plain HTTP for everything else.
Headless browsers leave fingerprints — navigator.webdriver set to true, missing plugins, or a Chromium build string that doesn't match a real Chrome release. Anti-bot vendors like Cloudflare, Akamai, and DataDome check exactly these signals, so a default headless setup gets blocked faster than a well-configured HTTP client.
Serious scraping infrastructure pairs the browser with stealth patches and a matching residential IP — the browser alone doesn't solve anti-bot detection, it just adds JavaScript execution to the toolkit.
from playwright.sync_api import sync_playwright
with sync_playwright() as p:
browser = p.chromium.launch(headless=True)
page = browser.new_page()
page.goto("https://example.com", wait_until="networkidle")
html = page.content() # fully-rendered DOM, not the raw server response
browser.close()How Crawlora handles this
Crawlora's browser-rendering tier runs this rendering step for you — headless Chromium, stealth patches, and a matched residential IP — so requests return the fully-rendered page, or normalized JSON for the platform endpoints, without you standing up and maintaining browser infrastructure.
Glossary
FAQ
Yes, by default. Headless Chrome and Firefox expose fingerprint differences — navigator.webdriver, missing plugins, WebGL renderer strings — that anti-bot services check for. Avoiding detection requires stealth patches and consistent, realistic browser and IP configuration, not just running headless.
No. Use one only when the data you need isn't present in the initial HTML response — for example, it's rendered by client-side JavaScript. Static or server-rendered pages can be scraped with a plain HTTP request, which is faster and cheaper.
Playwright and Puppeteer (both Chromium-based, with Playwright also supporting Firefox and WebKit) are the current standards; Selenium still works but has a heavier API and slower startup for pure scraping use.
Browse Crawlora APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.