Your scraper sends the right headers, a convincing User-Agent, even a residential proxy, and still gets a 403 before a single byte of HTML comes back. The reason is usually invisible to anything that inspects HTTP headers: it happens one layer down, in the TLS handshake, before your request has a User-Agent at all. curl-impersonate and its Python binding curl_cffi fix that specific layer by replaying a real browser's exact TLS and HTTP/2 fingerprint. This guide installs both, verifies the fingerprint actually changes, and is honest about the one layer they don't touch.
Why your headers were never the problem
A normal HTTP client library (Python's requests, Go's net/http, a plain curl call) negotiates TLS using whatever cipher suites, extensions, and elliptic curves its underlying TLS library ships with. Chrome negotiates TLS using BoringSSL's list, in Chrome's specific order. Those two lists are different enough that hashing them (the JA3/JA4 fingerprint) tells an anti-bot vendor which one just connected, accurately, and before your code has sent a single application-layer byte. Cloudflare, Akamai, DataDome, and PerimeterX all check this as a baseline signal, which is why a scraper can send a pixel-perfect User-Agent: Chrome/124 header and still 403 on the TLS handshake itself.
Spoofing the header layer changes nothing here. What has to change is the actual cipher suite list, extension order, and HTTP/2 SETTINGS frame your client sends: that means either patching the TLS library itself, or using a client someone already patched for you.
curl-impersonate: the underlying tool
curl-impersonate is exactly that patch: a fork of curl built against BoringSSL (for Chrome/Edge) or NSS (for Firefox) with each target browser's real handshake baked in. It ships as prebuilt binaries per browser version (curl_chrome116, curl_ff109, curl_safari15_5), each a drop-in curl replacement.
Docker, if you just want to try it:
docker pull lwthiker/curl-impersonate:0.6-chrome
docker run --rm lwthiker/curl-impersonate:0.6-chrome curl_chrome116 https://www.wikipedia.org
Native install needs the matching TLS libraries present (sudo apt install libnss3 nss-plugin-pem ca-certificates on Ubuntu, brew install nss ca-certificates on macOS), then a prebuilt binary from the project's GitHub releases. Once installed, it behaves exactly like curl:
curl_chrome116 https://tls.browserleaks.com/json
This is genuinely useful for shell scripts, cron jobs, and quick manual checks. For anything you're building in Python, which is most scraping code, you want the binding, not the CLI.
curl_cffi: the Python binding
curl_cffi wraps libcurl-impersonate in a Python API that looks almost exactly like requests, so migrating an existing scraper is usually a one-line import change plus one new keyword argument.
pip install curl_cffi --upgrade
A single impersonated GET request:
import curl_cffi
r = curl_cffi.get("https://tls.browserleaks.com/json", impersonate="chrome")
print(r.json())
impersonate="chrome" tracks the latest bundled Chrome fingerprint; pin an exact version with impersonate="chrome124" if a target is sensitive to which Chrome release connected. The library ships dozens of presets across Chrome, Safari (including safari_ios), and Firefox: pick whichever matches the User-Agent you're also sending, since a mismatch between the two is its own tell (more on that below).
For anything beyond a one-off request, use a Session: it persists cookies and connection reuse the same way requests.Session does:
s = curl_cffi.Session()
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies) # cookies persist across requests
r = s.get("https://httpbin.org/cookies")
print(r.json())
Headers merge with the impersonated profile's defaults rather than replacing them wholesale: pass your own headers={...} dict on any call the same way you would with requests, and set proxies with the same shape requests uses:
proxies = {"https": "http://user:pass@proxy-host:3128"}
r = curl_cffi.get(
"https://tls.browserleaks.com/json",
impersonate="chrome",
proxies=proxies,
headers={"Accept-Language": "en-US,en;q=0.9"},
)
SOCKS proxies work the same way with an socks:// scheme. For concurrent scraping, AsyncSession gives you the same asyncio ergonomics as httpx:
from curl_cffi import AsyncSession
async with AsyncSession() as s:
r = await s.get("https://tls.browserleaks.com/json", impersonate="chrome")
print(r.json())
Verify the fingerprint actually changed
Don't take the library's word for it. Check. tls.browserleaks.com/json echoes back the JA3/JA4 hash and negotiated cipher list your client just presented. Run the same request through plain requests and through curl_cffi and diff the output:
import requests
import curl_cffi
plain = requests.get("https://tls.browserleaks.com/json").json()
impersonated = curl_cffi.get(
"https://tls.browserleaks.com/json", impersonate="chrome"
).json()
print("plain ja3_hash: ", plain.get("ja3_hash"))
print("impersonated ja3_hash:", impersonated.get("ja3_hash"))
plain's hash will match Python's ssl module signature: recognizable and, on a real anti-bot deployment, often already denylisted. impersonated's hash should match a genuine Chrome installation's. If it doesn't, you're on a stale curl_cffi build or a target that's actively probing beyond JA3 (see the caveat below).
What TLS impersonation doesn't fix
This is the part vendor blog posts tend to skip. TLS/JA3 impersonation defeats exactly one detection layer: the network handshake. Real anti-bot deployments stack several independent signals, and getting one of them right doesn't help if the others still disagree with each other. We found a concrete version of this mismatch in a completely different tool: our own HTTP-only scraping tier once announced a convincing Chrome TLS fingerprint while sending the literal User-Agent string req/v3; every individual signal looked fine in isolation, but the two didn't agree with each other, which is exactly the kind of inconsistency an anti-bot layer is built to catch. A curl_cffi request with impersonate="chrome" and a Firefox User-Agent has the same problem.
Three practical implications:
- Keep the User-Agent and
impersonate=version aligned. If you impersonatechrome124, send a Chrome 124 User-Agent, not "the latest Chrome string I copy-pasted six months ago." - JavaScript challenges are a different problem entirely. curl_cffi never executes JavaScript, so a target that serves a JS-computed proof-of-work challenge (a common Cloudflare/DataDome pattern) will not be solved by a better TLS handshake. That needs a real or headless browser: see web scraping with Playwright or the stealth-browser fleet benchmark for what actually clears those.
- curl_cffi is a Python-first tool. Go and Node scrapers face the identical TLS-fingerprint problem: web scraping with Golang covers why Go's default
crypto/tlsstack gets flagged the same way, with community TLS-impersonation forks as the (fast-decaying) equivalent.
curl_cffi vs. a real browser vs. plain requests
- Static page, no JS, no anti-bot: plain requests or curl is simplest, don't add impersonation you don't need.
- Static-ish page behind a TLS/JA3 check (many Cloudflare/DataDome front pages): curl_cffi with a matched impersonate= version and User-Agent is the fastest fix, no browser overhead.
- Page requires JavaScript to render content or solve a challenge: curl_cffi can't help; use Playwright/Selenium, or route the fetch through a scraping API.
- High-volume production against a hard target: TLS impersonation alone decays as vendors update their fingerprint databases; a managed scraping API absorbs that churn for you.
When TLS impersonation isn't enough: bridge to an API
For a genuinely hard target (heavy JavaScript, rotating anti-bot vendors, CAPTCHAs), chasing the current fingerprint by hand becomes a maintenance job of its own. Crawlora exposes structured endpoints over plain HTTPS, so swapping in the API is a one-line change from either requests or curl_cffi:
import curl_cffi
r = curl_cffi.get(
"https://api.crawlora.net/api/v1/amazon/search",
params={"k": "mechanical keyboard"},
headers={"x-api-key": "YOUR_API_KEY"},
)
data = r.json()
for item in data["data"]:
print(item["asin"], item["list_price"], item["link"])
Proxies, browser rendering, CAPTCHA solving, and fingerprint upkeep happen behind the endpoint, and the response is normalized JSON instead of HTML you have to keep re-parsing after every site redesign. There's an official Python SDK if you'd rather not build the request by hand, and billing is pay-on-success: a blocked or failed fetch costs nothing (pricing). Whether a target is worth DIY-ing at all is its own judgment call; web scraping vs API walks through it.
Wrap-up
TLS fingerprinting is a real, specific detection layer, and curl_cffi is a real, specific fix for it: pip install curl_cffi, impersonate="chrome", done, for the class of sites whose block is exactly that check. Just don't mistake it for a general anti-bot bypass: keep your User-Agent honest, and route the JavaScript-challenge and IP-reputation problems to the tools built for those layers instead. Try the underlying endpoints in the playground with no code at all, or browse the docs for the full catalog.
Skip the fingerprint arms race on hard targets
Structured endpoints, managed proxies and rendering, pay-on-success billing. 2,000 free credits/month, no card.
Related reading
- How websites prevent web scraping in 2026: the full detection-layer landscape TLS fingerprinting sits inside
- undetected-chromedriver, nodriver, Playwright-Stealth, Camoufox: 4 engines, one real test: the signal-mismatch problem in a browser-automation context
- Web scraping with Camoufox: the browser-fingerprint layer this post's TLS layer doesn't cover
- Web scraping with Crawlee: a crawling framework whose default HTTP client now does the same TLS impersonation curl_cffi does, built in
- Scraper 403 on server, works locally? Debug it: isolating whether a block is IP, TLS, or headless detection
- Web scraping with Golang: the same TLS-fingerprint problem outside Python
- Proxies for web scraping, explained: the IP-reputation layer TLS impersonation doesn't touch
Frequently asked questions
What is curl_cffi used for?
curl_cffi is a Python binding for curl-impersonate, a patched curl build that replays a real browser's exact TLS and HTTP/2 handshake instead of Python's default TLS fingerprint. It's used to get past anti-bot systems (Cloudflare, Akamai, DataDome, PerimeterX) that fingerprint the TLS handshake before a single HTTP header is sent — a layer that spoofing headers or the User-Agent alone cannot fix.
curl_cffi vs curl-impersonate — what's the difference?
curl-impersonate is the underlying tool: a fork of curl built against BoringSSL or NSS with each target browser's handshake baked in, distributed as CLI binaries like curl_chrome116. curl_cffi is the Python binding on top of it, exposing the same impersonation through a requests-like API (curl_cffi.get(url, impersonate="chrome")) instead of shell commands. Most Python scraping code should reach for curl_cffi directly rather than shelling out to the CLI.
Does curl_cffi bypass all anti-bot detection?
No — it fixes exactly one detection layer: the TLS/JA3/JA4 handshake fingerprint. It does nothing for IP reputation (a datacenter IP is still a datacenter IP with a perfect Chrome fingerprint), JavaScript-computed challenges (curl_cffi never executes JS), or a User-Agent that doesn't match the impersonated browser version — that mismatch is itself a detectable inconsistency. Real anti-bot deployments stack multiple independent signals, so passing one doesn't guarantee passing all of them.
curl_cffi vs Playwright — which should I use?
Use curl_cffi when a target's block is a TLS/HTTP-2 fingerprint check and the data is in the HTML the server returns — it's an order of magnitude faster and lighter than launching a browser. Use Playwright (or another headless browser) when the target actually requires JavaScript execution to render content or solve a challenge, since curl_cffi has no JS engine at all. Many production scrapers use curl_cffi as the default fetch and only escalate to a real browser for the subset of pages that need one.
How do I verify my TLS fingerprint is actually being spoofed?
Request https://tls.browserleaks.com/json (or a similar JA3/JA4 checker) once with plain requests and once with curl_cffi's impersonate="chrome", then compare the returned ja3_hash values. The plain-requests hash will match Python's ssl module signature; the curl_cffi hash should match a genuine Chrome installation's. If they're identical, the impersonation isn't taking effect — check for a stale curl_cffi build first.
Does curl_cffi support proxies and async requests?
Yes to both. Pass proxies={"https": "http://user:pass@host:port"} (or a socks:// URL) to any request or Session the same way you would with the requests library, and use curl_cffi.AsyncSession for asyncio-based concurrent scraping with the same impersonate= parameter. It also has sync and async WebSocket clients that impersonate the same browser handshake for the upgrade request.
Is scraping with curl_cffi legal?
TLS impersonation itself is just a networking technique — it changes how your client negotiates a connection, not what data you request. Legality depends on what you scrape and how: public data collected respectfully (rate limits, no login-walled or personal data, checking a site's terms) is generally lower-risk than scraping private or paywalled content. curl_cffi doesn't change that calculus; it only removes one specific technical obstacle.