A user agent is a string every HTTP client sends in the User-Agent request header to identify itself — browser name and version, operating system, and rendering engine. It's the first signal a website reads when deciding whether a request comes from a person or a bot.
A modern Chrome user agent looks like: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36. Nearly every browser starts with Mozilla/5.0 for 1990s compatibility reasons, then layers in engine and browser tokens — the string is a fossil record of browser history more than a clean identifier.
Newer Chromium browsers also send Client Hints (sec-ch-ua, sec-ch-ua-platform) alongside the classic header, and sites increasingly cross-check the two. A scraper that spoofs User-Agent but omits or contradicts the Client Hints is advertising the mismatch.
HTTP libraries announce themselves by default — python-requests/2.31, curl/8.4, Go-http-client/1.1 — and many sites block those strings outright, since almost no legitimate visitor browses with them. Setting a real browser user agent is the first, most basic step of scraping traffic hygiene.
It's also the weakest signal on its own. Anti-bot systems compare the user agent against the TLS handshake fingerprint, HTTP/2 header ordering, and JavaScript-visible properties; a Chrome user agent on a Python TLS stack is a contradiction that gets flagged even though the header looks perfect. The user agent has to be consistent with everything else, not just plausible in isolation.
import requests
headers = {
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
"AppleWebKit/537.36 (KHTML, like Gecko) "
"Chrome/126.0.0.0 Safari/537.36"
}
response = requests.get("https://example.com", headers=headers, timeout=15)How Crawlora handles this
Crawlora sends coherent, current browser identities on every request — the user agent, Client Hints, TLS fingerprint, and header ordering all match the same real browser release — which is what actually passes inspection, not a copied User-Agent string on a mismatched stack.
Glossary
FAQ
Backwards compatibility. In the 1990s, servers sent enhanced pages only to Netscape ('Mozilla'), so competing browsers claimed the token to receive the same content — and every browser since has kept it. The meaningful browser identity comes later in the string.
It helps against the most basic filters, but modern anti-bot systems cross-check the user agent against your TLS fingerprint, header order, and JavaScript environment. A browser user agent on a non-browser HTTP stack is itself a detectable contradiction.
A current, mainstream browser string (recent Chrome on Windows is the most common real-world traffic), kept up to date as versions ship — and consistent with the rest of your request fingerprint. Stale versions from years-old tutorials are a red flag on their own.
Browse Crawlora APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.