Glossary
Plain-English definitions for the terms developers hit most when scraping the public web — proxies and rotation, headless browsers, browser fingerprinting, rate limiting and Cloudflare blocks, CAPTCHA, robots.txt, and the parsing and API layer on top. Each entry opens with a direct answer, then the detail engineers actually need.
A headless browser is a real web browser — Chrome, Firefox, or WebKit — that runs without a visible window, executing JavaScript and rendering pages exactly like a normal browser so scripts can read the fully-rendered page instead of raw HTML.
Read definitionA rotating proxy is a proxy service that assigns a new IP address to your requests — either on every request or on a timed schedule — so traffic to a target site looks like it's coming from many different visitors instead of one machine.
Read definitionRate limiting is a server-side rule that caps how many requests a client can make in a given time window; once you exceed it, the server responds with HTTP 429 Too Many Requests instead of serving the page or API response.
Read definitionCAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a challenge — an image grid, a checkbox, or an invisible behavioral check — that a site shows a visitor it suspects might be a bot, before letting the request through.
Read definitionrobots.txt is a plain-text file published at a site's root — for example example.com/robots.txt — that tells automated crawlers which paths they're allowed or asked not to request. It's a voluntary standard, not a technical access control.
Read definitionA user agent is a string every HTTP client sends in the User-Agent request header to identify itself — browser name and version, operating system, and rendering engine. It's the first signal a website reads when deciding whether a request comes from a person or a bot.
Read definitionBrowser fingerprinting is the technique of identifying a client from the combination of signals it exposes — TLS handshake, HTTP headers, JavaScript properties, canvas and font rendering — without needing cookies or a login. Anti-bot systems use it to spot automation even when the IP address looks clean.
Read definitionA residential proxy routes your requests through IP addresses that consumer ISPs assigned to real households, so a target site sees ordinary home-internet traffic instead of a server in a datacenter.
Read definitionCloudflare error 1015 — "You are being rate limited" — is the block page Cloudflare serves when your IP exceeds the request-rate rules the site owner configured at Cloudflare's edge. It's rate limiting enforced by the CDN in front of the site, one step short of harder responses like error 1020 or a challenge page.
Read definitionA scraping API is a hosted service that fetches web pages on your behalf — running the proxies, headless browsers, retries, and anti-bot handling on its own infrastructure — and returns the page content or structured JSON through a single HTTP endpoint you call like any other API.
Read definitionHTML parsing is the step that turns a downloaded page's raw markup into a structured tree of elements your code can query — so a scraper can pull out titles, prices, or links by their position and attributes instead of string-matching against raw text.
Read definitionHTTP 403 Forbidden means the server understood your request but refuses to fulfill it — you're not authorized to see the resource, and unlike a 401, authenticating won't change the answer. For scrapers, a 403 is most often an anti-bot verdict, not a permissions problem.
Read definitionWeb crawling is the automated discovery of pages: a crawler starts from seed URLs, downloads each page, extracts its links, and follows them — building a map of what exists. Web scraping is the extraction step — pulling specific data out of pages. Crawling finds pages; scraping harvests them.
Read definitionHTTP 520 ("Web server returned an unknown error") is a Cloudflare-specific status meaning Cloudflare's edge reached the site's origin server but got back something it couldn't interpret — an empty, malformed, or crashed response. Unlike a 403 or 1015, it usually signals a problem behind Cloudflare, not a judgment about you.
Read definitionWeb scraping is the automated extraction of data from websites — fetching pages with a script instead of a person's browser, then pulling specific fields (prices, listings, reviews) out of the response and saving them in a structured format like JSON or CSV.
Read definitionA datacenter proxy routes your requests through an IP address owned by a hosting or cloud provider — AWS, OVH, Hetzner — rather than a consumer internet connection. It's the fastest and cheapest proxy type, and the easiest for a site to flag, because real people rarely browse from those IP ranges.
Read definitionAn anti-bot system is software — usually running at a CDN's edge, in front of the origin server — that scores incoming requests across multiple signals (IP reputation, browser fingerprint, behavior) to decide whether a visitor is human, then serves the response, a CAPTCHA challenge, or a block accordingly.
Read definitionHTTP 429 Too Many Requests is the status code a server sends when a client has exceeded its rate limit — it's a pacing objection, not a permissions or content problem. The request itself was valid; there were just too many of them in the current window.
Read definitionAn XML sitemap is a file — conventionally at /sitemap.xml — that lists a site's URLs, usually with metadata like when each page last changed, so search engines and other crawlers can discover pages directly instead of finding every one by following links.
Read definitionXPath (XML Path Language) is a query language for selecting nodes in an HTML or XML document tree by describing a path to them — by tag, attribute, position, or text content — giving scrapers a way to reach elements that CSS selectors alone can't target.
Read definitionTLS fingerprinting identifies a client from the shape of its TLS handshake — which cipher suites, extensions, and elliptic curves it offers, and in what order — independent of any HTTP header. JA3 and its successor JA4 are the standard formats for hashing that shape into a short, comparable string.
Read definitionAkamai Bot Manager is Akamai's enterprise bot-detection product, running at the CDN edge in front of the origin server on a large share of e-commerce, airline, ticketing, and financial-services sites. It scores requests using sensor data, device fingerprinting, and behavioral analytics before deciding whether to serve, challenge, or block.
Read definitionA proxy server is an intermediary that sits between your client and a destination site, forwarding the request and returning the response — so the site sees the proxy's IP address instead of yours. Scraping proxies are a specialized use of that same mechanism, chosen for IP type and rotation rather than privacy alone.
Read definitionA mobile proxy routes your requests through an IP address assigned by a cellular carrier to a real mobile device — 4G or 5G, not Wi-Fi. It sits at the top of the trust ladder above residential and datacenter proxies, because carrier-grade NAT means thousands of real subscribers share the same IP.
Read definitionAn ASN (Autonomous System Number) is the unique number identifying a network on the internet — assigned to ISPs, cloud providers, and large organizations — that sites use to look up who owns an IP address and judge its trustworthiness almost instantly, independent of the IP's geography or history.
Read definitionDataDome is an enterprise anti-bot and fraud-prevention vendor whose module runs at a site's edge, scoring requests in real time across network, device, and behavioral signals, and issuing a distinctive DataDome-branded challenge or block when a request scores as automated.
Read definitionCloudflare Turnstile is Cloudflare's CAPTCHA alternative — a free widget that verifies a visitor using Cloudflare's own bot-management signals instead of an image puzzle, usually passing silently with no visible challenge at all. It's the human-verification layer on a large and growing share of Cloudflare-protected sites.
Read definitionAn AI crawler is an automated bot operated by an AI company — OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended — that fetches web pages either to train a model or to answer a live user query. Sites now detect and block these separately from traditional search-engine crawlers, often by user agent.
Read definitionThe Computer Fraud and Abuse Act (CFAA) is the primary U.S. federal law governing unauthorized computer access, and it's the statute most often invoked against web scrapers. Courts have narrowed it significantly — most notably in hiQ v. LinkedIn and the Supreme Court's Van Buren decision — to mean that scraping publicly accessible data generally isn't "unauthorized access" under the law.
Read definitionHTTP 404 Not Found means the server has no resource at the requested URL. For scrapers it usually signals a bad, stale, or mistyped link — but some sites deliberately shape their 404 behavior around bots, which makes it worth distinguishing from a genuinely dead page.
Read definitionA WAF (Web Application Firewall) is a security layer that inspects HTTP requests to a web application and blocks the ones matching malicious patterns — SQL injection, cross-site scripting, credential stuffing — before they reach the origin server. Modern cloud WAFs bundle bot-management modules that apply the same edge filtering to scraping traffic.
Read definitionPerimeterX was an anti-bot and account-protection vendor that merged with HUMAN Security in 2022; its detection product now runs as HUMAN Bot Defender, still recognizable by its original _px3 and _pxde cookies, scoring requests via device fingerprinting, behavioral analysis, and network reputation across e-commerce and enterprise sites.
Read definitionKasada is an anti-bot vendor whose defense — delivered as an obfuscated JavaScript file called p.js and referred to as KPSDK — runs a custom bytecode virtual machine inside the browser to compute a proof-of-work challenge, making it one of the most technically demanding anti-bot systems to satisfy without a real browser engine.
Read definitionScrapy is an open-source Python framework for large-scale web crawling and scraping — built on an asynchronous networking engine and structured around spiders (the crawl logic), item pipelines (post-processing and storage), and middlewares (hooks for proxies, retries, and headers).
Read definitionPlaywright is an open-source browser-automation library, originally built by Microsoft, that drives Chromium, Firefox, and WebKit through a single API — the most common choice for headless-browser scraping today alongside Puppeteer, largely due to its auto-waiting behavior and multi-browser support.
Read definitionAn anti-detect browser is a specialized browser, or browser-automation wrapper, that lets an operator run many isolated fingerprint profiles — each with its own consistent canvas, WebGL, font, and navigator signature — so multiple accounts or automated sessions don't share a detectable browser fingerprint with each other.
Read definitionThe GDPR (General Data Protection Regulation) is the EU's data-protection law, governing how personal data belonging to EU residents is collected and processed — including data pulled by a scraper, since GDPR applies based on whose data it is, not where the scraping happens or whether the source page was public.
Read definitionThe CCPA (California Consumer Privacy Act, expanded by the CPRA) gives California residents rights over their personal information — including data collected through scraping — and applies to any business meeting its size or data-volume thresholds, regardless of where that business is located.
Read definitionWeb Bot Auth is an emerging standard — backed by Cloudflare, Akamai, AWS, and OpenAI, and now under an IETF working group — that lets a bot prove its identity by cryptographically signing each request with RFC 9421 HTTP Message Signatures, instead of relying on a User-Agent string a site has to trust at face value.
Read definitionAn ISP proxy (also called a static residential proxy) is an IP address registered to a real consumer ISP — Comcast, AT&T, Deutsche Telekom — but hosted on datacenter-grade servers rather than an actual household device, so it carries residential-network trust with datacenter speed and a fixed, non-rotating address.
Read definitionA honeypot trap is an element a site deliberately hides from human visitors — an invisible link, a form field no real user would see, or an entire page never linked from normal navigation — that only an automated scraper following raw HTML would interact with, giving the site a reliable bot signal.
Read definitionPuppeteer is an open-source Node.js library, maintained by the Chrome team at Google, that drives Chrome or Chromium directly over the Chrome DevTools Protocol (CDP) with no intermediary driver — one of the two dominant headless-browser tools for scraping, alongside Playwright.
Read definitionPay Per Crawl is a monetization model — pioneered by Cloudflare and now part of a broader industry push — that lets a site owner charge an AI crawler for each page it accesses, using the long-dormant HTTP 402 Payment Required status code as the machine-to-machine payment signal instead of a human-facing paywall.
Read definitionSelenium is an open-source browser-automation framework, originally built for testing in 2004, that drives real browsers through the W3C WebDriver protocol and a separate driver binary per browser — the oldest and broadest-supported tool in its category, though scraping projects mostly favor Playwright or Puppeteer today.
Read definitionBrowse Crawlora APIs, test a request in Playground, and move from anti-bot infrastructure work to production data workflows.