Glossary
Plain-English definitions for the terms developers hit most when scraping the public web — proxies and rotation, headless browsers, browser fingerprinting, rate limiting and Cloudflare blocks, CAPTCHA, robots.txt, and the parsing and API layer on top. Each entry opens with a direct answer, then the detail engineers actually need.
A headless browser is a real web browser — Chrome, Firefox, or WebKit — that runs without a visible window, executing JavaScript and rendering pages exactly like a normal browser so scripts can read the fully-rendered page instead of raw HTML.
Read definitionA rotating proxy is a proxy service that assigns a new IP address to your requests — either on every request or on a timed schedule — so traffic to a target site looks like it's coming from many different visitors instead of one machine.
Read definitionRate limiting is a server-side rule that caps how many requests a client can make in a given time window; once you exceed it, the server responds with HTTP 429 Too Many Requests instead of serving the page or API response.
Read definitionCAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a challenge — an image grid, a checkbox, or an invisible behavioral check — that a site shows a visitor it suspects might be a bot, before letting the request through.
Read definitionrobots.txt is a plain-text file published at a site's root — for example example.com/robots.txt — that tells automated crawlers which paths they're allowed or asked not to request. It's a voluntary standard, not a technical access control.
Read definitionA user agent is a string every HTTP client sends in the User-Agent request header to identify itself — browser name and version, operating system, and rendering engine. It's the first signal a website reads when deciding whether a request comes from a person or a bot.
Read definitionBrowser fingerprinting is the technique of identifying a client from the combination of signals it exposes — TLS handshake, HTTP headers, JavaScript properties, canvas and font rendering — without needing cookies or a login. Anti-bot systems use it to spot automation even when the IP address looks clean.
Read definitionA residential proxy routes your requests through IP addresses that consumer ISPs assigned to real households, so a target site sees ordinary home-internet traffic instead of a server in a datacenter.
Read definitionCloudflare error 1015 — "You are being rate limited" — is the block page Cloudflare serves when your IP exceeds the request-rate rules the site owner configured at Cloudflare's edge. It's rate limiting enforced by the CDN in front of the site, one step short of harder responses like error 1020 or a challenge page.
Read definitionA scraping API is a hosted service that fetches web pages on your behalf — running the proxies, headless browsers, retries, and anti-bot handling on its own infrastructure — and returns the page content or structured JSON through a single HTTP endpoint you call like any other API.
Read definitionHTML parsing is the step that turns a downloaded page's raw markup into a structured tree of elements your code can query — so a scraper can pull out titles, prices, or links by their position and attributes instead of string-matching against raw text.
Read definitionHTTP 403 Forbidden means the server understood your request but refuses to fulfill it — you're not authorized to see the resource, and unlike a 401, authenticating won't change the answer. For scrapers, a 403 is most often an anti-bot verdict, not a permissions problem.
Read definitionWeb crawling is the automated discovery of pages: a crawler starts from seed URLs, downloads each page, extracts its links, and follows them — building a map of what exists. Web scraping is the extraction step — pulling specific data out of pages. Crawling finds pages; scraping harvests them.
Read definitionHTTP 520 ("Web server returned an unknown error") is a Cloudflare-specific status meaning Cloudflare's edge reached the site's origin server but got back something it couldn't interpret — an empty, malformed, or crashed response. Unlike a 403 or 1015, it usually signals a problem behind Cloudflare, not a judgment about you.
Read definitionWeb scraping is the automated extraction of data from websites — fetching pages with a script instead of a person's browser, then pulling specific fields (prices, listings, reviews) out of the response and saving them in a structured format like JSON or CSV.
Read definitionA datacenter proxy routes your requests through an IP address owned by a hosting or cloud provider — AWS, OVH, Hetzner — rather than a consumer internet connection. It's the fastest and cheapest proxy type, and the easiest for a site to flag, because real people rarely browse from those IP ranges.
Read definitionAn anti-bot system is software — usually running at a CDN's edge, in front of the origin server — that scores incoming requests across multiple signals (IP reputation, browser fingerprint, behavior) to decide whether a visitor is human, then serves the response, a CAPTCHA challenge, or a block accordingly.
Read definitionHTTP 429 Too Many Requests is the status code a server sends when a client has exceeded its rate limit — it's a pacing objection, not a permissions or content problem. The request itself was valid; there were just too many of them in the current window.
Read definitionAn XML sitemap is a file — conventionally at /sitemap.xml — that lists a site's URLs, usually with metadata like when each page last changed, so search engines and other crawlers can discover pages directly instead of finding every one by following links.
Read definitionXPath (XML Path Language) is a query language for selecting nodes in an HTML or XML document tree by describing a path to them — by tag, attribute, position, or text content — giving scrapers a way to reach elements that CSS selectors alone can't target.
Read definitionTLS fingerprinting identifies a client from the shape of its TLS handshake — which cipher suites, extensions, and elliptic curves it offers, and in what order — independent of any HTTP header. JA3 and its successor JA4 are the standard formats for hashing that shape into a short, comparable string.
Read definitionAkamai Bot Manager is Akamai's enterprise bot-detection product, running at the CDN edge in front of the origin server on a large share of e-commerce, airline, ticketing, and financial-services sites. It scores requests using sensor data, device fingerprinting, and behavioral analytics before deciding whether to serve, challenge, or block.
Read definitionA proxy server is an intermediary that sits between your client and a destination site, forwarding the request and returning the response — so the site sees the proxy's IP address instead of yours. Scraping proxies are a specialized use of that same mechanism, chosen for IP type and rotation rather than privacy alone.
Read definitionA mobile proxy routes your requests through an IP address assigned by a cellular carrier to a real mobile device — 4G or 5G, not Wi-Fi. It sits at the top of the trust ladder above residential and datacenter proxies, because carrier-grade NAT means thousands of real subscribers share the same IP.
Read definitionAn ASN (Autonomous System Number) is the unique number identifying a network on the internet — assigned to ISPs, cloud providers, and large organizations — that sites use to look up who owns an IP address and judge its trustworthiness almost instantly, independent of the IP's geography or history.
Read definitionDataDome is an enterprise anti-bot and fraud-prevention vendor whose module runs at a site's edge, scoring requests in real time across network, device, and behavioral signals, and issuing a distinctive DataDome-branded challenge or block when a request scores as automated.
Read definitionCloudflare Turnstile is Cloudflare's CAPTCHA alternative — a free widget that verifies a visitor using Cloudflare's own bot-management signals instead of an image puzzle, usually passing silently with no visible challenge at all. It's the human-verification layer on a large and growing share of Cloudflare-protected sites.
Read definitionAn AI crawler is an automated bot operated by an AI company — OpenAI's GPTBot, Anthropic's ClaudeBot, PerplexityBot, Google-Extended — that fetches web pages either to train a model or to answer a live user query. Sites now detect and block these separately from traditional search-engine crawlers, often by user agent.
Read definitionThe Computer Fraud and Abuse Act (CFAA) is the primary U.S. federal law governing unauthorized computer access, and it's the statute most often invoked against web scrapers. Courts have narrowed it significantly — most notably in hiQ v. LinkedIn and the Supreme Court's Van Buren decision — to mean that scraping publicly accessible data generally isn't "unauthorized access" under the law.
Read definitionHTTP 404 Not Found means the server has no resource at the requested URL. For scrapers it usually signals a bad, stale, or mistyped link — but some sites deliberately shape their 404 behavior around bots, which makes it worth distinguishing from a genuinely dead page.
Read definitionA WAF (Web Application Firewall) is a security layer that inspects HTTP requests to a web application and blocks the ones matching malicious patterns — SQL injection, cross-site scripting, credential stuffing — before they reach the origin server. Modern cloud WAFs bundle bot-management modules that apply the same edge filtering to scraping traffic.
Read definitionPerimeterX was an anti-bot and account-protection vendor that merged with HUMAN Security in 2022; its detection product now runs as HUMAN Bot Defender, still recognizable by its original _px3 and _pxde cookies, scoring requests via device fingerprinting, behavioral analysis, and network reputation across e-commerce and enterprise sites.
Read definitionKasada is an anti-bot vendor whose defense — delivered as an obfuscated JavaScript file called p.js and referred to as KPSDK — runs a custom bytecode virtual machine inside the browser to compute a proof-of-work challenge, making it one of the most technically demanding anti-bot systems to satisfy without a real browser engine.
Read definitionScrapy is an open-source Python framework for large-scale web crawling and scraping — built on an asynchronous networking engine and structured around spiders (the crawl logic), item pipelines (post-processing and storage), and middlewares (hooks for proxies, retries, and headers).
Read definitionPlaywright is an open-source browser-automation library, originally built by Microsoft, that drives Chromium, Firefox, and WebKit through a single API — the most common choice for headless-browser scraping today alongside Puppeteer, largely due to its auto-waiting behavior and multi-browser support.
Read definitionAn anti-detect browser is a specialized browser, or browser-automation wrapper, that lets an operator run many isolated fingerprint profiles — each with its own consistent canvas, WebGL, font, and navigator signature — so multiple accounts or automated sessions don't share a detectable browser fingerprint with each other.
Read definitionThe GDPR (General Data Protection Regulation) is the EU's data-protection law, governing how personal data belonging to EU residents is collected and processed — including data pulled by a scraper, since GDPR applies based on whose data it is, not where the scraping happens or whether the source page was public.
Read definitionThe CCPA (California Consumer Privacy Act, expanded by the CPRA) gives California residents rights over their personal information — including data collected through scraping — and applies to any business meeting its size or data-volume thresholds, regardless of where that business is located.
Read definitionWeb Bot Auth is an emerging standard — backed by Cloudflare, Akamai, AWS, and OpenAI, and now under an IETF working group — that lets a bot prove its identity by cryptographically signing each request with RFC 9421 HTTP Message Signatures, instead of relying on a User-Agent string a site has to trust at face value.
Read definitionAn ISP proxy (also called a static residential proxy) is an IP address registered to a real consumer ISP — Comcast, AT&T, Deutsche Telekom — but hosted on datacenter-grade servers rather than an actual household device, so it carries residential-network trust with datacenter speed and a fixed, non-rotating address.
Read definitionA honeypot trap is an element a site deliberately hides from human visitors — an invisible link, a form field no real user would see, or an entire page never linked from normal navigation — that only an automated scraper following raw HTML would interact with, giving the site a reliable bot signal.
Read definitionPuppeteer is an open-source Node.js library, maintained by the Chrome team at Google, that drives Chrome or Chromium directly over the Chrome DevTools Protocol (CDP) with no intermediary driver — one of the two dominant headless-browser tools for scraping, alongside Playwright.
Read definitionPay Per Crawl is a monetization model — pioneered by Cloudflare and now part of a broader industry push — that lets a site owner charge an AI crawler for each page it accesses, using the long-dormant HTTP 402 Payment Required status code as the machine-to-machine payment signal instead of a human-facing paywall.
Read definitionSelenium is an open-source browser-automation framework, originally built for testing in 2004, that drives real browsers through the W3C WebDriver protocol and a separate driver binary per browser — the oldest and broadest-supported tool in its category, though scraping projects mostly favor Playwright or Puppeteer today.
Read definitionCDP detection is how anti-bot systems catch browsers driven by the Chrome DevTools Protocol — the control channel Puppeteer, Playwright, and Selenium's Chromium driver all use internally — by probing side effects of the protocol itself, most famously a JavaScript getter that only fires when the Runtime.enable domain has been switched on. It catches automation even when every static fingerprint looks perfect.
Read definitionAn AI browser agent is an AI system — OpenAI's Operator, Anthropic's Claude Computer Use, Google's Mariner, or open-source frameworks like Browser Use — that drives a real browser engine to complete a task: clicking, typing, navigating, and reading the page the way a person would, rather than just fetching and parsing a URL's raw response.
Read definitionA virtual waiting room is a queueing layer a site puts in front of a high-demand event — a ticket on-sale, a sneaker drop, a limited product launch — that holds every visitor in an ordered line before letting them through to the actual page, screening for bot and scalping traffic as part of admission rather than relying on the destination page's own defenses.
Read definitionHTTP/2 fingerprinting identifies a client from how it opens an HTTP/2 connection — the parameters it sends in its initial SETTINGS frame, and the order of its four pseudo-headers (:method, :authority, :scheme, :path) — hashed into the JA4H format, a companion to JA4's TLS fingerprint that reads the application layer instead of the handshake.
Read definitionTCP fingerprinting identifies a client from the very first packet it sends — the SYN packet's window size, TCP options and their order, maximum segment size (MSS), and time-to-live (TTL) — hashed into the JA4T format, the transport-layer companion to JA4's TLS fingerprint and JA4H's HTTP/2 fingerprint, revealing the operating system before a single byte of encrypted traffic is exchanged.
Read definitionWebDriver BiDi (bidirectional) is a W3C standard automation protocol that lets a script send commands to and receive events from a real browser over a single bidirectional connection — a vendor-neutral, cross-browser replacement for the Chrome DevTools Protocol (CDP) that Selenium, and increasingly Playwright and Puppeteer, are migrating browser control onto.
Read definitionThe EU AI Act is the European Union's risk-based AI regulation, and its general-purpose AI (GPAI) provisions are the part that reaches web scraping directly: providers training or fine-tuning a GPAI model placed on the EU market must publicly summarize their training data sources and honor machine-readable rights-reservation signals, with enforcement teeth arriving August 2, 2026.
Read definitionA SOCKS5 proxy is a general-purpose tunnel that relays raw TCP (and UDP) traffic between a client and a destination without reading or modifying the traffic's contents — unlike an HTTP proxy, which operates at the application layer and understands HTTP requests and headers specifically.
Read definitionllms.txt is a proposed standard — a markdown file published at a website's root (or any subpath) that gives AI agents and coding assistants a curated, token-efficient index of a site's most important pages — created by Jeremy Howard of Answer.AI in September 2024 and revised to a v2 draft in August 2026.
Read definitionA content signal is a robots.txt directive — `Content-Signal: search=yes,ai-train=no,use=reference` — that lets a site declare, per crawler category, whether its content may be indexed for search, used as live AI input, or absorbed into model training. Cloudflare pushed the format to its managed robots.txt for millions of domains starting in 2025 and extended it with a `use=` parameter in July 2026.
Read definitionA proof-of-work challenge is an anti-scraping technique that forces a visitor's browser to solve a small, deliberately expensive cryptographic puzzle before a page loads — cheap enough that one human visitor never notices, but costly enough that a crawler fetching thousands of pages pays a real compute bill. Anubis, an open-source reverse proxy Xe Iaso released in January 2025, is the tool that popularized the approach against AI scraper traffic.
Read definitionModel Context Protocol (MCP) is an open standard, introduced by Anthropic in November 2024, that lets an AI agent discover and call external tools — a web search, a database query, a scraping endpoint — through one consistent interface instead of a bespoke integration per data source. It has become the default tool-calling protocol most agent frameworks support by 2026.
Read definitionHTTP/3 fingerprinting profiles a client from its QUIC handshake — the UDP-based transport HTTP/3 runs over — reading the QUIC version, the TLS 1.3 ClientHello carried inside it, and the ordered set of transport parameters a client advertises, producing a signal distinct from JA4's TLS fingerprint and JA4H's HTTP/2 SETTINGS-frame fingerprint.
Read definitionPatchright is an open-source fork of Playwright that patches the library's own source code — not just the page's JavaScript environment — to suppress the CDP-protocol-level tells covered in this glossary's CDP Detection entry, including the Runtime.enable getter leak, stale automation bindings, and non-trusted input events that a JavaScript-only stealth plugin can mask on the page but can't touch at the control-channel level where they actually originate.
Read definitionSection 1201 of the Digital Millennium Copyright Act bars circumventing a technological measure that controls access to a copyrighted work, regardless of whether the underlying content itself is independently copyrightable — in 2026 it became the central legal theory platforms use against scrapers, arguing that defeating a CAPTCHA or bot-detection system is itself the circumvention, separate from any copyright claim over the scraped data.
Read definitionProxy anonymity level — transparent, anonymous, or elite (high-anonymity) — describes what an HTTP proxy reveals about itself through request headers, independent of the IP-type tiers covered elsewhere in this glossary (residential, datacenter, mobile, ISP): it's a question of header behavior, not of where the IP address comes from.
Read definitionVercel BotID is an invisible bot-detection product built directly into Vercel's hosting platform, powered by Kasada's detection engine under the hood, that lets any route deployed on Vercel add a bot challenge without integrating a separate anti-bot vendor — relevant to scrapers because Vercel hosts a large and growing share of the Next.js and AI-app sites a scraping operation is likely to target.
Read definitionAn ETag is an opaque, quoted value a server attaches to one representation of a resource. A scraper can send that value back in an If-None-Match header on a later request; if the selected representation is unchanged, the server can answer 304 Not Modified with no response body, letting the scraper reuse the body it already stored.
Read definitionCursor pagination is a way to traverse a large result set by returning an opaque continuation value with each page. The client sends that cursor with the next request — commonly as after in GraphQL or as a next-page token in REST — and stops when the response says no next page remains.
Read definitionA soft block is an informal scraping term for a response that looks successful at the HTTP layer but does not contain the requested data — for example, a challenge page, login shell, empty result template, or deliberately incomplete payload returned with a plausible status such as 200 OK.
Read definitionA crawl frontier is the crawler's managed collection of discovered requests that have not been fetched yet. It decides which URL is eligible next, filters duplicate requests, applies priorities and per-site limits, and preserves enough state for a large crawl to stop and resume without starting over.
Read definitionURL canonicalization is the process of choosing one representative URL for a resource that can be reached through multiple URL forms. Crawlers use canonical identities to avoid fetching and storing duplicates, while publishers and search engines use redirects, rel=canonical annotations, and sitemaps as signals about the preferred public URL.
Read definitionCrawl budget is the practical set of URLs a crawler can and wants to fetch from a site during a period. Google describes its own budget as the combination of crawl capacity — how much the host can serve safely — and crawl demand — which URLs the crawler considers worth revisiting or discovering.
Read definitionJSON-LD is a W3C-standard JSON syntax for Linked Data. Web pages commonly embed it in script elements to describe entities such as products, articles, organizations, and breadcrumbs with explicit types and properties that machines can parse without reconstructing the page's visual layout.
Read definitionShadow DOM is a browser mechanism that attaches an encapsulated DOM subtree to a host element. The subtree begins at a shadow root and has its own selector and style boundary, allowing web components to expose a stable outer element while keeping their internal markup separate from the page's light DOM.
Read definitionInfinite scrolling is an interface pattern that loads another batch of records when the user approaches the end of a list, replacing numbered page navigation with repeated viewport-driven requests. Scrapers must trigger the load, wait for a measurable state change, collect stable record identities, and define when the feed is finished.
Read definitionRequest interception is a browser-automation feature that pauses matching network requests before the browser sends them, giving code a chance to inspect the request and then continue it, modify supported fields, abort it, or fulfill it with a synthetic response.
Read definitionA cookie jar is a client's cookie store plus the rules that decide which stored cookies belong on each outgoing request. It processes Set-Cookie response headers, retains each cookie's domain, path, expiry, and security attributes, then constructs the Cookie header for matching requests.
Read definitionHTTP content negotiation is the process of selecting one representation of a resource based on available variants and client preferences. A client commonly sends Accept, Accept-Language, or Accept-Encoding, and the server answers with a representation whose Content-Type, Content-Language, and Content-Encoding describe what was selected.
Read definitionAn HTTP CONNECT tunnel is a byte-forwarding connection created when a client asks an HTTP proxy to connect to a destination host and port. After a successful 2xx response, the proxy stops treating subsequent bytes as ordinary HTTP messages and relays them in both directions until the tunnel closes.
Read definitionBrowse Crawlora APIs, test a request in Playground, and move from anti-bot infrastructure work to production data workflows.