Glossary
Plain-English definitions for the terms developers hit most when scraping the public web — proxies and rotation, headless browsers, browser fingerprinting, rate limiting and Cloudflare blocks, CAPTCHA, robots.txt, and the parsing and API layer on top. Each entry opens with a direct answer, then the detail engineers actually need.
A headless browser is a real web browser — Chrome, Firefox, or WebKit — that runs without a visible window, executing JavaScript and rendering pages exactly like a normal browser so scripts can read the fully-rendered page instead of raw HTML.
Read definitionA rotating proxy is a proxy service that assigns a new IP address to your requests — either on every request or on a timed schedule — so traffic to a target site looks like it's coming from many different visitors instead of one machine.
Read definitionRate limiting is a server-side rule that caps how many requests a client can make in a given time window; once you exceed it, the server responds with HTTP 429 Too Many Requests instead of serving the page or API response.
Read definitionCAPTCHA (Completely Automated Public Turing test to tell Computers and Humans Apart) is a challenge — an image grid, a checkbox, or an invisible behavioral check — that a site shows a visitor it suspects might be a bot, before letting the request through.
Read definitionrobots.txt is a plain-text file published at a site's root — for example example.com/robots.txt — that tells automated crawlers which paths they're allowed or asked not to request. It's a voluntary standard, not a technical access control.
Read definitionA user agent is a string every HTTP client sends in the User-Agent request header to identify itself — browser name and version, operating system, and rendering engine. It's the first signal a website reads when deciding whether a request comes from a person or a bot.
Read definitionBrowser fingerprinting is the technique of identifying a client from the combination of signals it exposes — TLS handshake, HTTP headers, JavaScript properties, canvas and font rendering — without needing cookies or a login. Anti-bot systems use it to spot automation even when the IP address looks clean.
Read definitionA residential proxy routes your requests through IP addresses that consumer ISPs assigned to real households, so a target site sees ordinary home-internet traffic instead of a server in a datacenter.
Read definitionCloudflare error 1015 — "You are being rate limited" — is the block page Cloudflare serves when your IP exceeds the request-rate rules the site owner configured at Cloudflare's edge. It's rate limiting enforced by the CDN in front of the site, one step short of harder responses like error 1020 or a challenge page.
Read definitionA scraping API is a hosted service that fetches web pages on your behalf — running the proxies, headless browsers, retries, and anti-bot handling on its own infrastructure — and returns the page content or structured JSON through a single HTTP endpoint you call like any other API.
Read definitionHTML parsing is the step that turns a downloaded page's raw markup into a structured tree of elements your code can query — so a scraper can pull out titles, prices, or links by their position and attributes instead of string-matching against raw text.
Read definitionHTTP 403 Forbidden means the server understood your request but refuses to fulfill it — you're not authorized to see the resource, and unlike a 401, authenticating won't change the answer. For scrapers, a 403 is most often an anti-bot verdict, not a permissions problem.
Read definitionWeb crawling is the automated discovery of pages: a crawler starts from seed URLs, downloads each page, extracts its links, and follows them — building a map of what exists. Web scraping is the extraction step — pulling specific data out of pages. Crawling finds pages; scraping harvests them.
Read definitionHTTP 520 ("Web server returned an unknown error") is a Cloudflare-specific status meaning Cloudflare's edge reached the site's origin server but got back something it couldn't interpret — an empty, malformed, or crashed response. Unlike a 403 or 1015, it usually signals a problem behind Cloudflare, not a judgment about you.
Read definitionBrowse Crawlora APIs, test a request in Playground, and move from anti-bot infrastructure work to production data workflows.