Rate limiting is a server-side rule that caps how many requests a client can make in a given time window; once you exceed it, the server responds with HTTP 429 Too Many Requests instead of serving the page or API response.
Common implementations include fixed windows ("100 requests per minute, reset every minute"), sliding windows (a rolling look-back instead of a hard reset), and token buckets (you accumulate request "credits" over time and spend them, which smooths bursts). Rate limits are usually tracked per IP address, per API key, or per authenticated account — often several of these at once.
A well-behaved 429 response includes a Retry-After header — seconds, or a timestamp — telling the client exactly when it's safe to try again. Not every site sends one, which is why blind retry-with-backoff is the safer default.
Scraping naturally produces bursty, high-volume traffic from a small number of IPs — exactly the pattern rate limiters are built to catch. A single-threaded scraper looping over thousands of product pages from one IP will trip almost any rate limit within minutes.
The fix isn't to ignore the limit; it's to read Retry-After (or back off exponentially when it's absent), reduce concurrency, and — where per-IP limits are the bottleneck — spread the same request volume across more IPs so no single one crosses the threshold.
import time
import requests
def get_with_backoff(url, max_retries=5):
for attempt in range(max_retries):
response = requests.get(url, timeout=15)
if response.status_code != 429:
return response
wait = int(response.headers.get("Retry-After", 2 ** attempt))
time.sleep(wait)
response.raise_for_status()How Crawlora handles this
Crawlora's API absorbs 429s internally — retries, backoff, and request distribution across the proxy pool happen before a response reaches your code — and Crawlora only bills successful, 2xx responses, so a rate-limited attempt against the target site doesn't cost you a credit.
Related reading
Glossary
FAQ
It means the server understood your request but is refusing it because you (or your IP or API key) sent more requests than its rate limit allows in the current time window. It's not a sign your request was malformed — it's purely a volume and pacing signal.
Read the Retry-After header if present and wait that long before retrying; otherwise use exponential backoff. Longer-term, reduce request concurrency or distribute requests across more IPs so per-IP volume stays under the limit.
Yes — retry and backoff logic, plus IP distribution across the proxy pool, are handled inside the API, and failed attempts against the target site aren't billed.
Browse Crawlora APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.