Proxies for Web Scraping, Explained (2026)

Send a few hundred requests to most websites from one IP address and you will be throttled, challenged or blocked. A proxy is the standard fix: an intermediary that forwards your request and returns the response, so the site sees the proxy's IP instead of yours. Spread requests across a pool of proxies and no single address looks abusive. This guide explains the four proxy types and what they cost in 2026, how rotation and sticky sessions differ, how to compute the cost that actually matters, where proxies stop helping, and when a managed API makes the whole question somebody else's problem.

Why scraping needs proxies at all

Web servers rate-limit by IP because it is the cheapest signal they have. The limits are usually generous for a human (nobody loads 500 product pages a minute) and immediately fatal for a scraper. Proxies address exactly that:

• Distribution. A pool of a thousand IPs turns 10,000 requests an hour into 10 per IP, well under most thresholds.
• Geo-targeting. Prices, availability, search results and ad inventory differ by country and sometimes by city. A proxy in the right location returns what a local user sees.
• Ban recovery. When an IP is blocked, you rotate to another and carry on, instead of waiting for your own address to be unblocked.

Proxies are not a licence to ignore a site's rules. They are infrastructure. Rate limits, terms of service and the law still apply; see Is web scraping legal in 2026? before you scale anything.

The four proxy types, with 2026 prices

Two things the table hides. First, the residential range is wide because the market is: budget providers sell pay-as-you-go traffic near $1 per GB, established networks charge $4 to $8, and enterprise contracts with SLAs and compliance paperwork sit above $10. Second, published $/GB rates come with plan minimums, monthly commitments and traffic that expires, all of which raise your effective price. A $2/GB plan you must buy 50 GB of and use within 30 days is not $2/GB if you use 20.

Rule of thumb: start with the cheapest tier that works on your target and move up only when you get blocked. Many sites never require more than datacenter IPs with polite pacing.

The number that matters: cost per successful request

$/GB is the sticker price. What you pay for is successful responses. Divide price by success rate:

On that target the budget residential pool wins. Change the success rates (a lenient site might pass datacenter at 95%) and the ranking flips. Measure on your own target with a few hundred requests before committing to a plan. Also count the bandwidth you waste: blocked responses, challenge pages and retries all consume GB you pay for.

Rotating vs sticky sessions

• Rotating proxies hand you a different IP on each request, or every few minutes, from a large pool. Use them for independent requests: search result pages, product listings, profile lookups, anything where request N does not depend on request N-1.
• Sticky (session) proxies keep the same IP for a set window, typically 1 to 30 minutes, sometimes longer. Use them when a flow must keep one identity: logging in, adding to a cart, walking paginated results that are tied to a server-side session, or any target that treats an IP change mid-flow as a red flag.

Most providers sell both from the same pool, switching by a session parameter in the proxy username or a dedicated port. ISP proxies are the extreme sticky case: one IP, yours for the month.

Where proxies stop helping

A clean IP with a sloppy client still gets blocked. Modern anti-bot vendors (Cloudflare, Akamai, DataDome, Kasada, PerimeterX and others) score requests on far more than the address:

• TLS fingerprint. The cipher suites and extensions your HTTP library offers identify it as Python requests, Go net/http or a real Chrome build. See what is a TLS fingerprint.
• HTTP/2 and header order. Real browsers send headers in a particular order with particular defaults; most scraping libraries do not.
• JavaScript challenges and CAPTCHAs. Many pages render nothing useful until a script runs, and some gate on a challenge a headless browser must solve.
• Behaviour. Request timing, navigation paths and mouse or scroll events on rendered pages.

So the proxy is one layer of a working setup: proxies, plus a client whose TLS and headers look like a browser, plus a real browser engine for pages that render client-side, plus retry and challenge handling. That stack is the actual cost of DIY scraping on defended targets, and it is why the residential bill is often the smaller half of the total. Our anti-bot index tracks which vendors the top sites use, which is a fair predictor of how much of this stack a given target will need.

How scraping APIs price proxies

Generic scraping APIs bundle proxies into their per-request price and expose the tier as a flag, which makes the cost visible in a different way:

• ZenRows charges 1 credit for a plain request, 10 with premium (residential) proxies, 25 with premium proxies and JavaScript rendering, and bills residential bandwidth separately at 25,000 credits per GB.
• ScrapingBee charges 1 credit plain, 10 with a premium proxy, 25 with premium plus rendering, and 75 for its stealth proxy tier.
• Scrape.do and Crawlbase include proxies in every request and vary credits by proxy class or rendering rather than charging per GB.

The pattern is the same as buying proxies directly: the residential tier is roughly an order of magnitude dearer than datacenter, and rendering adds another multiplier on top. See the ZenRows, ScrapingBee and Scrape.do comparisons for the full ladders.

When you don't need to manage proxies at all

If your target is a supported public platform, a structured API removes the tier decision entirely. With Crawlora, proxy routing, browser rendering, and retries run behind each endpoint and are priced into one fixed credit weight, billed only on success. You call the Google Search, Google Maps or Amazon endpoint and get normalized JSON back; which proxy tier the request needed that day is not something you see or pay extra for.

curl -s -X POST "https://api.crawlora.net/api/v1/google/search" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"keyword": "coffee shops in austin", "language": "en", "country": "us", "limit": 10}'

You manage your own proxies when you crawl arbitrary or unsupported sites; you skip them when a documented endpoint already covers the source. Most teams end up with both: a platform API for the sites everyone scrapes, and a proxy pool (or a generic scraping API) for the long tail.

Quick decision guide

• Lenient target, high volume, tight budget → datacenter proxies, paced politely.
• Defended target, need to look like a real user, many independent requests → rotating residential.
• Defended target, multi-step flow or login → sticky residential, or ISP proxies if the session must survive for days.
• Most aggressively defended target, and it is worth it → mobile proxies, and expect to need a real browser too.
• Supported public platform → a structured API, and skip proxy management.

Sources

Start collecting

Skip proxy management for supported sources: test an endpoint in the Playground, browse the API docs, and review pricing. For the rest of the stack, see how websites prevent web scraping, scraping sites that block bots, and is web scraping legal.