Web Scraping with Python — The Complete 2026 Guide

You need data that lives on a web page — prices, listings, search results, reviews — and there is no download button. Every tutorial you have opened either stops at "hello world" scraping a demo site, or jumps straight to a framework diagram with nothing runnable in between. Meanwhile your actual target returns a 403, renders everything with JavaScript, or changes its markup the week after your script finally works.

This guide is the whole path in one place: a runnable first scraper, then pagination and politeness, then JavaScript pages with Playwright, then an honest account of why production scrapers break — and what to do when yours does. Every code block below runs as written against public demo sites. We scrape public data only, and we say clearly where DIY stops being the right tool.

The Python scraping stack in 2026

Python is still the default language for web scraping because the three tools you need are mature, documented, and one pip install away:

The versions current as of mid-2026: requests 2.x, BeautifulSoup 4.15 (released June 2026), and Playwright for Python 1.61. The rule that keeps projects sane: use the smallest tool that fits the page. A headless browser pointed at static HTML wastes 50× the resources for the same result; requests pointed at a React app gets you an empty div.

Install the first tier and you are ready:

pip install requests beautifulsoup4

Quickstart: your first scraper in 15 lines

We will scrape books.toscrape.com, a demo bookstore built for exactly this purpose — 1,000 books with titles, prices, and stock status. The pattern is fetch, parse, select:

import requests
from bs4 import BeautifulSoup

url = "https://books.toscrape.com/"
resp = requests.get(url, timeout=10)
resp.raise_for_status()  # crash loudly on 4xx/5xx instead of parsing an error page

soup = BeautifulSoup(resp.text, "html.parser")

books = []
for card in soup.select("article.product_pod"):
    books.append({
        "title": card.h3.a["title"],
        "price": card.select_one("p.price_color").get_text(strip=True),
        "in_stock": "In stock" in card.select_one("p.availability").get_text(),
    })

print(f"Scraped {len(books)} books")
print(books[0])
# {'title': 'A Light in the Attic', 'price': '£51.77', 'in_stock': True}

Three things carry all the weight here:

• resp.raise_for_status() — beginners skip this, then spend an hour debugging why BeautifulSoup "found nothing" on what was actually a 404 page.
• soup.select(...) takes CSS selectors, the same ones you test in your browser's DevTools. Right-click the element, Inspect, and build the selector there before writing a line of Python. This step — turning messy markup into named fields — is HTML parsing, and it is where most of your scraper's code (and most of its future breakage) lives.
• Structured output from the start. A list of dicts converts to CSV, JSON, or a DataFrame in one line — we wire up the export a section from now.

BeautifulSoup goes much deeper than select — find_all with attribute filters, tree navigation, handling malformed markup. We cover the full API in our BeautifulSoup tutorial; this guide moves on to the problems that appear after the first page works.

Level 2: pagination and politeness

One page is a demo. Real datasets span pagination — and the moment you request pages in a loop, you are a bot traffic pattern, so this is also where scraping etiquette starts to matter.

import time
import requests
from bs4 import BeautifulSoup

BASE = "https://books.toscrape.com/catalogue/page-{}.html"
HEADERS = {"User-Agent": "book-price-research/1.0 ([email protected])"}

books = []
for page in range(1, 6):  # first 5 of 50 pages
    resp = requests.get(BASE.format(page), headers=HEADERS, timeout=10)
    if resp.status_code == 404:
        break  # walked off the end of the catalogue
    resp.raise_for_status()

    soup = BeautifulSoup(resp.text, "html.parser")
    for card in soup.select("article.product_pod"):
        books.append({
            "title": card.h3.a["title"],
            "price": card.select_one("p.price_color").get_text(strip=True),
        })

    time.sleep(1.0)  # ~1 request/second

print(f"{len(books)} books across 5 pages")

The habits that separate a good citizen from a nuisance:

• Throttle yourself. The time.sleep(1.0) is a floor, not a formality. A site that would never notice one request per second will absolutely notice fifty. Servers defend themselves with rate limiting — and a 429 response is the polite warning before the 403 that is not.
• Check robots.txt first. Fetch https://the-site.com/robots.txt and honor its Disallow rules — Python's built-in urllib.robotparser can evaluate them for you. It is not a law, but ignoring it is the fastest way to convert a tolerant site into a hostile one, and it drags your legality posture down with it. (On that topic: scraping public data can be lawful, but it depends on the data, the terms, and your jurisdiction — see is web scraping legal in 2026.)
• Identify yourself honestly in the User-Agent with a way to reach you, and respect 429s by backing off exponentially rather than retrying in a tight loop.
• Scrape public pages only. Nothing behind a login, nothing personal, nothing the site's terms clearly fence off.

Save it somewhere useful

Because we collected dicts, exporting is standard library territory — no pandas required for the simple case:

import csv

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=["title", "price"])
    writer.writeheader()
    writer.writerows(books)

Two habits worth forming now, while the dataset is small. First, clean at the edge: convert "£51.77" to a float when you scrape it, not three scripts downstream — float(price.lstrip("£")) today saves a locale-parsing bug later. Second, keep the raw HTML for anything you might re-parse — disk is cheap, and when you discover next month that you also needed the star rating, re-parsing saved pages beats re-crawling the site. If the destination is a spreadsheet on someone's desk rather than a database, the full pipeline — formatting, multiple sheets, recurring refreshes — is in scrape a website to Excel.

Level 3: JavaScript pages with Playwright

Run the quickstart pattern against a modern React or Vue site and you get a hollow shell — the HTML arrives nearly empty and JavaScript fills it in afterward. requests never runs that JavaScript. The quick diagnostic: if View Source doesn't contain the data but the rendered page does, you need a headless browser.

Playwright is the 2026 default for this in Python — it bundles its own browsers and waits for content intelligently:

pip install playwright
playwright install chromium

from playwright.sync_api import sync_playwright

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page()
    page.goto("https://quotes.toscrape.com/js/", timeout=30000)
    page.wait_for_selector("div.quote")  # block until JS has rendered the data

    quotes = []
    for q in page.locator("div.quote").all():
        quotes.append({
            "text": q.locator("span.text").inner_text(),
            "author": q.locator("small.author").inner_text(),
        })

    browser.close()

print(f"{len(quotes)} quotes rendered by JavaScript")

That target — quotes.toscrape.com/js/ — serves an empty page and builds every quote client-side, so it proves the point: requests gets zero quotes, Playwright gets ten. The key line is wait_for_selector, which replaces the fragile sleep(5)-and-hope approach with an explicit wait for the element you actually need.

The costs are real, though. A headless browser session uses hundreds of megabytes of RAM where requests used a few, and runs an order of magnitude slower. Two rules of thumb: only reach for Playwright when the data genuinely is not in the raw HTML, and check the Network tab first — many "JavaScript sites" fetch their data from a clean JSON endpoint you can call directly with requests. When you do need a browser, we go deep on selectors, waits, and scaling in web scraping with Playwright. If you are maintaining an older stack, Selenium still works fine and has fifteen years of Stack Overflow answers behind it — we compare the two in Selenium web scraping.

The honest part: why production scrapers break

Everything above works — genuinely, not just on demo sites. Plenty of internal tools and research projects run for years on requests, BeautifulSoup, and a cron job. But there is a wall between "works on my machine against a demo site" and "runs every night against a commercial target," and it is worth describing plainly, because this is where most scraping projects actually die.

Anti-bot systems answer before the content does. Cloudflare, DataDome, Akamai and their peers sit in front of a large share of commercial sites. They score every request, and a Python client fails that score fast: 403s on the first request, 429s once you have a pattern, JavaScript challenges that requests cannot execute. This is not an edge case — it is the default posture of e-commerce, travel, real estate, and search targets in 2026.

Your IP has a reputation, and it is bad. Datacenter IPs — your laptop on cloud Wi-Fi, your AWS box, your CI runner — are catalogued and pre-flagged. Getting past that means residential or mobile IPs cycled per request through a rotating proxy pool, which means a proxy vendor, per-GB billing, and a new category of infrastructure to babysit.

Even headless browsers get unmasked. Anti-bot vendors probe TLS handshakes, canvas rendering, installed fonts, and dozens of other signals to compute a device signature — browser fingerprinting — and stock Playwright has a recognizable one. Patched builds and stealth plugins exist, but they are a treadmill: vendors update, patches lag, your scraper silently starts failing on a Tuesday.

Layout drift is the quiet killer. The site ships a redesign, p.price_color no longer exists, and your scraper does not crash — it returns empty fields into your pipeline until someone notices the dashboard looks wrong. Every selector you write is an unversioned contract the other side never signed.

Add it up and the pattern is familiar: the scraper took a weekend to write, and now consumes a few hours every week in proxy invoices, selector repairs, and block investigations. That maintenance tax — not any missing Python skill — is the real cost of DIY, and we tally it honestly in web scraping vs API. For the counter-measures that do work when you want to stay DIY, see scraping sites that block bots.

The API path: one call replaces the stack

The alternative to fighting that wall is to hand the whole stack — proxies, browsers, fingerprints, parsers — to a web scraping API that fights it for you full-time. Here is a real Amazon search, structured JSON back, no browser, no proxy pool, no parser:

import requests

resp = requests.get(
    "https://api.crawlora.net/api/v1/amazon/search",
    params={"k": "mechanical keyboard"},
    headers={"x-api-key": "YOUR_API_KEY"},
)
resp.raise_for_status()

for item in resp.json()["data"]:
    print(item["title"], item["price"], item["rating"])

Same requests library you learned in the quickstart — the skill transfers directly. What changed is what you are responsible for:

• Structured fields, not HTML. The response is normalized JSON — title, price, rating, asin, review_count — so when Amazon redesigns, that is Crawlora's parser to fix before you notice, not yours.
• The anti-bot fight happens on the other side of the key. Proxy rotation, rendering, retries, and fingerprint upkeep are the vendor's full-time job.
• Billing is pay-on-success — you're only charged for successful 2xx responses, which matters precisely because scraping hard targets involves failures.
• The free tier is 2,000 credits/month, no card, which is enough to find out whether the data is worth building around before you spend anything.

The same pattern covers dozens of platforms — Google Search, TikTok, Zillow, GitHub, app stores — each with documented endpoints in the API docs and copy-paste Python for every one in the Python examples. If your scraper feeds an LLM agent rather than a spreadsheet, the same endpoints are exposed as a hosted MCP server (see the MCP integration) — and that agent-plus-scraping pattern is its own topic, covered in web scraping with AI.

To be fair in both directions: an API has a per-request price, covers the platforms it covers, and puts a vendor in your pipeline. DIY has none of those constraints — which is exactly why the right answer is usually a mix.

DIY vs scraping API: how to decide

The honest split: DIY wins on cost, coverage of niche targets, and control; the API wins on protected platforms, maintenance, and time-to-data. Most teams end up scraping easy and internal targets themselves and routing the hostile, high-value platforms through an API.

Before you put either into production, run the checklist:

• Is the data public, and do robots.txt and the site's terms allow what you're doing?
• Is the data in the raw HTML (requests + BeautifulSoup) or JavaScript-rendered (Playwright)?
• Are you throttled to a polite rate, with timeouts, raise_for_status, and exponential backoff on 429s?
• Do you validate scraped fields (non-empty, sane types) so layout drift fails loudly instead of silently?
• Have you priced the maintenance tax — proxies, selector repairs, block-fighting — against a scraping API's per-request cost?
• For protected targets: have you tested whether one API call gets you the same data with none of the infrastructure?

Where to go next

You now have the full 2026 map: requests and BeautifulSoup for static pages, Playwright when JavaScript is in the way, real politeness habits, and a clear-eyed view of the production wall plus the API door through it. The best next step is to run the quickstart against a target you actually care about — you will know within an hour which tier of the stack your project needs. And if that first request comes back 403, you know exactly what you are looking at, and that it is a solved problem either way. The pricing page has the credit math if you want to compare it against your proxy invoice.

Skip the wall — try the API side by side with your scraper

Structured JSON from Amazon, Google, TikTok and more. Managed proxies, rendering, and retries; pay-on-success billing. 2,000 credits/month free, no card.

Try the PlaygroundBrowse APIs

Related reading:

• BeautifulSoup tutorial — the full parsing API
• Web scraping with Playwright
• Selenium web scraping — when the older stack still makes sense
• Scrape a website to Excel
• Web scraping with AI
• Web scraping vs API
• Is web scraping legal in 2026?
• Scraping sites that block bots