BeautifulSoup Tutorial — Python Web Scraping from Scratch

BeautifulSoup is the standard Python library for parsing HTML and pulling data out of web pages. It does not download pages — requests does that — and it does not run JavaScript. What it does, better than anything else at its size, is turn messy real-world markup into a tree you can search with two lines of code. This tutorial takes you from pip install to a complete pagination-aware scraper that writes CSV, using one consistent example site throughout, then is honest about where BeautifulSoup stops and what to reach for next. It is one chapter of our broader web scraping with Python guide — start there if you want the full map.

Setup: install bs4, lxml, and requests

Everything in this tutorial needs three packages. Note the pip name is beautifulsoup4, not bs4 — the import is bs4, the package is beautifulsoup4:

python -m pip install beautifulsoup4 lxml requests

BeautifulSoup itself does not parse HTML — it delegates to a parser backend and gives you a friendly API over the result. You choose the backend when you build the soup:

Two rules of thumb: use "lxml" unless you have a reason not to, and always pass the parser explicitly. If you omit it, BeautifulSoup picks the best parser installed on that machine — which means the same script can parse badly broken markup slightly differently on your laptop and your server.

Your first soup

We will use books.toscrape.com for every example — a sandbox bookstore built specifically for scraping practice, with 1,000 books across 50 paginated pages. Fetch the homepage and parse it:

import requests
from bs4 import BeautifulSoup

resp = requests.get("https://books.toscrape.com/")
resp.raise_for_status()

soup = BeautifulSoup(resp.content, "lxml")
print(soup.title.get_text(strip=True))
# All products | Books to Scrape - Sandbox

One deliberate choice there: we pass resp.content (raw bytes), not resp.text. This site — like plenty of real ones — omits the charset from its Content-Type header, so resp.text decodes the page wrongly and every £ becomes £. Given bytes, BeautifulSoup sniffs the encoding from the document itself and gets it right. More on this in the errors table below.

The soup object is the whole document as a searchable tree. Every book on the page lives in an article tag with the class product_pod — that one fact drives everything below.

find and find_all

find_all returns a list of every matching element; find returns the first match or None:

books = soup.find_all("article", class_="product_pod")
print(len(books))  # 20 — one per book on the page

first = soup.find("article", class_="product_pod")

Note class_ with a trailing underscore, because class is a reserved word in Python. You can filter on any attribute the same way (soup.find_all("a", href=True)), pass a dict for awkward attribute names (attrs={"data-id": "42"}), cap results with limit=5, or pass recursive=False to search only direct children. If you see findAll in an old tutorial, it is the deprecated pre-4.0 spelling of find_all — same method.

CSS selectors: select and select_one

Since bs4 4.7, full CSS selector support ships via the bundled soupsieve library. select returns a list, select_one returns the first match or None:

books = soup.select("article.product_pod")
first_price = soup.select_one("article.product_pod p.price_color")
print(first_price.get_text(strip=True))  # £51.77

Anything you can write in a browser dev-tools search works here: descendant combinators (section ol li), attribute selectors (a[href^="catalogue/"]), pseudo-classes (li:nth-of-type(3)). find_all and select overlap almost completely — selectors are terser for nested paths, find_all is easier when you filter by function or regex. Pick one style per project so the code stays readable.

Attributes and text

Tags behave like dictionaries for attributes. Square brackets raise KeyError on a missing attribute; .get() returns None instead:

link = first.h3.a           # dot access walks to the first matching child
print(link["href"])         # catalogue/a-light-in-the-attic_1000/index.html
print(link["title"])        # A Light in the Attic
print(link.get("rel"))      # None — no such attribute, no crash

Note the page truncates the visible link text ("A Light in the ...") but stores the full title in the title attribute — real sites hide data in attributes constantly, so check them before you fight with visible text. For text, get_text(strip=True) collapses whitespace and is almost always what you want; plain .get_text() preserves the raw whitespace between nested tags.

Navigating up and sideways

Sometimes the element you can select is not the element you need — you match a title, but want the price sitting next to it. Every tag has .parent, .children, and sibling hooks:

title_link = soup.find("a", title="A Light in the Attic")
pod = title_link.find_parent("article")            # climb to the container
price = pod.select_one("p.price_color")            # search back down
availability = pod.select_one("p.instock.availability")
print(price.get_text(strip=True), "|", availability.get_text(strip=True))
# £51.77 | In stock

find_parent climbs until a match; find_next_sibling / find_previous_sibling move sideways. The reliable pattern for listing pages is exactly what this snippet does: locate the repeating container, then search within it — never scrape prices and titles in two separate global queries and hope the lists line up.

Real task: product listing to CSV

Here is a complete, runnable script that scrapes one listing page into a list of dicts and writes CSV. The only awkward field is the star rating, which the site encodes as a class name (star-rating Three):

import csv
import requests
from bs4 import BeautifulSoup

RATINGS = {"One": 1, "Two": 2, "Three": 3, "Four": 4, "Five": 5}

def parse_book(pod):
    """Turn one article.product_pod into a dict."""
    link = pod.h3.a
    rating_classes = pod.select_one("p.star-rating")["class"]  # ["star-rating", "Three"]
    return {
        "title": link["title"],
        "price": pod.select_one("p.price_color").get_text(strip=True),
        "rating": RATINGS.get(rating_classes[-1]),
        "in_stock": "In stock" in pod.select_one("p.availability").get_text(),
        "url": link["href"],
    }

resp = requests.get("https://books.toscrape.com/")
resp.raise_for_status()
soup = BeautifulSoup(resp.content, "lxml")

books = [parse_book(pod) for pod in soup.select("article.product_pod")]

with open("books.csv", "w", newline="", encoding="utf-8") as f:
    writer = csv.DictWriter(f, fieldnames=books[0].keys())
    writer.writeheader()
    writer.writerows(books)

print(f"Wrote {len(books)} books to books.csv")

Two habits in that script are worth stealing: a parse_book function that owns one container (so a layout change breaks one function, not your whole script), and building dicts before touching the CSV writer, so the data structure is testable on its own. If your destination is a spreadsheet rather than a CSV file, the same dict-list feeds straight into the workflow from our scrape a website to Excel guide.

Pagination and politeness

The sandbox has 50 pages, each linking the next via li.next a. Follow that link until it disappears — and be polite about it: identify yourself with a User-Agent header instead of the default python-requests/x.y, and sleep between requests so you are not hammering the server. Most sites enforce rate limits, and the sites that do not still deserve the courtesy:

import csv
import time
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "books-tutorial/1.0 (learning BeautifulSoup)"}

def scrape_all(start_url):
    url, books = start_url, []
    while url:
        resp = requests.get(url, headers=HEADERS, timeout=10)
        resp.raise_for_status()
        soup = BeautifulSoup(resp.content, "lxml")

        books += [parse_book(pod) for pod in soup.select("article.product_pod")]

        next_link = soup.select_one("li.next a")           # None on the last page
        url = urljoin(url, next_link["href"]) if next_link else None
        time.sleep(1)                                       # be polite
    return books

books = scrape_all("https://books.toscrape.com/")
print(f"Scraped {len(books)} books")   # 1000

urljoin matters more than it looks: the next-page href is relative (catalogue/page-2.html), and — a genuine quirk of this site — the path prefix changes between page 1 and page 2. Resolving each link against the URL you actually fetched handles that automatically; string concatenation would 404 on page 3.

Common errors and how to fix them

Four failures account for most BeautifulSoup debugging time:

The second row deserves emphasis because it is the one that stumps everyone eventually: your browser is not your scraper. Dev tools show the DOM after JavaScript has run; requests sees only the initial HTML response. When they disagree, no selector will save you — which brings us to the honest part.

When BeautifulSoup isn't enough

BeautifulSoup has two hard limits, and knowing them saves you days.

JavaScript-rendered pages. If a site builds its content client-side — infinite scroll feeds, React storefronts, dashboards — the HTML that requests downloads is a nearly empty shell, and BeautifulSoup can only parse what is actually there. You need a headless browser that executes the JavaScript first. Our Playwright web scraping tutorial covers the modern way to do that (and Selenium the older one); a common hybrid is letting the browser render, then handing page.content() to BeautifulSoup for the parsing you already know.

Anti-bot systems at scale. The sandbox never blocks you. Amazon, Google, and most commercial sites will — with CAPTCHAs, TLS fingerprinting, and IP bans — once your traffic stops looking like a person. Solving that DIY means proxy pools, fingerprint rotation, and retry logic: a second engineering project bolted onto your parser. The alternative is a structured scraping API that handles the blocking and returns JSON, so there is no HTML to parse at all:

import requests

resp = requests.get(
    "https://api.crawlora.net/api/v1/amazon/search",
    params={"k": "mechanical keyboard"},
    headers={"x-api-key": "YOUR_API_KEY"},
)
for item in resp.json()["data"]:
    print(item["asin"], item["price"], item["title"])

That is Crawlora's Amazon search endpoint — one of a catalog of per-platform endpoints documented in the API docs. Billing is pay-on-success: you are charged only for successful 2xx responses, and the free tier is 2,000 credits/month, no card. For a fuller framework on when to parse HTML yourself versus call an API, see web scraping vs API; pricing has the numbers.

None of that diminishes what you built in this tutorial. For static, public, reasonably sized sites — which is a lot of the web — requests plus BeautifulSoup remains the simplest tool that works, and every skill above transfers directly to the browser-rendered and API-assisted setups you graduate into.

Skip the parsing on the hard sites

Documented per-platform endpoints, normalized JSON, managed proxies and retries. Pay only for successful 2xx responses — 2,000 credits/month, no card.

Try the PlaygroundSee the Web Scraping API

Related reading: the pillar web scraping with Python guide · web scraping with Playwright for JavaScript-heavy sites · Selenium web scraping · scrape a website to Excel · what is HTML parsing.