HTML parsing is the step that turns a downloaded page's raw markup into a structured tree of elements your code can query — so a scraper can pull out titles, prices, or links by their position and attributes instead of string-matching against raw text.
An HTML parser reads markup and builds a tree of nested elements — the same document-object-model shape a browser builds before rendering. Crucially, HTML parsers are forgiving by design: real-world pages are full of unclosed tags and invalid nesting, and the HTML5 parsing algorithm defines exactly how to recover, so BeautifulSoup, lxml, and a browser all agree on the resulting tree.
In Python, BeautifulSoup is the common friendly interface with a choice of underlying engines (html.parser from the standard library, or the faster C-based lxml). For large-scale extraction, using lxml directly trades convenience for speed.
You query the tree with CSS selectors (div.price > span, the same syntax as stylesheets) or XPath (//div[@class='price']/span), which adds axes CSS lacks, like selecting by text content or walking up to a parent. Most scrapers use CSS selectors for readability and reach for XPath on the awkward cases.
Parsing is also where scrapers die: selectors are coupled to the page's structure, so a site redesign — or just a rotated CSS class name from a build tool — silently breaks extraction. That maintenance tax, multiplied by every site you scrape, is the recurring cost of DIY parsing; it's also why JavaScript-rendered pages need a headless browser first, since the interesting elements aren't in the raw HTML at all.
import requests
from bs4 import BeautifulSoup
html = requests.get("https://example.com/products", timeout=15).text
soup = BeautifulSoup(html, "lxml")
for card in soup.select("div.product-card"):
name = card.select_one("h2.title").get_text(strip=True)
price = card.select_one("span.price").get_text(strip=True)
print(name, price)How Crawlora handles this
Crawlora's structured endpoints move parsing to the provider side: platform routes return normalized JSON fields that stay stable when the underlying site redesigns, because maintaining the extraction layer is Crawlora's job, not a selector file in your repo.
Related reading
Glossary
FAQ
BeautifulSoup for ergonomics — it's the friendlier API and can use lxml as its engine (BeautifulSoup(html, 'lxml')), which is the usual best-of-both setup. Drop to raw lxml when parsing speed across large page volumes actually matters.
CSS selectors for the common cases — they're shorter and more readable. XPath when you need what CSS can't do: matching on text content, selecting parents or preceding siblings, or complex positional logic.
For a quick one-off match on a known page, it works. As a parsing strategy it's famously fragile — HTML's nesting and real-world malformation defeat regex patterns in ways a real parser's error recovery handles for free. Use a parser; they're fast and tolerant.
Browse Crawlora APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.