Selenium drives a real browser from Python, which makes it the tool you reach for when requests gets back an empty shell and the data you need only appears after JavaScript runs. This guide covers Selenium web scraping the way it actually works in 2026: Selenium 4 with built-in driver management (no chromedriver downloads), the new headless mode, By. locators, and explicit waits that replace the time.sleep guesswork. Every example is runnable as-is.
If you are new to scraping in general, start with our pillar guide to web scraping with Python — this post assumes you know what a CSS selector is and picks up where static HTML ends and a headless browser begins.
When you actually need Selenium
Selenium's home turf is browser automation and testing, and it is genuinely good at it: it speaks the W3C WebDriver standard, drives real Chrome/Firefox/Edge, and has two decades of battle scars. For scraping, that power is only worth paying for in specific situations:
| Situation | Right tool |
|---|---|
Data is in the initial HTML (view-source shows it) | requests + BeautifulSoup — see our BeautifulSoup tutorial |
| Data arrives from a JSON API the page calls | requests hitting that API directly (check the Network tab first) |
| Data renders only after JavaScript runs | Selenium or Playwright |
| Data appears only after clicking, scrolling, or typing | Selenium or Playwright |
| You need screenshots or PDF rendering | Selenium or Playwright |
The first two rows matter more than they look. A real browser spends seconds and hundreds of megabytes per page; a plain HTTP request spends milliseconds. Before writing any Selenium, open DevTools, watch the Network tab, and check whether the "JavaScript-only" data is actually sitting in a tidy JSON endpoint you can call directly. Roughly half the time, it is — and that is the whole web scraping with Python toolkit doing its job. Reach for Selenium when the answer is genuinely no.
Setup in 2026: one pip install, no driver downloads
This part got dramatically better. Since Selenium 4.6, Selenium Manager ships inside the package: it detects your installed browser, downloads the matching driver binary, and caches it. The old ritual — download chromedriver, match the version, fix your PATH, or bolt on webdriver-manager — is gone. If a tutorial tells you to install webdriver-manager, it is out of date.
python -m venv venv && source venv/bin/activate
pip install selenium
from selenium import webdriver
options = webdriver.ChromeOptions()
options.add_argument("--headless=new") # the current headless mode
options.add_argument("--window-size=1920,1080") # default is 800x600 — breaks responsive layouts
driver = webdriver.Chrome(options=options) # Selenium Manager fetches the driver on first run
driver.get("https://www.selenium.dev")
print(driver.title)
driver.quit()
Two flags worth explaining:
--headless=newis Chrome's modern headless mode — it shares far more code with headed Chrome than the legacy--headlessflag did, so pages render the same way they do on your screen. Use it for servers and CI; drop it while debugging so you can watch the browser work.--window-size=1920,1080matters because headless defaults to 800x600, and responsive sites quietly serve you the mobile layout — then your desktop selectors match nothing.
The core scraping workflow
The loop is always the same: navigate, wait for the content, find elements with By. locators, extract text and attributes into plain dicts. Here it is against quotes.toscrape.com/js, a practice site that renders everything with JavaScript (a plain requests.get returns zero quotes):
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
options = webdriver.ChromeOptions()
options.add_argument("--headless=new")
driver = webdriver.Chrome(options=options)
try:
driver.get("https://quotes.toscrape.com/js/")
# Block until at least one quote card exists in the DOM (max 10s)
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".quote"))
)
quotes = []
for card in driver.find_elements(By.CSS_SELECTOR, ".quote"):
quotes.append({
"text": card.find_element(By.CSS_SELECTOR, ".text").text,
"author": card.find_element(By.CSS_SELECTOR, ".author").text,
"tags": [t.text for t in card.find_elements(By.CSS_SELECTOR, ".tag")],
})
print(f"scraped {len(quotes)} quotes")
print(quotes[0])
finally:
driver.quit()
Notes on the modern API, since a lot of stale tutorials still circulate:
- Locators are
find_element(By.CSS_SELECTOR, "..."). The oldfind_element_by_css_selectorhelpers were removed in Selenium 4.3 — code using them raisesAttributeErrortoday.By.CSS_SELECTORcovers almost everything;By.IDandBy.XPATHfill the gaps (XPath is the one that can select by text content). find_elementvsfind_elements: singular raisesNoSuchElementExceptionwhen nothing matches; plural returns an empty list. Use the plural for card grids, then the singular scoped to each card — callingcard.find_element(...)searches only inside that card.- Attributes come from
element.get_attribute("href")(or"src","data-id", ...); visible text fromelement.text. try/finallywithdriver.quit()— otherwise every crashed run leaks a Chrome process, and ten crashed runs later your machine is unhappy.
Real interactions: click, scroll, type
The reason you accepted Selenium's overhead is interaction. Three patterns cover most sites.
Clicking "load more" until it disappears. Wait for clickability, click, repeat; when the button is gone, the loop ends:
from selenium.common.exceptions import TimeoutException
while True:
try:
button = WebDriverWait(driver, 5).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "button.load-more"))
)
button.click()
except TimeoutException:
break # no more button — everything is loaded
Scrolling an infinite feed. Scroll to the bottom, let new content land, and stop when the page height stops growing:
last_height = driver.execute_script("return document.body.scrollHeight")
while True:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
try:
WebDriverWait(driver, 5).until(
lambda d: d.execute_script("return document.body.scrollHeight") > last_height
)
last_height = driver.execute_script("return document.body.scrollHeight")
except TimeoutException:
break # height stopped growing — feed exhausted
Filling a search box. send_keys types like a user, Keys.RETURN submits:
from selenium.webdriver.common.keys import Keys
box = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.CSS_SELECTOR, "input[name='q']"))
)
box.clear()
box.send_keys("mechanical keyboard", Keys.RETURN)
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.CSS_SELECTOR, ".result"))
)
After any interaction, wait for its effect (new cards, changed height, a results container) before extracting — the click returning does not mean the page finished reacting.
Common failures and how to fix them
Every Selenium scraper eventually meets these three. The fix is almost never "add a sleep":
| Exception | What it means | Fix |
|---|---|---|
StaleElementReferenceException | You held a reference to an element, then the page re-rendered it (common after clicks and AJAX updates). The old reference points at a DOM node that no longer exists. | Re-find the element after any action that mutates the page. In loops, re-query find_elements each iteration instead of iterating a saved list. |
ElementNotInteractableException / ElementClickInterceptedException | The element exists but is hidden, zero-sized, or covered by something else (cookie banner, sticky header, modal). | Wait for element_to_be_clickable, dismiss the overlay first, or driver.execute_script("arguments[0].scrollIntoView();", el) before clicking. |
TimeoutException | Your WebDriverWait condition never became true. Either the selector is wrong, the content loads in an iframe, or the site served you a block page. | Print driver.title and save driver.page_source when it fires — you will often find a CAPTCHA or "verify you are human" page instead of the content. Check for iframes with driver.switch_to.frame(...). |
NoSuchElementException | find_element (singular) found nothing — often the same root causes as above, minus the waiting. | Wait first, then find; or use find_elements and handle the empty list. |
That third row deserves emphasis: a TimeoutException is frequently a detection problem wearing a selector-problem costume. Which brings us to the honest section.
The honest part: Selenium is slow and detectable
Two costs come with every real browser.
Speed and weight. A browser session costs seconds per page and roughly 300-500 MB of RAM, where an HTTP request costs milliseconds and almost nothing. At ten pages that is irrelevant; at a hundred thousand pages it is a fleet of servers and a queueing system you now maintain.
Detection. Out of the box, Selenium-driven Chrome announces itself: navigator.webdriver is true, and dozens of subtler signals — the TLS handshake, missing plugins, canvas and font quirks, your datacenter IP — feed browser fingerprinting systems that score you as a bot before your first selector runs. The symptoms are rate limiting, endless CAPTCHA walls, or silent garbage data.
Tools like undetected-chromedriver and SeleniumBase's UC mode patch the loudest signals, and they work — until the vendor ships a new check, which they do continuously, because detecting automation is their entire business. It is an arms race in which you are the hobbyist and they are the defense contractor. We wrote up the full landscape in scraping sites that block bots; the short version is that patching your browser is a treadmill, not a destination.
What about Playwright? Playwright is the younger alternative: auto-waiting locators baked in, faster and more stable in headless operation, async support, and a nicer API for scraping specifically. Selenium still wins when you need the broadest real-browser matrix, the WebDriver standard, a decade of Stack Overflow answers, or bindings beyond the big four languages — and it remains the default in QA teams for good reason. For a new scraping-only project, we would usually start with Playwright; for automation that must mirror real user coverage across browsers, Selenium. Neither is any less detectable, though — a fingerprinting system does not care which library moves the mouse.
Scaling out: when a structured API beats a browser fleet
Here is the decision that actually matters at scale. If your target is a large, well-defended platform — Amazon, Google, social networks — the Selenium path becomes: proxies, fingerprint patches, CAPTCHA handling, retry logic, HTML parsers that break on every redesign, and servers to run the browsers. Each piece is doable; together they are a part-time infrastructure job.
The alternative is a web scraping API that runs that infrastructure behind an endpoint and returns structured JSON. The same Amazon search that takes ~40 lines of Selenium, waits, and parsing is one HTTP request:
import requests
resp = requests.get(
"https://api.crawlora.net/api/v1/amazon/search",
params={"k": "mechanical keyboard"},
headers={"x-api-key": "YOUR_API_KEY"},
)
products = resp.json()["data"] # structured fields — no parser to maintain
No browser, no driver, no selectors to babysit — and Crawlora is pay-on-success, meaning you are charged only for successful 2xx responses, so a blocked or failed request costs you nothing. The sensible split most teams land on: Selenium for the sites where you need bespoke interaction or it is genuinely the right browser-automation tool, a structured API for the high-volume, heavily defended platforms. You can compare the costs yourself — browser-fleet math versus per-request pricing — and test endpoints in the playground before writing any code.
- Checked the Network tab first — is the data already in a JSON endpoint requests can hit?
- pip install selenium (4.6+) and let Selenium Manager handle drivers — no chromedriver downloads
- Running --headless=new with an explicit --window-size
- Every wait is WebDriverWait + expected_conditions — zero time.sleep calls
- driver.quit() in a finally block so crashed runs do not leak browsers
- On TimeoutException, dumped page_source to check for block pages before blaming the selector
- Honest about scale: bespoke interactions in Selenium, high-volume protected platforms via a structured API
Skip the browser fleet for the hard targets
Structured JSON from Amazon, Google, and 700+ endpoints — proxies, rendering, and retries handled, charged only on successful 2xx responses. 2,000 credits/month, no card.
Related reading
- Web scraping with Python: the complete guide — the pillar: requests, parsing, storage, and when to escalate to a browser
- BeautifulSoup tutorial — the faster path when the HTML is static
- Web scraping with Playwright — the modern browser-automation alternative compared above
- Scraping sites that block bots — the detection arms race in full
Frequently asked questions
Is Selenium good for web scraping?
Yes, when the data only appears after JavaScript runs or after an interaction like clicking or scrolling — Selenium drives a real browser, so it sees exactly what a user sees. For static HTML it is overkill: a plain HTTP request with requests and BeautifulSoup is 10-50x faster and far lighter.
Selenium vs BeautifulSoup — which should I use?
They solve different problems. BeautifulSoup parses HTML you already fetched; Selenium runs a browser to produce that HTML in the first place. If the data is in the page source or a JSON endpoint, use requests plus BeautifulSoup. If it renders client-side or requires clicks and scrolls, use Selenium — and you can still hand driver.page_source to BeautifulSoup for parsing.
Do I still need to download chromedriver for Selenium?
No. Since Selenium 4.6, the built-in Selenium Manager detects your installed browser and downloads the matching driver automatically. pip install selenium is the entire setup — separate chromedriver downloads and the webdriver-manager package are obsolete.
Can websites detect Selenium?
Yes. Selenium-driven Chrome sets navigator.webdriver to true, and browser fingerprinting systems read many subtler signals — TLS handshake, missing plugins, canvas quirks, datacenter IPs. Patches like undetected-chromedriver hide the loudest tells, but anti-bot vendors update their checks continuously, so evasion is an ongoing arms race rather than a one-time fix.
How do I wait for a page to load in Selenium?
Use WebDriverWait with expected_conditions instead of time.sleep. WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".quote"))) polls until the element exists and returns immediately when it does — faster on fast pages, tolerant on slow ones. Use element_to_be_clickable before clicks and visibility_of_element_located when you need rendered content.
Is Selenium or Playwright better for web scraping?
For a new scraping-only project, Playwright usually wins: auto-waiting locators, faster headless runs, and async support. Selenium wins on the broadest real-browser coverage, the W3C WebDriver standard, and its ecosystem, which is why QA teams still default to it. Neither is less detectable — anti-bot systems fingerprint the browser, not the library driving it.
Why is my Selenium scraper getting TimeoutException?
A TimeoutException means your wait condition never became true. Common causes: a wrong selector, content inside an iframe you have not switched to, or — very often — the site served a block page or CAPTCHA instead of the content. Dump driver.page_source when it fires; if you see an anti-bot page, the problem is detection, not your selector.