Web Scraping with Scrapling in 2026

Most Python scrapers break twice: once when a target ships an anti-bot check your requests call can't get past, and again months later when a frontend redesign moves the CSS class your selector depended on. Scrapling is a scraping framework built around fixing both — three fetcher classes that scale from plain HTTP up to a full stealth browser, and a parser that can relocate an element by similarity instead of by the selector you originally wrote for it. This guide installs it, runs all three fetchers, and shows the one detection layer its own benchmark data says it still doesn't beat.

What actually differentiates Scrapling: adaptive parsing

Every scraping library eventually hits the same failure mode: a site redesign renames a CSS class, and every selector written against the old markup silently returns nothing. Scrapling's parser has a specific answer for this. Call a selector once with auto_save=True and it fingerprints the matched element — tag name, text, attribute names and values, sibling tag names, and the surrounding path — and stores that fingerprint. Later, pass adaptive=True to the same selector and Scrapling searches the current page for the element with the highest similarity score to the saved fingerprint, not the literal selector string. If the match falls below a similarity threshold, it logs the best score it did find instead of silently returning nothing, so a genuinely broken selector still tells you something.

This is a real, working feature rather than a vague pitch: it's what the project's own performance benchmarks measure against a competing library (AutoScraper), and it's the reason Scrapling bills itself as "adaptive" rather than just another parser wrapper.

Install it

pip install scrapling

That bare install is the parser engine only — importing anything from scrapling.fetchers or scrapling.spiders raises ModuleNotFoundError until you pull in the fetcher extras and their browser binaries:

pip install "scrapling[fetchers]"
scrapling install

scrapling install downloads the bundled Chromium/Firefox builds plus fingerprint-manipulation dependencies the stealth fetcher needs. Extra installs cover the AI-facing pieces if you want them: scrapling[ai] for the MCP server, scrapling[rag] for Markdown conversion, scrapling[shell] for the interactive CLI, or scrapling[all] for everything at once.

Pick a fetcher: HTTP, stealth, or a plain browser

Scrapling's three fetcher classes share one calling convention, so switching between them when a target turns out to need more than you assumed is a class swap:

from scrapling.fetchers import Fetcher, StealthyFetcher, DynamicFetcher

# Fast HTTP request with TLS/browser impersonation, no browser process at all
page = Fetcher.get("https://example.com", impersonate="chrome")

# Full Chromium/Firefox automation via Playwright, no anti-bot handling
page = DynamicFetcher.fetch("https://example.com")

# Stealth browser: auto-solves Cloudflare Turnstile, spoofs canvas/WebRTC/CDP fingerprints
page = StealthyFetcher.fetch("https://example.com", solve_cloudflare=True, headless=True)

# Every fetcher returns the same Response/Selector object
title = page.css("title::text").get()

Fetcher is built on curl_cffi, so impersonate="chrome" (or "firefox102", "safari15_5", and a growing list of pinned versions) replays a real browser's TLS handshake — the same JA3-fingerprint problem our curl_cffi post covers, without an extra dependency. DynamicFetcher is plain Playwright automation with no stealth handling. StealthyFetcher is the one built for hard targets: pass solve_cloudflare=True and it detects and solves Cloudflare's managed, interactive, and invisible Turnstile challenges before handing you the response, on top of canvas noise, WebRTC-leak blocking, and CDP-runtime-leak mitigation it applies by default.

Scrape data that survives a redesign

Here's the adaptive-parsing loop end to end — save a selector's fingerprint on a first pass, then rely on similarity matching instead of the literal selector on later runs:

from scrapling.fetchers import Fetcher

page = Fetcher.get("https://quotes.toscrape.com/")

# First run: save this element's fingerprint alongside the normal result
quotes = page.css(".quote", auto_save=True)

# ... months later, after the site's markup has changed underneath you ...

# adaptive=True relocates by similarity score instead of trusting the old class name
quotes = page.css(".quote", adaptive=True)
for quote in quotes:
    print(quote.css(".text::text").get())

The same object also gives you BeautifulSoup-style traversal (find_all, find_by_text) and structural helpers most parsers don't bother with — first_quote.find_similar() to locate elements structurally like the one you already found, and .parent / .next_sibling for moving around the DOM without re-selecting from the document root each time.

Scale up: the Spider framework

A single fetch call doesn't need orchestration, but a multi-thousand-URL crawl does — request queueing, concurrency limits, and a way to resume after it dies halfway through. Scrapling's Spider class covers that with a Scrapy-like API:

from scrapling.spiders import Spider, Response

class QuotesSpider(Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/"]
    concurrent_requests = 10

    async def parse(self, response: Response):
        for quote in response.css(".quote"):
            yield {
                "text": quote.css(".text::text").get(),
                "author": quote.css(".author::text").get(),
            }
        next_page = response.css(".next a")
        if next_page:
            yield response.follow(next_page[0].attrib["href"])

result = QuotesSpider().start()
result.items.to_json("quotes.json")

AutoThrottle (autothrottle_enabled = True) measures how fast each domain actually responds and tunes the per-domain delay itself instead of you guessing a fixed number — it doubles the delay when a site starts blocking or rate-limiting you, honors a Retry-After header exactly, and backs off from there once the site stops complaining. Long crawls checkpoint themselves: run QuotesSpider(crawldir="./crawl_data").start(), hit Ctrl+C for a graceful pause, and restarting with the same crawldir resumes from where it stopped instead of re-crawling from zero.

• CrawlSpider and CrawlRule: define link-following rules once (an allow/deny LinkExtractor pattern per rule) instead of hand-writing enqueue logic in every handler.
• SitemapSpider: seed a crawl straight from a sitemap or robots.txt, gzip-compressed sitemaps included.
• ShopifySpider: subclass it, set target_website, and get every product from any Shopify store's own JSON API — no HTML parsing at all.
• XMLFeedSpider / CSVFeedSpider: iterate an RSS/Atom/product XML feed or a CSV file's rows as dictionaries, both with automatic gzip handling.
• SiteToMarkdownSpider: crawl an entire site into a folder of clean Markdown files for a RAG pipeline, one file per page.

What Scrapling doesn't fix

Scrapling's own claims are specific about what StealthyFetcher handles: Cloudflare Turnstile/Interstitial solving, WebRTC and CDP-leak mitigation, canvas noise, and headless-mode method patching. What it doesn't claim is that every fingerprint surface comes back clean — and Crawlora's own stealth-browser engine benchmark is directly relevant here, because StealthyFetcher's engine is the same one we tested. Scrapling ran on Camoufox before version 0.3.13; its docs say the switch to patchright (a JS-patched Chromium) happened "for many reasons," and patchright is one of the four engines our benchmark ran against a real detector and a live Cloudflare target. The result: patchright passed the core 8-test detection suite but failed the WebGL check — the one item that only Camoufox passed among everything we tested. StealthyFetcher leaves WebGL enabled by default (its docs warn that disabling it is a worse idea, since WAFs increasingly check whether WebGL exists at all) — but enabled and convincing aren't the same claim, and our data says the vendor/renderer string patchright reports doesn't clear that bar.

The other layer no fetcher class touches is IP reputation. Scrapling's built-in ProxyRotator (cyclic or custom rotation across all session types) spreads your requests across whatever proxies you already have — it does not source clean ones. A perfectly solved Turnstile challenge from a datacenter IP with no browsing history is still a datacenter IP to a vendor that scores IP reputation independently of the browser fingerprint. See proxies for web scraping, explained for what actually moves that signal.

Scrapling vs. the alternatives

• One-off script against a static page, no anti-bot: Fetcher (or plain requests + BeautifulSoup — see our beautifulsoup tutorial) is lighter for something you'll run once.
• Selectors keep breaking after site redesigns: Scrapling's adaptive parsing (auto_save / adaptive) is the actual reason to reach for it over a plain parser — no other library in this series does element relocation by similarity.
• Multi-thousand-URL crawl needing queueing, retries, and storage across mixed rendering needs: both Scrapling's Spider and Crawlee's AdaptivePlaywrightCrawler solve this; Crawlee's edge is its request-queue/dataset abstractions, Scrapling's is the ready-made platform templates (ShopifySpider, SitemapSpider) plus the adaptive parser underneath.
• Deep existing Scrapy investment: Scrapling ships a scrapling_response decorator that lets you parse with Scrapling inside a Scrapy callback with no rewrite — you don't have to choose one framework outright.
• Target fingerprints WebGL specifically, not just Cloudflare's managed challenge: swap to Camoufox rather than trusting StealthyFetcher's patchright engine alone, per the benchmark note above.

When you'd rather not maintain any of this

Fetchers, proxy rotation, and a stealth engine that needs re-checking every time a target changes its detection stack — that's an ongoing maintenance job once you're past a handful of targets. Crawlora exposes the same kind of data as structured endpoints instead:

import requests

r = requests.get(
    "https://api.crawlora.net/api/v1/amazon/search",
    params={"k": "mechanical keyboard"},
    headers={"x-api-key": "YOUR_API_KEY"},
)
data = r.json()
for item in data["data"]:
    print(item["asin"], item["list_price"], item["link"])

Proxy rotation, fingerprint upkeep, and Cloudflare/anti-bot handling happen behind the endpoint, and you get normalized JSON back instead of a DOM you keep re-selecting after every redesign — Scrapling's adaptive parser helps with that problem, an endpoint contract sidesteps it. There's an official Python SDK, and billing is pay-on-success: a blocked or failed fetch costs nothing (pricing). Web scraping vs API walks through when DIY is still the right call.

Wrap-up

Scrapling's three fetcher classes cover the same escalation path most scrapers eventually need — cheap HTTP, a plain browser, then a stealth engine — behind one calling convention, and its adaptive parser is a genuine answer to the "selector broke after a redesign" problem rather than a marketing line. pip install "scrapling[fetchers]" && scrapling install, pick a fetcher, auto_save=True your selectors on the first pass: that covers the common case. Just remember StealthyFetcher's patchright engine is the one our own benchmark measured failing the WebGL check, and no fetcher here sources clean IPs for you — pair it with Camoufox or a managed proxy for whichever detection layer you're actually up against. Try the underlying endpoints in the playground with no code at all, or browse the docs for the full catalog.

Skip the fetcher-and-fingerprint stack on hard targets

Structured endpoints, managed proxies and rendering, pay-on-success billing. 2,000 free credits/month, no card.

Try the PlaygroundSee the Web Scraping API

Related reading

• Web scraping with Camoufox: the engine that passes the WebGL check our benchmark says patchright (StealthyFetcher's engine) fails
• Web scraping with curl_cffi and curl-impersonate: the same TLS-impersonation approach behind Scrapling's plain Fetcher class
• Web scraping with Crawlee: the other orchestration-first framework, and how its adaptive HTTP-vs-browser choice compares to Scrapling's adaptive parsing
• Our stealth-browser engine benchmark: the patchright/Camoufox/zendriver test this post's WebGL claim is sourced from
• BeautifulSoup tutorial: the lighter option for a genuine one-off static-page script
• Proxies for web scraping, explained: the IP-reputation layer no fetcher class here touches