Scrapy Tutorial - Build Your First Python Spider in 2026

Scrapy is what you graduate to when requests scripts stop scaling. It is a full crawling framework for Python: you write a small spider class that says what to extract and which links to follow, and Scrapy handles scheduling, concurrency, retries, deduplication, and export. This tutorial takes you from scrapy startproject to a two-level crawl with production-safe settings, using current Scrapy 2.x syntax throughout.

Why Scrapy instead of requests + BeautifulSoup?

If you have written a few scrapers with requests and BeautifulSoup — the stack covered in our Python web scraping guide and the BeautifulSoup tutorial — you already know the pattern: fetch a page, parse it, loop. That works brilliantly for one page or one list. It falls apart when the job becomes crawling: visiting many pages, discovering links as you go, and not fetching the same URL twice. That discovery-and-scheduling loop is what web crawling actually is, and it is precisely the part Scrapy owns:

• A scheduler and request queue. You yield requests; Scrapy decides when to send them, how many to run concurrently, and retries the failures.
• A URL frontier with deduplication. Follow the same link from five pages and Scrapy fetches it once.
• Item pipelines. Every scraped record flows through a chain of small classes that clean, validate, or store it — separate from parsing logic.
• Feed exports. JSON, JSON Lines, and CSV output from a command-line flag, no serialization code.

To be clear about the trade: BeautifulSoup is a parsing library and a great one; Scrapy has its own selector engine and is a heavier dependency with a steeper learning curve. For a single page, Scrapy is overkill. For five hundred pages behind pagination, it replaces a hundred lines of queue-and-retry plumbing you would otherwise write badly once and debug forever.

Project setup: what startproject actually creates

Install Scrapy (a virtual environment is strongly recommended) and generate a project:

pip install scrapy
scrapy startproject quotes_crawler
cd quotes_crawler

The generated tree looks like this, and every file has a job:

quotes_crawler/
├── scrapy.cfg            # deploy/config entry point — tells Scrapy where settings live
└── quotes_crawler/
    ├── items.py          # optional typed containers for scraped data
    ├── middlewares.py    # hooks into the request/response cycle (headers, proxies)
    ├── pipelines.py      # post-processing for scraped items (clean, validate, store)
    ├── settings.py       # project-wide knobs: concurrency, delays, throttling
    └── spiders/          # your spider classes live here — one class per crawl target

You can ignore items.py and middlewares.py for now — plain dicts and default middleware are fine for a first project. The two files you will actually touch today are a new spider in spiders/ and, later, settings.py and pipelines.py.

Scaffold a spider with genspider:

scrapy genspider quotes quotes.toscrape.com

Your first spider

We will scrape quotes.toscrape.com, the sandbox site the official Scrapy tutorial uses. Replace the generated spiders/quotes.py with:

import scrapy


class QuotesSpider(scrapy.Spider):
    name = "quotes"
    start_urls = ["https://quotes.toscrape.com/page/1/"]

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }

Three ideas carry the whole framework:

1. start_urls is where the crawl begins. Scrapy fetches each URL and hands the response to parse(). (Since Scrapy 2.13 you can instead define async def start(self) for dynamic start requests; start_urls remains the idiomatic simple case, but note that the older start_requests() method is deprecated.)
2. response.css() selects elements with CSS selectors — ::text extracts text nodes, ::attr(href) extracts an attribute, .get() returns the first match or None, .getall() returns every match as a list. response.xpath() is there when CSS cannot express what you need; under the hood CSS selectors are translated to XPath anyway. This is the same HTML parsing problem BeautifulSoup solves, just with Scrapy's selector API.
3. yield a dict and Scrapy treats it as a scraped item — no return values, no accumulating lists.

Run it and export straight to JSON:

scrapy crawl quotes -O quotes.json

The -O flag overwrites the output file each run; lowercase -o appends instead, which is mainly useful with the JSON Lines format (-o quotes.jsonl). Open quotes.json and you will find ten structured records — one page's worth, because nothing follows links yet.

Following links: pagination and two-level crawls

Crawling starts when parse() yields requests alongside items. The quotes site has a Next button, and response.follow() handles it in two lines — it accepts relative URLs directly, so no manual urljoin:

    def parse(self, response):
        for quote in response.css("div.quote"):
            yield {
                "text": quote.css("span.text::text").get(),
                "author": quote.css("small.author::text").get(),
                "tags": quote.css("div.tags a.tag::text").getall(),
            }

        next_page = response.css("li.next a::attr(href)").get()
        if next_page is not None:
            yield response.follow(next_page, callback=self.parse)

Run scrapy crawl quotes -O quotes.json again and you get all ten pages — roughly a hundred quotes. The scheduler queued each Next link, fetched it, and routed the response back through parse(). Duplicate URLs would have been filtered automatically.

The pattern that unlocks real projects is the two-level crawl: a listing page yields links to detail pages, and a second callback parses each detail page. Here is a spider that walks the quote listings but extracts data from each author's biography page:

import scrapy


class AuthorSpider(scrapy.Spider):
    name = "authors"
    start_urls = ["https://quotes.toscrape.com/"]

    def parse(self, response):
        # Level 1: listing pages — follow every author link
        author_links = response.css(".author + a")
        yield from response.follow_all(author_links, callback=self.parse_author)

        # ...and keep paginating the listing itself
        yield from response.follow_all(css="li.next a", callback=self.parse)

    def parse_author(self, response):
        # Level 2: detail pages
        yield {
            "name": response.css("h3.author-title::text").get().strip(),
            "born": response.css(".author-born-date::text").get(),
            "bio": response.css(".author-description::text").get().strip()[:200],
        }

response.follow_all() yields a request per matched link, and each callback only worries about its own page type. Even though many quotes point to the same author, each biography is fetched once — deduplication again. The same shape scales to an e-commerce catalog (category pages to product pages, as on books.toscrape.com) or a job board (search results to postings).

For crawls where "follow every link matching this pattern" is the whole strategy, Scrapy ships CrawlSpider, a subclass where you declare Rule objects with a LinkExtractor instead of writing follow logic by hand. Start with plain Spider — you understand exactly what it fetches — and reach for CrawlSpider when your follow rules become repetitive.

Production settings: be fast, but not rude

The defaults are tuned for the sandbox, not for someone else's server. Scrapy will happily run 16 concurrent requests with zero delay, which is a good way to trip rate limiting or hurt a small site. Before any real crawl, open settings.py:

# settings.py — a polite production baseline

# Respect robots.txt (on by default in new projects — leave it on)
ROBOTSTXT_OBEY = True

# Identify yourself honestly
USER_AGENT = "quotes_crawler (+https://yourdomain.example/contact)"

# Cap concurrency per site and add a base delay
CONCURRENT_REQUESTS_PER_DOMAIN = 4
DOWNLOAD_DELAY = 1.0  # seconds; Scrapy randomizes it a bit by default

# Let Scrapy adapt speed to the server's actual latency
AUTOTHROTTLE_ENABLED = True
AUTOTHROTTLE_START_DELAY = 5.0
AUTOTHROTTLE_MAX_DELAY = 60.0
AUTOTHROTTLE_TARGET_CONCURRENCY = 2.0

AutoThrottle is the setting most tutorials skip and the one most worth knowing. It measures response latency and adjusts the delay dynamically — when the server slows down or errors, Scrapy backs off toward AUTOTHROTTLE_MAX_DELAY; when it is healthy, Scrapy speeds up toward your target concurrency. DOWNLOAD_DELAY acts as the floor it will never go below, and CONCURRENT_REQUESTS_PER_DOMAIN stays a hard cap. Set AUTOTHROTTLE_DEBUG = True temporarily to watch the throttle decisions in the log.

ROBOTSTXT_OBEY deserves a sentence: it makes Scrapy fetch and honor each site's robots.txt rules before crawling. It defaults to on in generated projects. Keep it on.

A minimal item pipeline

Pipelines keep cleanup out of your parse code. Each scraped item passes through process_item() in every enabled pipeline class:

# pipelines.py
from scrapy.exceptions import DropItem


class CleanQuotePipeline:
    def process_item(self, item, spider):
        if not item.get("text"):
            raise DropItem("missing quote text")
        item["text"] = item["text"].strip("“” ")  # strip curly quotes
        item["author"] = item["author"].strip()
        return item

Enable it in settings.py — the number is an ordering priority, lower runs first:

ITEM_PIPELINES = {
    "quotes_crawler.pipelines.CleanQuotePipeline": 300,
}

The same shape handles validation, deduplication by field, or writing to a database (pipelines also get open_spider and close_spider hooks for connections). Parsing stays about selectors; pipelines own data quality.

• Spider yields dicts from parse() and follows links with response.follow / follow_all
• Selectors were tested in scrapy shell before going into the spider
• ROBOTSTXT_OBEY is on and the USER_AGENT identifies your crawler
• AUTOTHROTTLE_ENABLED is true, with DOWNLOAD_DELAY as a floor
• Cleanup and validation live in a pipeline, not in parse()
• Output verified with scrapy crawl name -O output.json

Where Scrapy stops: JavaScript and anti-bot systems

Scrapy is excellent at what it does — it is hard to beat for crawling static HTML at scale. But two walls are worth knowing about before you hit them.

JavaScript-rendered pages. Scrapy's downloader fetches raw HTML; it does not run a browser engine. If the data appears only after client-side rendering, your selectors will match nothing. The standard fix is the scrapy-playwright plugin, which routes requests through a headless browser while keeping Scrapy's scheduling — or a standalone browser stack, as covered in our Playwright scraping guide. Either way, rendering costs real CPU and memory per page.

Anti-bot systems at scale. Large platforms fingerprint TLS handshakes, score IP reputation, and issue JavaScript challenges. A well-throttled Scrapy spider from one IP will still get blocked there, and stacking proxy middleware and header tricks becomes a second engineering project — we cover that arms race in scraping sites that block bots.

For those targets, the pragmatic move is to keep Scrapy for the sites it handles and call a scraping API for the ones it cannot. Crawlora exposes hardened platforms as documented endpoints returning structured JSON — here is an Amazon search, no proxies or parsers on your side:

import requests

resp = requests.get(
    "https://api.crawlora.net/api/v1/amazon/search",
    params={"k": "mechanical keyboard"},
    headers={"x-api-key": "YOUR_API_KEY"},
)
results = resp.json()["data"]  # structured listings, not HTML

You can even call it from inside a pipeline or spider to enrich crawled records. Billing is pay-on-success — you are charged only for successful 2xx responses — and the free tier is 2,000 credits/month, no card. Browse the endpoint catalog in the docs or check pricing for volume math.

Wrap-up

You now have the complete Scrapy loop: scaffold with startproject and genspider, extract with response.css, yield dicts, follow links with response.follow, export with -O, throttle with AutoThrottle, and clean with pipelines. That covers a very large share of real crawling work. When a target renders in JavaScript or fights back with anti-bot systems, bolt on scrapy-playwright or hand that one site to an API — and let your spider keep doing what it is genuinely great at.

Hitting sites your spider can't crawl?

Keep Scrapy for the easy targets and call documented endpoints for the hard ones — structured JSON, managed proxies and retries. 2,000 free credits a month, no card.

Try the PlaygroundSee the Web Scraping API

Related reading: the Python web scraping guide for the full landscape, the BeautifulSoup tutorial if you want the lighter stack, and web scraping with Playwright for JavaScript-heavy targets.