93 step-by-step guides, grouped by what you are collecting. Every one shows the DIY Python route with requests and BeautifulSoup, why that route breaks in production, and the structured API call that returns the same data as JSON — plus the legal and anti-bot reality for that specific site.
Engine results, demand curves, local listings, and the academic index.
Independent-index web results without Google's bot defenses in the way.
Web, image, news, and video results — since Microsoft retired its own Search API.
Interest over time, rising and top queries — the official API is still allowlisted.
Places, categories, hours, and coordinates for local-business datasets.
Papers, citation counts, and authors from the literature index.
Catalogs, prices, sellers, and variants across the major marketplaces.
Product detail, search results, and pricing — the canonical scraping target.
Listings, sellers, and feedback across auction and fixed-price inventory.
Any storefront's full catalog, including headless stores, with facet and filter data.
Shopify's consumer marketplace — cross-store products, shops, and reviews.
Handmade and vintage marketplace — listing, shop, and review data.
Categories, search, product detail, reviews, and Q&A.
Search, product detail, availability, reviews, and warehouses.
Nearby stores, live product pricing, and search suggestions.
Search, product detail, and reviews — no self-serve catalog API.
Sneaker and streetwear resale — live market bids, asks, and releases.
Fashion resale listings, closets, brands, and trends.
European second-hand fashion — catalog, items, brands, and members.
Live-video auction catalog — browse, categories, and per-show listings.
US and Japan resale marketplace — search, item detail, and taxonomy.
New-retail fashion across 25 European markets — search, product, variants.
Search, full per-colorway product detail, and the Men/Women/Kids/Jordan category tree.
Search, category listings, and per-color product detail with per-size stock.
Search, category listings, and per-color pricing with per-size stock and reviews.
Category browsing, product detail, pricing, stock, and rating breakdowns.
Product search and full detail with pricing, variations, and merchant data.
Product search, pricing, ratings, featured reviews, and color variants.
Local classifieds search — no official API, Meta's strictest terms yet.
Restaurant search, menus, reviews, and reservation availability from delivery and booking platforms.
Vehicle listings, pricing, and dealer inventory across the major used-car sites.
Ratings and review text, wherever customers actually leave it.
Business ratings and review text past the Places API's five-review cap.
Business profiles, category rankings, and full review streams.
Mobile ratings and reviews from both stores via the iTunes lookup pattern.
Hotels, attractions, and traveler reviews with locale-aware pagination.
Business search, profiles, reviews, menus, and photos past the Fusion API's free-tier cap.
Listings, availability, and pricing for places people stay or buy.
Hotel search, detail, reviews, flights, and attractions from the largest OTA.
Search, room detail, reviews, and the calendar behind nightly availability.
Property records, search, and estimates from the largest US portal.
The portal-by-portal overview: which sites allow what, and how to normalize.
Listings, estimates, and market trends — one of the harder anti-bot targets.
Hotels, homes, activities, and flights from the major Asia-based OTA.
Hotel search, detail, reviews, activities, and flights.
Hotel search and detail from the largest Asia-based OTA.
Catalogs, charts, and where a title is actually available.
Tracks, artists, albums, playlists, and podcast charts.
Show and episode metadata plus the country-by-country charts.
Titles, cast and crew, ratings, reviews, and awards.
Movie and TV detail, curated lists, and people — the free-API alternative.
Metascores, user scores, and reviews across games, movies, and TV.
Film ratings, histograms, reviews, and member stats from a social film community.
Tomatometer and Audience Score — a paid, approval-gated official API.
Grosses, weekend charts, and franchise rollups — no public API.
Titles, characters, staff, rankings, and airing schedules — AniList-sourced.
Community-built catalogs — creative works and the public reviews around them.
Book detail, ratings, reviews, authors, and lists — no official API since 2020.
Releases, masters, artists, and labels from a community-built music database.
Ebook and audiobook search, catalog detail, reviews, series, and charts.
Titles, scores, genres, and rankings from a community-built database.
Standalone datasets that don't fit a single vertical.
Stores, player counts, and live match data.
Reviews, charts, and concurrent player counts for any app ID.
App listings, rankings, permissions, and data-safety disclosures.
Live scores, lineups, statistics, and standings across competitions.
Scoreboards, standings, rosters, and athlete pages across leagues.
Schedules, standings, rosters, stats, and play-by-play for baseball.
Game prices, editions, and deals — the console side of storefront scraping.
What's on sale, where, and to whom.
Prices, fundamentals, and the market context around them.
Quotes, price history, financials, holders, and earnings dates.
Coin prices, market caps, exchanges, and trending tokens.
Quotes, financials, and market-wide movers, earnings, and categories.
Filings, financials, and insider data — EDGAR's free API vs. a parsed one.
The largest prediction market — per-token order books, prices, and depth.
Regulated event markets — public, keyless market data straight from Kalshi.
Crowd-sourced forecasts — a mostly keyless official API, not a trading market.
Launches, traction, and the capital behind them.
Code, contributors, and open roles.
Repos, users, contributors, and the trending pages.
Extension search, permissions, and privacy disclosures.
Openings straight from 13 ATS platforms rather than an aggregator.
Search and job detail — the Publisher/XML feed is deprecated.
Freelance jobs and profiles — the official API needs an approved app.
Gig search, detail, and seller profiles — no public read API.
Routes, clubs, and the communities built around personal activity data.
The three guides that apply to every site on this page.
Pick the tool before you write the scraper — managed APIs, proxies, and what each is actually for.
The search-results side: which SERP APIs return clean rankings, and at what cost.
Public data, ToS, hiQ v. LinkedIn, the CFAA, and where personal data changes the answer.
Looking for a site that is not listed? The platform APIs cover more sources than there are guides, and the generic scraping API handles any URL you can point it at. New guides are added to this page as they publish — see the blog for everything else we write.
Cross-cutting basics, answered in one or two sentences — the deep, per-platform FAQ lives on each individual guide.
Every guide here ends at the same place: a documented endpoint that returns normalized JSON and bills only on success.
Social platforms
Profiles, posts, comments, and the engagement signals behind them.
Reddit
Subreddit posts, comment trees, and user history for sentiment and RAG pipelines.
YouTube
Videos, channels, comments, and transcripts — including the caption track.
TikTok
Posts, profiles, hashtag challenges, and the trending surfaces.
Instagram
Public profiles, posts, and reels without a logged-in session.
Twitter / X
Profiles and posts after the API pricing changes closed off the cheap routes.
LinkedIn
Company pages and products — and where the hiQ ruling actually leaves you.
Telegram
Public channel history via the t.me preview surface or Telethon.
Threads
Profiles, posts, and reply trees on Meta's text network.
Bluesky
The most open platform in the series — AT Protocol needs no API key.
Pinterest
Public pins, boards, and search — the official API is account-scoped only.