Web Scraping with Node.js — Fetch, Cheerio, and Puppeteer

If you write JavaScript all day, you do not need to learn Python to scrape the web. Node.js ships everything a basic scraper needs — fetch has been built in since Node 18, Cheerio gives you jQuery-style HTML parsing over raw markup, and Puppeteer drives a real Chrome when the page only exists after JavaScript runs. This guide builds a working scraper with each, in that order, on one consistent practice site, then is honest about where DIY Node scrapers hit a wall in production. It is the JavaScript counterpart to our web scraping with Python guide — same journey, your language.

Static pages: native fetch + Cheerio

Most scraping tutorials start by installing an HTTP client. Skip that — since Node 18, fetch is global. The only dependency for a static-page scraper is Cheerio:

npm install cheerio

Cheerio 1.x is a dual CommonJS/ESM package and wants Node 18.17 or newer, which lines up nicely with native fetch. We will scrape books.toscrape.com, a sandbox bookstore built for exactly this kind of practice: 1,000 books, 50 pages, no login, no tricks.

The pattern never changes: fetch the HTML, load it into Cheerio, select elements with CSS selectors, and map each one into a plain object.

// scrape-books.mjs — Node 18+, run with: node scrape-books.mjs
import * as cheerio from "cheerio";
import { writeFile } from "node:fs/promises";

const BASE = "https://books.toscrape.com/";

const res = await fetch(BASE);
if (!res.ok) throw new Error(`HTTP ${res.status}`);
const html = await res.text();

const $ = cheerio.load(html);

const books = $("article.product_pod")
  .map((_, el) => {
    const $el = $(el);
    const link = $el.find("h3 a");
    return {
      title: link.attr("title"),
      price: $el.find(".price_color").text(),
      inStock: $el.find(".availability").text().trim().startsWith("In stock"),
      url: new URL(link.attr("href"), BASE).href,
    };
  })
  .get();

await writeFile("books.json", JSON.stringify(books, null, 2));
console.log(`Saved ${books.length} books`);

A few things worth noticing:

• cheerio.load(html) returns $ — a function you use exactly like jQuery. $("article.product_pod") selects every book card; .find(), .text(), and .attr() walk into each one.
• .map(...).get() converts a Cheerio selection into a real JavaScript array. Inside the callback, el is a raw DOM node, so wrap it back in $() before querying it.
• new URL(href, BASE) resolves relative links — the site links to catalogue/some-book_1/index.html, and this turns it into an absolute URL you can actually fetch later.
• Check res.ok before parsing. A 404 page or an error page is still HTML; Cheerio will happily parse it and hand you an empty array, which is a much more confusing failure than a thrown error.

Want CSV instead of JSON? You do not need a library for a flat structure like this:

const header = "title,price,inStock,url";
const rows = books.map((b) =>
  [JSON.stringify(b.title), b.price, b.inStock, b.url].join(",")
);
await writeFile("books.csv", [header, ...rows].join("\n"));

The JSON.stringify on the title is a cheap way to quote-and-escape a field that might contain commas. For anything nested or messy, reach for a proper CSV package — but do not reach for one before you need it.

One thing Cheerio will never do: run JavaScript. It parses the markup the server sent, and that is the whole story. That makes it fast — no browser, no rendering — and it is the right default for any page whose data is visible in "view source." When the data is not there, skip ahead to the Puppeteer section.

Pagination and politeness

One page is a demo. A real scraper walks all 50 pages, and how it walks them is the difference between a useful tool and a banned IP. Three habits matter: identify yourself, pace yourself, and fail loudly.

// scrape-all.mjs
import * as cheerio from "cheerio";
import { writeFile } from "node:fs/promises";

const CATALOGUE = "https://books.toscrape.com/catalogue/";
const sleep = (ms) => new Promise((resolve) => setTimeout(resolve, ms));

const books = [];

for (let page = 1; ; page++) {
  const res = await fetch(`${CATALOGUE}page-${page}.html`, {
    headers: { "user-agent": "books-demo/1.0 (learning project)" },
  });

  if (res.status === 404) break; // ran past the last page — we're done
  if (!res.ok) throw new Error(`HTTP ${res.status} on page ${page}`);

  const $ = cheerio.load(await res.text());
  $("article.product_pod").each((_, el) => {
    books.push({
      title: $(el).find("h3 a").attr("title"),
      price: $(el).find(".price_color").text(),
    });
  });

  console.log(`page ${page}: ${books.length} books so far`);
  await sleep(1000); // one request per second is a polite default
}

await writeFile("books.json", JSON.stringify(books, null, 2));

This loop encodes the three habits:

1. A descriptive user-agent. Site operators seeing your traffic in their logs should be able to tell what it is. An empty or fake browser UA is the first thing that looks like abuse.
2. A delay between requests. The sleep helper is four lines shorter than any package that does the same thing. One request per second costs you under a minute on this site and keeps you far away from any rate limiting threshold. Node's async model makes it tempting to fire all 50 fetches with Promise.all — resist it. Concurrency is for your own servers, not someone else's.
3. Explicit error handling. A 404 here means "past the last page," so it ends the loop. Anything else non-OK throws with the page number in the message. Silent failures produce datasets with holes you discover weeks later.

JavaScript-rendered pages: Puppeteer

Sooner or later you will fetch a page, load it into Cheerio, and get nothing — the selectors that clearly exist in DevTools return an empty array. That is a client-rendered page: the server sent a skeleton and the data arrived later via JavaScript. Cheerio never runs JavaScript, so as far as it is concerned the data does not exist.

The fix is a headless browser — a real Chrome without a window, driven from your script. In Node the standard choice is Puppeteer:

npm install puppeteer

// scrape-rendered.mjs
import puppeteer from "puppeteer";

const browser = await puppeteer.launch(); // headless by default
const page = await browser.newPage();

await page.goto("https://quotes.toscrape.com/js/", {
  waitUntil: "networkidle2",
});
await page.waitForSelector(".quote"); // wait for the JS to render them

const quotes = await page.$$eval(".quote", (cards) =>
  cards.map((card) => ({
    text: card.querySelector(".text")?.textContent.trim(),
    author: card.querySelector(".author")?.textContent.trim(),
  }))
);

await browser.close();
console.log(quotes);

The target here is the JavaScript-rendered twin of our bookstore's sibling site — quotes that only exist after client-side rendering. Two lines carry all the weight: waitUntil: "networkidle2" holds goto until the network has mostly gone quiet, and waitForSelector confirms the specific elements you care about actually rendered before you read them. page.$$eval then runs your callback inside the browser, so you write ordinary querySelector code and get plain JSON back out.

The cost is real, though. A Cheerio scrape is a few milliseconds of parsing; a Puppeteer scrape launches a whole Chrome, loads every asset, and executes every script — roughly two orders of magnitude slower and far heavier on memory. Use it only for pages that need it, and keep using fetch + Cheerio for everything else. Many production setups mix the two: Puppeteer to render, then cheerio.load(await page.content()) to parse with the nicer API.

The other mainstream option is Playwright, which has a first-class JavaScript API, auto-waiting locators, and one interface across Chromium, Firefox, and WebKit. Our Playwright scraping guide is written around Python, but the concepts — waiting strategies, locators versus selectors, when a headless browser is worth its cost — carry over to the JS API almost line for line. For scraping specifically, Puppeteer and Playwright are close enough that either is a fine choice; Playwright's auto-waiting removes a class of flaky-selector bugs, while Puppeteer is lighter and Chrome-only.

Node vs Python for web scraping

The honest answer up front: both are excellent, and the deciding factor is almost never the language. Here is how the pieces line up.

Node's real advantages: fetch with zero dependencies, an async runtime where polite concurrency (say, three sites in parallel, each paced) is idiomatic rather than bolted on, and the fact that the pages you scrape are themselves written in JavaScript — debugging a client-rendered app is more natural when you speak its language. Python's real advantages: Scrapy has no Node peer at its maturity, and the moment your scrape feeds analysis, pandas and the scientific stack take over.

But the tiebreaker is boring: use the language your project already uses. A scraper is a component, not an identity. If your stack is TypeScript, writing the scraper in Python buys you a second toolchain to maintain and nothing else. If you are genuinely starting from zero on a data-heavy project, the Python guide makes the case for the other side.

The production wall — and the API bridge

Everything above works flawlessly on practice sites, and that is not an accident: the sandbox sites exist because real targets fight back. Point the same code at a large e-commerce or travel site and you will meet the production wall — Cloudflare, DataDome, PerimeterX and friends checking TLS handshakes, browser fingerprints, IP reputation, and behavior before your request ever reaches the page. Native fetch fails the TLS check alone; vanilla Puppeteer leaks automation markers that fingerprinting scripts test for directly. You can fight it — stealth plugins, rotating residential proxies, header forgery — and our guide to scraping sites that block bots covers that arms race honestly. But it is an arms race, and it becomes a job.

The alternative is to let a web scraping API fight it for you. Your code stays exactly what you already know — one fetch call — but the proxies, fingerprints, rendering, and retries happen on the other side of the endpoint, and you get structured JSON instead of HTML to parse:

// Same fetch you've used all guide — the hard part moved behind the endpoint
const keyword = encodeURIComponent("mechanical keyboard");

const res = await fetch(
  `https://api.crawlora.net/api/v1/amazon/search?k=${keyword}`,
  { headers: { "x-api-key": process.env.CRAWLORA_API_KEY } }
);
if (!res.ok) throw new Error(`HTTP ${res.status}`);

const { data } = await res.json();
for (const item of data) {
  console.log(item.title, item.price);
}

No Cheerio, no selectors, no parser to fix when the layout shifts — the endpoint returns normalized fields for Amazon search results, and there are equivalents for other platforms in the docs. Billing is pay-on-success: you are charged only for successful 2xx responses, so a blocked or failed request costs nothing. Where DIY and API each make sense is a real question with a real trade-off — web scraping vs API walks through it — but the short version is: DIY for learning and for cooperative sites, an API when the target fights back or when parser maintenance starts eating your sprints. Pricing scales with usage, and the free tier is 2,000 credits/month, no card.

• Node 18+ installed, so fetch is available globally
• Static pages: fetch + cheerio.load, select with CSS selectors, map to objects, write JSON or CSV
• Pagination: loop with a sleep helper, a descriptive user-agent, and loud error handling
• Checked robots.txt and terms before scraping any real site
• JS-rendered pages: Puppeteer with waitForSelector, or Playwright for auto-waiting locators
• Hitting anti-bot walls: stop fighting fingerprints and switch to a scraping API endpoint

Skip the anti-bot arms race

Keep writing JavaScript — call documented endpoints that return normalized JSON while proxies, rendering, and retries are handled for you. 2,000 credits/month, no card.

Try the PlaygroundSee the Web Scraping API

Related reading

• Web scraping with Python — the same journey in the other language, with BeautifulSoup and Scrapy.
• Web scraping with Playwright — deeper on headless browsers, waiting strategies, and locators.
• Scraping sites that block bots — what anti-bot systems actually check and what it takes to pass.
• Web scraping vs API — a framework for the build-or-buy decision.