Web Scraping with Puppeteer — A Practical Node.js Guide

Puppeteer is Google's Node.js library for driving Chrome (and Firefox) over the Chrome DevTools Protocol or WebDriver BiDi. When a page only yields data after JavaScript runs, a plain fetch returns an empty shell — you need a real browser. This guide is the practical path in 2026: install, scrape a JS sandbox, block heavy assets, intercept JSON, stay honest about detection, and know when to stop running Chromium yourself.

If you are new to scraping, start with web scraping with Python for the static-HTML tier. For the cross-browser Python default, see web scraping with Playwright. This post is for Node.js teams that already want Puppeteer.

Why Puppeteer for scraping?

Puppeteer stays the natural choice when your stack is already Node, you only need Chromium, and you want a thin CDP wrapper. Playwright wins for multi-browser coverage and auto-waiting ergonomics in greenfield projects; see the comparison later in this post.

Setup and your first scrape

npm i puppeteer
# If install scripts are blocked by your package manager:
npx puppeteer browsers install

puppeteer downloads a matched Chrome build. Use puppeteer-core when you point at a system Chrome or a remote browser service.

Here is a complete scraper against quotes.toscrape.com/js, a sandbox that renders quotes with JavaScript (plain HTTP returns zero quotes):

import puppeteer from "puppeteer";

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.setViewport({ width: 1280, height: 800 });

await page.goto("https://quotes.toscrape.com/js/", {
  waitUntil: "domcontentloaded",
});
await page.waitForSelector(".quote");

const quotes = await page.$$eval(".quote", (cards) =>
  cards.map((card) => ({
    text: card.querySelector(".text")?.textContent?.trim() ?? "",
    author: card.querySelector(".author")?.textContent?.trim() ?? "",
    tags: [...card.querySelectorAll(".tag")].map((t) => t.textContent.trim()),
  })),
);

console.log(`scraped ${quotes.length} quotes`);
console.log(quotes[0]);
await browser.close();

Notes that save hours later:

• waitUntil: "domcontentloaded" is usually enough; networkidle0 hangs on pages with analytics or websockets.
• waitForSelector (or page.locator(...).wait()) replaces setTimeout guesswork.
• Explicit viewport avoids the tiny default headless size that triggers mobile layouts and missing desktop selectors.

Extracting data cleanly

Three patterns cover most scrapers:

// 1) One field
const title = await page.$eval("h1", (el) => el.textContent.trim());

// 2) Many records ($$eval runs in the page context — fast)
const rows = await page.$$eval("article.product", (nodes) =>
  nodes.map((n) => ({
    name: n.querySelector("h2")?.textContent?.trim(),
    price: n.querySelector(".price")?.textContent?.trim(),
  })),
);

// 3) Locator API (Puppeteer 22+) — auto-retry style waits
await page.locator("button#load-more").click();

Paginate by clicking "Next" in a loop, or by walking ?page=N URLs when the site exposes them. Cap concurrency; each open page is a real browser process.

Power features for scrapers

Block images, fonts, and media

A scraper does not need pixels. Abort heavy resource types before navigation:

await page.setRequestInterception(true);
page.on("request", (req) => {
  const type = req.resourceType();
  if (type === "image" || type === "font" || type === "media") {
    req.abort();
  } else {
    req.continue();
  }
});
await page.goto("https://quotes.toscrape.com/js/");

Expect large bandwidth and CPU savings on image-heavy catalogs. Two caveats: blocking stylesheets can break visibility-dependent selectors, and stripping analytics on some anti-bot sites can make you more suspicious.

Intercept JSON instead of scraping the DOM

If DevTools Network shows an XHR that already returns structured data, capture it:

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();

/** @type {any[]} */
const payloads = [];
page.on("response", async (res) => {
  const url = res.url();
  if (!url.includes("/api/") || res.request().resourceType() !== "xhr") return;
  try {
    payloads.push(await res.json());
  } catch {
    // not JSON
  }
});

await page.goto("https://example.com/catalog", { waitUntil: "networkidle2" });
console.log(payloads[0]);
await browser.close();

When it works, this beats CSS selectors: typed fields, fewer redesign breakages, often extra fields the UI never shows. Check the Network tab before you write a single locator.

Parallel pages without N browsers

Launch one browser, open many pages (or browser contexts), and throttle with a simple pool:

import puppeteer from "puppeteer";

async function mapPool(items, limit, worker) {
  const ret = [];
  let i = 0;
  await Promise.all(
    Array.from({ length: limit }, async () => {
      while (i < items.length) {
        const idx = i++;
        ret[idx] = await worker(items[idx]);
      }
    }),
  );
  return ret;
}

const browser = await puppeteer.launch({ headless: true });
const urls = [
  "https://quotes.toscrape.com/js/page/1/",
  "https://quotes.toscrape.com/js/page/2/",
];

const results = await mapPool(urls, 3, async (url) => {
  const page = await browser.newPage();
  try {
    await page.goto(url, { waitUntil: "domcontentloaded" });
    await page.waitForSelector(".quote");
    return page.$$eval(".quote .text", (els) => els.map((e) => e.textContent));
  } finally {
    await page.close();
  }
});

await browser.close();
console.log(results.flat().length);

Three to five concurrent pages per machine is a realistic starting point. Unbounded concurrency OOMs the host and rate-limits the target.

Common failures and how to fix them

Can Puppeteer be detected? Yes

Out of the box, headless Chrome driven by Puppeteer is easy to spot. Common signals:

• navigator.webdriver === true
• Headless UA / missing plugins / odd WebGL renderer strings
• CDP runtime artifacts that detection scripts probe for
• Datacenter IP reputation layered on top of the fingerprint

The usual DIY fix is puppeteer-extra with puppeteer-extra-plugin-stealth:

import puppeteer from "puppeteer-extra";
import StealthPlugin from "puppeteer-extra-plugin-stealth";

puppeteer.use(StealthPlugin());

const browser = await puppeteer.launch({ headless: true });
const page = await browser.newPage();
await page.goto("https://example.com");
await browser.close();

Stealth still helps against naive checks. Against Cloudflare, DataDome, PerimeterX, and peers in 2026 it is an arms race you are on the wrong side of: behavioral scoring, TLS fingerprinting, and continuous lab testing against known automation stacks. Pair any browser with residential proxies and still expect CAPTCHAs and silent junk data on hard targets. The escalation ladder lives in scraping sites that block bots.

Puppeteer vs Playwright vs Selenium

Pick Puppeteer when you are Chrome-first in Node and already invested. Pick Playwright for a greenfield scraper that may need WebKit/Firefox or Python. Pick Selenium when Grid and existing WebDriver suites dominate. None of them are free of detection; all of them are expensive at fleet scale.

When to stop running browsers

A browser page needs a real CPU slice and hundreds of MB of RAM. A host that pushes thousands of plain HTTP requests per minute might sustain only a few dozen concurrent Puppeteer pages. Add residential proxies and CAPTCHA solving, and unit economics collapse.

For public pages where you just need clean HTML or Markdown, Crawlora's web scraping API (POST /api/v1/web/scrape) runs the fetch (and escalates to a headless browser when render: "auto" requires it) behind one key:

curl -X POST "https://api.crawlora.net/api/v1/web/scrape" \
  -H "x-api-key: $CRAWLORA_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com", "formats": ["markdown", "html"], "render": "auto"}'

const res = await fetch("https://api.crawlora.net/api/v1/web/scrape", {
  method: "POST",
  headers: {
    "x-api-key": process.env.CRAWLORA_API_KEY,
    "Content-Type": "application/json",
  },
  body: JSON.stringify({
    url: "https://example.com",
    formats: ["markdown", "html"],
    render: "auto",
  }),
});
const { data } = await res.json();
console.log(data.markdown?.slice(0, 400));
console.log("fetch method:", data.scrape?.method); // "http" | "browser" | ...

For platforms Crawlora already normalizes (Amazon, Reddit, Google SERP, …), prefer the platform endpoint over a generic browser: you get typed JSON instead of maintaining selectors. Free tier is 2,000 credits/month, no card; billing is pay-on-success. See pricing, try any URL in the Free Web Scraper, or open the playground.

• Static HTML — use fetch + cheerio (or Python requests + BeautifulSoup), not Puppeteer.
• JS-rendered, lightly protected — Puppeteer with asset blocking and response interception.
• Data already arrives as XHR JSON — intercept it; skip the DOM.
• Hard WAFs at volume — managed /web/scrape or a platform API beats a stealth farm.
• Mixed workloads — keep Puppeteer for the long-tail UI flows; API for high-volume targets.

Skip the Chromium farm for protected public pages

Documented endpoints, managed rendering and proxies, pay-on-success billing. 2,000 free credits/month, no card.

Try Free Web ScraperWeb scraping API

Related reading

• Web scraping with Playwright — the Python multi-browser counterpart
• Selenium web scraping — WebDriver path and when it still makes sense
• Scraping sites that block bots — detection escalation ladder
• Proxies for web scraping explained — residential vs datacenter economics
• Best web scraping APIs in 2026 — when to buy the access layer