HTTP 404 Not Found means the server has no resource at the requested URL. For scrapers it usually signals a bad, stale, or mistyped link — but some sites deliberately shape their 404 behavior around bots, which makes it worth distinguishing from a genuinely dead page.
A real 404 means the page never existed or has been permanently removed — the correct response is to drop that URL and move on. A soft 404, by contrast, returns HTTP 200 with a page that says "not found" in the body (common on sites whose framework doesn't set the status code correctly), or the reverse: a 404 status for a URL that objectively still exists, served specifically to traffic the site doesn't want to serve. Status code alone can't distinguish these — checking the response body content is what tells them apart.
That second case is uncommon but real: a handful of sites use 404-shaped responses as a soft anti-bot measure, hiding content from suspected automated traffic without the more visible signal of a 403 or CAPTCHA. If a URL 404s for your scraper but loads fine in a browser, treat it with the same diagnosis as a disguised 403, not as a genuinely missing page.
A sudden spike in 404s partway through a run is rarely random — the common causes are a URL pattern that changed after a site redesign, off-by-one pagination logic walking past the last real page, or product/listing IDs that expired or were delisted since your URL list was built. Manually spot-checking one or two failing URLs in a browser is the fastest way to tell which case you're in, before assuming it's a detection issue.
Because 404s are cheap for a target server to return, they're also low-signal for rate-limiting purposes compared to a 429 or 403 — but a large volume of 404 requests still wastes your own request budget and adds unnecessary load to the target, so it's worth fixing the URL source rather than just tolerating the failure rate.
How Crawlora handles this
Crawlora's platform endpoints return structured errors that distinguish a genuinely missing resource from a blocked or malformed request, and discovery through sitemap-backed or search-based endpoints reduces the stale-URL problem that causes most legitimate 404 waves in the first place.
Glossary
FAQ
A 404 says the resource doesn't exist at that URL; a 403 says it exists but access is refused. Sites occasionally use a 404 deliberately to hide a resource's existence from unwanted traffic rather than revealing it with a more informative 403 or CAPTCHA.
That mismatch means it's very likely not a real missing page — treat it as a disguised anti-bot response and apply the same diagnosis as a 403: check your fingerprint, headers, and IP type rather than assuming the content is actually gone.
Prefer sitemap- or API-based URL discovery over guessed or hand-built pagination patterns, since those track a site's actual current structure. When 404s do spike, spot-check a few URLs manually before re-running the whole job — the fix is usually in the URL source, not the request logic.
Beyond 404 Not Found Error, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.