An ETag is an opaque, quoted value a server attaches to one representation of a resource. A scraper can send that value back in an If-None-Match header on a later request; if the selected representation is unchanged, the server can answer 304 Not Modified with no response body, letting the scraper reuse the body it already stored.
An ETag might look like "abc123" or W/"abc123", but its contents have no client-visible meaning. Store and replay the entire value exactly, including the quotes and any W/ prefix. The prefix marks a weak validator: two representations can count as equivalent even when their bytes are not identical. For GET and HEAD requests, If-None-Match uses that weak comparison because the useful question is whether the cached representation is still suitable to reuse.
A validator belongs to the URL and representation that produced it. Changing Accept, authentication, cookies, language, encoding, or another input that changes the selected representation can make an old tag inapplicable. Cache the validator beside those request dimensions instead of treating one ETag as a global version for a URL.
Send the stored ETag as If-None-Match. A match on a GET produces 304 Not Modified without a message body; that response tells you to reuse the previously validated body. Never overwrite stored content with the empty 304 body. A non-matching tag normally produces the current representation and a validator to save for the next visit.
Last-Modified and If-Modified-Since provide a date-based fallback when a server does not emit ETags. If a request contains both If-None-Match and If-Modified-Since, the server ignores the date condition and evaluates the ETag condition. Dates are less precise, so keep the ETag as the primary validator when both are available.
A conditional request still reaches the server and can count toward rate limits. It saves response bytes and parsing work when content is unchanged; it does not grant extra request quota or bypass anti-bot checks. Some servers omit validators or change them on every response, so a crawler needs an ordinary fetch-and-compare fallback.
Use conditional fetching for revisit-heavy jobs such as news monitoring, price tracking, and periodic site audits. Record the status, validator, and last successfully parsed body together so a failed parse or a bodyless 304 cannot corrupt the last known-good record.
How Crawlora handles this
Conditional requests are most useful when your crawler fetches target URLs directly. For supported sources, Crawlora's documented platform endpoints return structured JSON, which can reduce the raw-page download and parsing work that ETag-based revisits are designed to avoid.
Related reading
FAQ
No. Treat the ETag as opaque and replay it exactly as received, including its quotes and W/ prefix. Rewriting it can turn a valid condition into a different value the server does not recognize.
No. A 304 deliberately has no message body. Keep using the stored body that the validator just confirmed; replacing it with the 304 payload would erase valid cached content.
No. The server still receives and evaluates each conditional request, so it can count toward the same rate limit as a full response. ETags primarily save transfer bandwidth and downstream parsing work.
Beyond ETag and Conditional Requests, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.