HTTP content negotiation is the process of selecting one representation of a resource based on available variants and client preferences. A client commonly sends Accept, Accept-Language, or Accept-Encoding, and the server answers with a representation whose Content-Type, Content-Language, and Content-Encoding describe what was selected.
RFC 9110 describes proactive negotiation, where the server chooses from request preferences; reactive negotiation, where the server offers alternatives; and request-content negotiation, where a response advertises formats accepted by a later request. The familiar Accept header is only one dimension among media type, language, and content coding.
Preference values can include quality weights. A client might prefer application/json but accept text/html at a lower weight. The server is not required to honor the first choice, so a scraper must inspect the response Content-Type rather than assuming the body format from its request header.
When a response selection depends on request headers, the server can use Vary to identify those dimensions to caches. A shared cache that ignores the relevant header in its cache key can serve one client's language or media type to another client, even though the origin negotiated correctly.
Cache behavior must be verified at the deployed intermediary, not inferred from the origin's Vary header alone. Some CDNs require explicit cache rules or separate representation URLs when their available cache key cannot include the negotiation header.
Send narrow, honest preferences and treat a 406 Not Acceptable response as evidence that no available representation matched. After any redirect, re-check the final Content-Type and encoding before passing the body to a JSON, HTML, XML, or decompression parser.
Content negotiation changes representation, not necessarily the underlying resource identity. Preserve the request headers that selected the body alongside cached validators and provenance, because the same URL can legitimately produce different bytes for different clients.
How Crawlora handles this
Crawlora's documented platform APIs return structured responses through an explicit API contract. When fetching arbitrary web URLs directly, callers should still set intentional Accept headers and validate the final media type before parsing or caching a body.
Related reading
Glossary
FAQ
No. It states a preference. The server may choose another acceptable representation, ignore the preference, redirect, or return 406. Always validate the response Content-Type and body.
No. Accept describes media types the recipient prefers to receive; Content-Type describes the media type of content actually carried in this message.
It identifies request fields that affected representation selection. A cache key that omits those fields can reuse HTML for a JSON request or one language for another.
Beyond HTTP Content Negotiation, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.