URL canonicalization is the process of choosing one representative URL for a resource that can be reached through multiple URL forms. Crawlers use canonical identities to avoid fetching and storing duplicates, while publishers and search engines use redirects, rel=canonical annotations, and sitemaps as signals about the preferred public URL.
RFC 3986 defines transformations that preserve URI equivalence, such as lowercasing the scheme and host, removing dot segments, normalizing percent-encoding, and removing a default port. These are safe, syntax-based comparisons. They do not prove that arbitrary query parameters, trailing slashes, or path case are interchangeable on a particular site.
Page canonicalization handles duplicate content that remains under genuinely different URLs, such as tracking parameters, print views, or filtered category routes. Google treats redirects and rel=canonical as strong signals and sitemap inclusion as a weaker signal, but the search engine can still select a different representative URL.
Resolve relative links against the response URL, remove fragments because they are not part of the HTTP request target, and apply only standards-backed syntax normalization by default. Keep a site-specific allowlist for query parameters known to be tracking-only instead of deleting every parameter with a broad regex.
Record both the fetched URL and the canonical identity. The fetched URL explains redirects and provenance; the canonical key supports deduplication. If the page declares a cross-domain canonical, treat it as evidence to review rather than an instruction to discard the current page automatically, because canonical annotations can be stale or misconfigured.
Duplicate URL variants consume frontier capacity, request budget, parsing time, and storage while adding no new record. Collapsing proven equivalents lets a crawler spend that capacity on unseen content and keeps downstream counts from treating one page as several observations.
Over-aggressive canonicalization causes the opposite failure: distinct locale, sort, pagination, or product-variant URLs collapse into one key and disappear from the dataset. Test normalization rules against known pairs of equivalent and distinct pages before applying them across a domain.
How Crawlora handles this
Crawlora's platform endpoints expose source-specific identifiers and structured fields for supported sources, reducing the need to infer identity from arbitrary page URLs. Direct crawlers should still keep conservative canonicalization and provenance rules for every fetched URL.
Related reading
Glossary
FAQ
No. A redirect sends the client to another URL. A rel=canonical annotation leaves the current page accessible and signals a preferred representative to consumers such as search engines.
No. Some parameters are tracking-only, but others select a page, language, product variant, or result filter. Remove parameters only when the site's behavior proves they do not change the resource you need.
No. Store it as a strong hint and verify it against redirects, content, and domain policy. Misconfigured or cross-domain canonical tags can otherwise merge unrelated records.
Beyond URL Canonicalization, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.