An XML sitemap is a file — conventionally at /sitemap.xml — that lists a site's URLs, usually with metadata like when each page last changed, so search engines and other crawlers can discover pages directly instead of finding every one by following links.
A sitemap is a <urlset> containing one <url> entry per page, each with a <loc> (the URL) and optionally <lastmod> (last modified date), <changefreq>, and <priority> hints. Large sites exceeding 50,000 URLs or 50MB per file split into multiple sitemaps referenced by a sitemap index — a <sitemapindex> file that points to the individual sitemap files, which is the pattern e-commerce catalogs and content sites with hundreds of thousands of pages use.
The file is publicly readable by design, the same as robots.txt, and robots.txt commonly points to it directly with a Sitemap: line — so checking whether a target publishes one is as simple as requesting the URL.
Sitemaps exist for search engine discovery, but they solve the exact same problem a scraping project's crawl step solves: finding the full list of URLs on a site. When a target publishes a sitemap covering the pages you need, reading it replaces a discovery crawl entirely — no following category pages, no pagination logic, no risk of missing pages that aren't linked from anywhere else on the site.
The tradeoff is completeness and freshness: sitemaps are only as current as the site's generation process, and some publishers omit pages deliberately (out-of-stock products, deprecated listings) that a link-following crawl would still discover through internal navigation. Treat a sitemap as the fast path, and a link-following crawl as the fallback when the sitemap looks incomplete.
import requests
from lxml import etree
xml = requests.get("https://example.com/sitemap.xml", timeout=15).content
root = etree.fromstring(xml)
ns = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}
urls = [loc.text for loc in root.findall(".//sm:loc", ns)]How Crawlora handles this
Crawlora's platform endpoints often read a target's sitemap internally as part of discovery — a Shopify store's product catalog, for example, resolves through its sitemap rather than a manual crawl — so the discovery step is handled the same way the extraction step is: server-side, without you writing crawl logic.
Related reading
Glossary
FAQ
Try /sitemap.xml on the domain directly, or check the site's robots.txt for a Sitemap: line pointing to it — large sites often use a non-default path or a sitemap index file listing several sub-sitemaps.
Not always. Sitemaps are generated by the site's own tooling and can lag behind real content or deliberately omit certain pages. Treat a sitemap as a fast, usually-reliable discovery shortcut, and fall back to link-following if the URL count looks lower than expected.
Usually not for discovery — reading the sitemap gives you the URL list directly, which is faster and lighter on the target than following links. You still need a separate extraction step to pull data out of each page the sitemap points to.
Beyond XML Sitemap, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.