A crawl frontier is the crawler's managed collection of discovered requests that have not been fetched yet. It decides which URL is eligible next, filters duplicate requests, applies priorities and per-site limits, and preserves enough state for a large crawl to stop and resume without starting over.
A crawler starts with seed URLs and discovers more links as responses are parsed. Those new requests enter the frontier rather than going straight to the network. Download workers ask the frontier for the next eligible request, then feed newly discovered links back into it. Scrapy calls this component its scheduler: it stores requests and returns them to the engine in the order determined by queues and priorities.
The frontier is therefore more than a list. It combines pending-request storage with duplicate filtering, ordering, retry readiness, and often host-level politeness. The crawl is complete only when no requests remain pending and no active download can discover more work.
FIFO queues tend toward breadth-first crawling, reaching many shallow pages before going deeper; LIFO queues tend toward depth-first crawling. Explicit priorities can move sitemaps, recent pages, or high-value record URLs ahead of low-value navigation paths. Concurrency means completion order can still differ from dequeue order, so data pipelines should not depend on pages finishing in strict sequence.
A production frontier also needs backpressure. If parsing discovers URLs faster than workers can fetch them, an unbounded in-memory queue can exhaust the process. Persistent disk or database queues let the crawler absorb bursts, checkpoint work, and recover after a restart.
Frontier deduplication normally fingerprints a normalized URL and sometimes the HTTP method and body. The rule must match the target: dropping fragments is usually safe because they are not sent in HTTP requests, while dropping every query parameter can collapse distinct search pages or product variants into one request.
In distributed crawls, enqueue and claim operations need atomic semantics so two workers do not fetch the same request simultaneously. Keep retry state separate from the permanent seen set; otherwise a transient timeout can mark a URL as finished and make the frontier silently skip it forever.
How Crawlora handles this
Crawlora's documented platform endpoints give callers bounded, structured operations for supported sources, which can replace a general-purpose frontier for those workflows. A custom crawler still needs its own queue, deduplication, checkpoint, and completion rules for arbitrary URL discovery.
Related reading
Glossary
FAQ
A queue is its core, but a useful frontier also applies duplicate filtering, priority, readiness, persistence, and per-host pacing. Those policies decide which discovered request becomes eligible next.
It is empty when it has no pending eligible or delayed requests. A crawl is only finished when the frontier is empty and active workers cannot discover or retry more requests.
Not automatically. Track bounded retries separately so transient failures can return to the frontier. Mark a request complete only after its success or terminal-failure policy has been applied.
Beyond Crawl Frontier, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.