How to Scrape Job Postings in 2026 (13 ATS Platforms & Python)

Why scrape job postings?

Job posting data is the base layer for hiring-market research and recruiting products:

• Hiring-signal research — track which companies are actively hiring, by department, location, and role.
• Compensation and market benchmarking — aggregate posted roles and requirements across a sector.
• Recruiting and sourcing tools — surface open roles across many companies in one feed.
• Company research — infer growth or contraction from a company's own posting volume over time.

Is it legal to scrape job postings?

Option 1: DIY, one integration per ATS

The practical difficulty isn't anti-bot defense — it's that there is no standard. A company's careers page is a thin wrapper around whichever ATS it uses, and each ATS exposes a different (often undocumented) JSON endpoint:

import requests

# Greenhouse — token identifies the company
gh = requests.get("https://boards-api.greenhouse.io/v1/boards/stripe/jobs").json()

# Lever — a different host, a different shape
lever = requests.get("https://api.lever.co/v0/postings/netflix").json()

# Workday — tenant + datacenter + site, POST not GET, and a third shape again
wd = requests.post(
    "https://acme.wd1.myworkdayjobs.com/wday/cxs/acme/External/jobs",
    json={"limit": 20, "offset": 0},
).json()

Each integration is its own small project:

• A dozen-plus vendors, a dozen-plus schemas. Greenhouse nests requisitions under jobs[] with a numeric id; Lever returns a flat list keyed by a string id with HTML-formatted description fields; Workday requires a POST with a tenant, datacenter, and site you have to discover per company, and paginates with limit/offset instead of a token. Multiply that by iCIMS, SmartRecruiters, Recruitee, Rippling, Oracle Recruiting, UKG, Personio, Ashby, Eightfold, and Teamtailor, and you're maintaining a dozen-plus parsers.
• You have to find which ATS a company uses first. There's no directory — you either guess from the careers-page URL pattern (boards.greenhouse.io/<company>, jobs.lever.co/<company>, <company>.wd1.myworkdayjobs.com) or crawl the page and detect the embedded API calls.
• Workday alone is worth building for. It's one of the largest ATS platforms by enterprise posting volume, so skipping it because its POST-based, tenant-scoped API is more work than a simple GET endpoint means missing a meaningful share of the market.
• Closed and stale postings drift silently. Most vendors don't hard-delete a closed requisition immediately, so a naive full-refresh can carry stale "open" roles for days; you need to detect and reconcile status changes, not just append new postings.

Option 2: No-code tools

Generic web scrapers can pull an individual careers page, but the actual bottleneck — integrating a dozen-plus different ATS backends and normalizing their schemas — is the same problem whether you're clicking through a visual tool or writing Python. A tool built around one page shape doesn't solve the vendor-fragmentation problem.

Option 3: A structured jobs API

A jobs scraping API covers 14 ATS platforms — Ashby, Eightfold, Greenhouse, iCIMS, Lever, Oracle Recruiting, Personio, Recruitee, Rippling, SmartRecruiters, Teamtailor, UKG, Workable, and Workday — behind one schema, plus a lookup that tells you which ATS a company actually uses.

Find which ATS a company posts through:

curl "https://api.crawlora.net/api/v1/jobs/company-search?slug=stripe" \
  -H "x-api-key: $CRAWLORA_API_KEY"

Pull a company's board once you know the provider — Greenhouse and Workday shown here, the other 11 follow the same pattern with their own path:

import requests

h = {"x-api-key": "YOUR_API_KEY"}
base = "https://api.crawlora.net/api/v1/jobs"

greenhouse_board = requests.get(f"{base}/greenhouse/board", headers=h, params={"token": "stripe"}).json()["data"]

workday_board = requests.get(f"{base}/workday/board", headers=h, params={
    "tenant": "acme", "datacenter": "wd1", "site": "External", "limit": 50,
}).json()["data"]

Or skip the per-vendor calls entirely and search across every covered company at once through the normalized dataset:

api_base = "https://api.crawlora.net/api/v1"
postings = requests.get(f"{api_base}/datasets/jobs/search", headers=h, params={
    "q": "data engineer", "location": "Tokyo", "remote": "true", "page_size": 50,
}).json()["data"]

datasets/jobs/search accepts company, provider, department, location, employment_type, and remote filters, and include_closed if you need status history rather than only open roles. datasets/jobs/companies lists which companies are hiring and through which provider, with a min_open_roles filter — useful for finding active hirers without knowing their ATS in advance.

What you can collect

Per posting: title, department, location(s), employment type, remote flag, posting/updated date, open/closed status, and the full description — plus which company and which ATS provider it came from. jobs/hiring-signals aggregates a company's board into rollups (open-role counts by department and location over time) without pulling every individual posting.

Limitations and common challenges

• No universal schema. Each ATS has its own field names and nesting; normalize at ingestion, not per-report.
• You need the provider before you can query it directly. Use a company lookup rather than guessing the careers-page URL pattern, or query the unified dataset search instead of the per-vendor endpoints.
• Closed postings don't disappear immediately. Track status explicitly (include_closed on the dataset search) rather than assuming a missing posting means it just closed.
• Workday's API is POST-based and tenant-scoped, unlike the GET-based APIs most other ATS platforms use — budget for it separately if you're integrating vendors yourself.

Where this gets used

• Hiring-signal and market research — track which companies and sectors are actively hiring. See the job market research hub for related use cases.
• Recruiting and talent-sourcing tools — aggregate open roles across companies into one searchable feed.
• The Job Postings dataset — the same coverage pre-collected and queryable without running a crawl. See also Workday runs the job market for what the aggregate posting data actually shows.

Sources

Start collecting

Try it first, free: run any public URL through the Free Web Scraper, or check whether a site blocks bots with the Anti-Bot Checker — no signup.

Test the board endpoints in the Playground, check the schema in the API docs, and review pricing. A posting is only half the picture — pair it with the company behind it. How to scrape LinkedIn covers the company pages, headcount, and product listings the postings belong to, and how to scrape Yahoo Finance covers the fundamentals to read hiring velocity against, since a public company's headcount plans usually move before its guidance does. For the broader toolkit, how to choose a web scraping API, and is web scraping legal.

Part of our how-to-scrape guide series — every platform we cover, in one index.