How to Scrape Twitter/X in 2026 (API & Python)

The fastest way to scrape Twitter/X in 2026 is to call a structured X API that returns normalized JSON — public profiles, posts, and first-page timelines — instead of driving a headless browser through X's hidden GraphQL API, guest tokens, and rotating identifiers. DIY is still possible, but the easy paths have all closed: the free libraries are dead, the official API is priced for enterprises, and X reworks its defenses every few weeks. This guide covers all three approaches, what each returns, where each breaks, and the legal basics.

Is it legal to scrape Twitter/X?

Tread carefully. Scraping public X data (public profiles, public posts) is generally legal in the US — hiQ v. LinkedIn held that accessing public web data isn't a Computer Fraud and Abuse Act (CFAA) violation. But that's the criminal line, not the whole picture: X's Terms of Service explicitly prohibit scraping, so it's a contract issue that can get your account or IP banned, and X has litigated aggressively against scrapers. Public posts can also contain personal data governed by GDPR/CCPA when you store or process them. Rules of thumb: public, non-personal data only; never touch protected accounts, DMs, or anything behind a login; don't hoard data longer than you need; respect rate limits. See is web scraping legal. Not legal advice.

Option 1: DIY in Python (and why it breaks)

X's web app is a GraphQL client: every profile and timeline you see is fetched from a hidden internal API guarded by guest tokens, doc_ids, and rate limits. The old shortcuts — snscrape, Twint, and Nitter — all shared one dependency, anonymous guest access, and all broke when X locked it down. What still works is browser automation that lets the page request its own tokens while you capture the JSON it fires in the background:

from playwright.sync_api import sync_playwright

# Conceptual skeleton — not a maintained solution
with sync_playwright() as pw:
    browser = pw.chromium.launch(
        proxy={"server": "http://residential-proxy:8000"}  # datacenter IPs get banned fast
    )
    page = browser.new_page()
    page.goto("https://x.com/nasa")
    page.wait_for_selector("[data-testid='primaryColumn']")
    # From here you still handle guest tokens, doc_ids, and rate limits yourself
    print(page.content())

It works in a demo and then breaks:

• Guest tokens expire. Every GraphQL call needs one, they last a few hours, and X ties each token to the requesting IP — so rotating IPs mid-session breaks the token.
• doc_ids rotate. The identifiers that route GraphQL queries (UserTweets, TweetResultByRestId) change every 2–4 weeks; hardcoded queries then fail silently or return empty results.
• Datacenter IPs get banned on sight. You need rotating residential proxies with careful pacing (~300 requests/hour anonymously) or you hit 429s and blocks within a request or two.
• The official API won't rescue DIY. Since February 2026, X defaults new developers to pay-per-use (~$0.005 per third-party post read, capped at 2M reads/month) with no free read tier; legacy Basic ($200/mo, 7-day search only) and Pro ($5,000/mo, full-archive) are closed to new signups. It's the compliant path via Tweepy, but the cost makes scraping public pages at scale a non-starter.

The real cost isn't writing the scraper — it's re-fixing it every few weeks, which typically eats 10–15 hours a month.

Option 2: No-code tools

Visual extractors and browser extensions export CSV/JSON — fine for a one-off pull of a public profile, less so for in-product pipelines that need predictable fields, and they inherit the same guest-token and rate-limit fragility as DIY.

Option 3: A structured X API

For repeatable workflows over public data, an X scraping API returns normalized JSON with no browser to run, no tokens to refresh, and proxies handled behind one key. Fetch a public profile:

curl https://api.crawlora.net/api/v1/x/profile/USERNAME \
  -H "x-api-key: $CRAWLORA_API_KEY"

The same call in Python:

import requests

profile = requests.get(
    "https://api.crawlora.net/api/v1/x/profile/USERNAME",
    headers={"x-api-key": "YOUR_API_KEY"},
).json()["data"]
print(profile["username"], profile["metrics"]["followers"])

A response is normalized JSON (fields are illustrative — check the docs):

{
  "data": {
    "username": "NASA",
    "name": "NASA",
    "id": "11348282",
    "description": "Making the seemingly impossible, possible. ✨",
    "location": "Pale Blue Dot",
    "is_blue_verified": true,
    "metrics": { "followers": 92157321, "following": 119, "posts": 74188 }
  }
}

Pull a profile's first-page posts, then a single post by its numeric id:

h = {"x-api-key": "YOUR_API_KEY"}
base = "https://api.crawlora.net/api/v1/x"

posts = requests.get(f"{base}/profile/NASA/posts", headers=h, params={"limit": 20}).json()["data"]
first_id = posts["posts"][0]["id"]
post = requests.get(f"{base}/post/{first_id}", headers=h, params={"username": "NASA"}).json()["data"]
print(post["text"], post["metrics"]["likes"], post["metrics"]["views"])

The posts endpoint reads the first public page payload only (limit 1–50, default 20) — it doesn't paginate replies, media tabs, or search. Pass username to /post/{id} to reject mismatched authors. Keep to public, non-personal fields.

Which approach should you use?

What you can collect

Where the public page exposes them: username, display name, numeric id, bio, location, verification, follower/following/post counts, and profile media — plus per-post text, timestamp, URL, author context, and visible engagement (likes, replies, reposts, views, bookmarks). Protected accounts, DMs, full follower lists, and keyword/timeline search sit behind a login and are off-limits.

Limitations and common challenges

• Anonymous surface only. Public profiles, individual posts, and the first-page timeline are reachable; protected accounts, DMs, follower lists, and search need a login and cross ToS lines.
• First page, not the archive. The profile-posts endpoint returns the initial public payload, not a paginated back-catalog — historical search is a Pro/Enterprise API feature ($5,000/mo+), not a scraping target.
• The moving target. X ships defensive changes every 2–4 weeks (guest tokens, doc_ids, IP scoring); a structured API absorbs those so your pipeline keeps returning data.
• Find URLs off-platform. X's own search is login-walled, but Google indexes public posts — site:x.com inurl:status <keyword> builds a target list without an account.
• Identities are personal data. Usernames, names, and post content are personal under GDPR/CCPA — collect only public, factual fields with a lawful basis.

Where this gets used

• Brand monitoring — track public brand and competitor presence on X. See the brand monitoring use case.
• Social listening & sentiment — sample public posts on a topic for research.
• Creator research — evaluate public profiles, reach, and engagement; pair with TikTok and YouTube signals.

Sources

Start collecting

Try it first, free: run any public URL through the Free Web Scraper, or check whether a site blocks bots with the Anti-Bot Checker — no signup.

Test the profile endpoint in the Playground, check the schema in the API docs, and review pricing. See also how to scrape Instagram, how to scrape TikTok, how to scrape YouTube, and how to scrape LinkedIn for the rest of the social stack, how to choose a web scraping API, and is web scraping legal. For a full comparison against other X data APIs — including the official API's 2026 pricing tiers, TwitterAPI.io, and SocialData.tools — see best Twitter/X scraper APIs in 2026.