llms.txt is a proposed standard — a markdown file published at a website's root (or any subpath) that gives AI agents and coding assistants a curated, token-efficient index of a site's most important pages — created by Jeremy Howard of Answer.AI in September 2024 and revised to a v2 draft in August 2026.
An llms.txt file is plain markdown: an H1 with the site or brand name, a short blockquote describing what the site covers, then headed sections grouping links to the pages an agent would need, each with a one-line description. It can sit at the domain root to cover a whole site, or at a subpath like /docs/llms.txt to scope itself to just that section. The companion half of the proposal is per-page markdown — a page.html.md or page.md file alongside the regular HTML, discoverable via a rel="alternate" type="text/markdown" link or an HTTP Link response header — so a client can fetch a clean, nav- and ad-stripped version of any page at the same URL instead of parsing the rendered page itself.
The idea solves a narrow but real problem: an HTML page wraps its real content in navigation, ads, and client-side JavaScript, and an LLM's context window is too small and too expensive to burn on all of that for every page it touches. llms.txt gives an agent a short, structured map it can hold entirely in context, then fetch the specific linked page only when it actually needs the detail — the opposite of crawling a whole site to find one fact.
Thousands of sites already publish an llms.txt, documentation platforms generate one automatically, and Chrome's Lighthouse now audits for one as part of its agentic-browsing checks. OpenAI, Anthropic, and Google all publish llms.txt files for their own developer docs. But as of early 2026 none of the major AI labs has publicly confirmed that its production crawlers or agents actually read and act on the file — it remains a convention sites adopt hoping agents will use it, not a mechanism those agents are required to honor.
That's starting to shift on the standards side: in June 2026 the W3C published a working draft, "AI Crawler Guidance Standardization," opening public comment on formalizing something like llms.txt as an actual web standard rather than a community convention — roughly the path robots.txt itself took decades after it started as an informal practice.
How Crawlora handles this
Crawlora's own API already speaks the language this proposal is built around: a `/web/scrape` call can request `formats: ["markdown"]` to get a page back as clean, LLM-ready markdown instead of raw HTML, and crawlora.net serves its own pages as `text/markdown` via content negotiation. An llms.txt file tells an agent which pages are worth fetching; Crawlora's scrape endpoint handles turning whichever page it picks into the clean format the agent actually wants.
Glossary
FAQ
Adoption on the publishing side is real — thousands of sites have one, and OpenAI, Anthropic, and Google publish them for their own docs — but as of early 2026 no major AI lab has confirmed its production crawlers or agents actually fetch and act on the file. It's a convention, not a guarantee.
No. robots.txt tells crawlers which URLs they may fetch at all; llms.txt assumes fetching is fine and instead curates which pages are worth an agent's limited context, with short descriptions pointing it to the right content.
At a site's root for whole-site coverage, or at any subpath — an llms.txt at /docs/llms.txt covers everything under /docs/. There's no requirement to have only one; a site can scope different files to different sections.
Beyond llms.txt, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.