A content signal is a robots.txt directive — `Content-Signal: search=yes,ai-train=no,use=reference` — that lets a site declare, per crawler category, whether its content may be indexed for search, used as live AI input, or absorbed into model training. Cloudflare pushed the format to its managed robots.txt for millions of domains starting in 2025 and extended it with a `use=` parameter in July 2026.
The Content-Signal line sits inside a site's robots.txt and expresses preferences as comma-separated key=value pairs: `search` (may this content be indexed and shown in search results) and `ai-train` (may it be used to train or fine-tune a model), plus — in the version Cloudflare shipped in July 2026 — a `use` parameter with three levels: `use=immediate` (an agent may read and act on it in the moment but not store or reuse it), `use=reference` (the default: index it, excerpt it, link back to it), or `use=full` (summarize and reproduce it freely). Cloudflare's managed robots.txt now ships `search=yes,ai-train=no,use=reference` by default for the millions of domains behind it.
Like robots.txt itself, a content signal is a declaration, not an enforcement mechanism — nothing stops a crawler from reading it and ignoring it. What gives it teeth is who's checking: Cloudflare says it will report, through its public BotBase directory, whether a Verified Bot's actual behavior matches the preferences a site declared, and a verified operator that reproduces full content against a stricter signal risks losing its Verified Bot status — a reputational and access cost even though the robots.txt line itself can't technically block a request.
Plain robots.txt only ever answers one question — can a given user agent fetch this URL at all — with a binary allow or disallow. It has no vocabulary for the question publishers started asking once AI crawlers scaled up: fetching is fine, but what happens to the content after that? A content signal answers that second question without replacing the first; a crawler still needs an Allow under the standard robots.txt rules before a content signal's finer-grained preference even becomes relevant.
It's also a different layer from llms.txt: llms.txt is a site volunteering a curated map of pages an agent should read, aimed at making itself more useful to visit; a content signal is a site setting conditions on content a crawler can already reach, aimed at constraining how traffic that's already indexing or training on the page is allowed to treat it.
How Crawlora handles this
Crawlora's scraping doesn't bypass or spoof a site's declared preferences — the platform's job is proxy routing and browser rendering for publicly reachable pages, not getting around access controls a site has actually enforced. A declared content signal like `ai-train=no` is exactly the kind of publisher preference worth checking before you store or reuse what Crawlora returns, especially for a pipeline feeding a model rather than a one-off lookup.
Related reading
Glossary
FAQ
No — like the rest of robots.txt, it's a declared preference, not a technical block. Enforcement, where it exists at all, happens through reputational mechanisms like Cloudflare's Verified Bot program, not at the robots.txt level itself.
Disallow controls whether a URL can be fetched at all. A content signal assumes fetching is allowed and instead states what may be done with the content afterward — index it, use it live, or train on it.
No — opposite direction. llms.txt is a site offering agents a curated path to useful content; a content signal is a site restricting what automated traffic may do with content it can already reach.
Beyond Content Signal, Crawlora's own docs cover the rest of the stack — browse the APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.