robots.txt is a plain-text file published at a site's root — for example example.com/robots.txt — that tells automated crawlers which paths they're allowed or asked not to request. It's a voluntary standard, not a technical access control.
A robots.txt file is made of User-agent blocks, each followed by Allow and Disallow rules for specific paths, an optional Crawl-delay, and usually a Sitemap line pointing to the site's XML sitemap. "User-agent: *" applies the rules to every crawler that doesn't have its own named block — Googlebot, GPTBot, and others often get specific blocks with different rules.
You can check any site's rules directly — its robots.txt is publicly readable by design; there's no authentication or special access needed to see what a site is asking crawlers to do.
robots.txt is a request, not a lock. Nothing about the file technically prevents a request to a disallowed path from succeeding — it relies on the crawler choosing to honor it. Well-behaved search engine and AI crawlers respect it; a scraper that ignores it will usually still get a normal HTTP response, unless the site backs the rule with actual technical blocking such as rate limiting, IP blocks, or CAPTCHA.
That said, courts have treated a site's robots.txt as evidence of the site owner's stated intent in scraping-related disputes, alongside terms of service and technical access controls — so it isn't a technical barrier, but it isn't legally irrelevant either.
How Crawlora handles this
robots.txt sits at the boundary of Crawlora's responsible-use policy: Crawlora's endpoints are built for public, structured data workflows, and honoring a target site's stated crawling preferences is part of using the API within its acceptable-use terms.
Related reading
Glossary
FAQ
Not by itself — robots.txt is a voluntary standard, not a law. But it can be used as evidence of a site's stated intent in legal disputes over scraping, alongside a site's terms of service and any technical access controls it enforces.
Open the /robots.txt path on the domain directly in a browser — it's a public, unauthenticated plain-text file. Look for User-agent blocks and the Disallow or Allow paths listed under the one that matches your crawler, or User-agent: * if there's no specific match.
No — it's advisory. A site that wants to actually block a crawler backs the request with technical controls like rate limiting, IP blocking, or CAPTCHA challenges; robots.txt alone only works against crawlers that choose to respect it.
Browse Crawlora APIs, test a request in Playground, and move from scraping infrastructure work to production data workflows.