Search every robots.txt on the web.

One index of robots.txt files from about 175 million domains, fetched from the live web. Search by crawler name, blocked path, sitemap or any text, and get back the domains that match.

Sign in with Google to run a search · free while in beta.

Try: User-agent: GPTBot Disallow: /wp-admin/ Crawl-delay: Sitemap: https:// ClaudeBot
example.com/robots.txt200 · text/plain
  1. # robots.txt for example.com
  2. User-agent: *
  3. Disallow: /admin/
  4. Disallow: /cart/
  5. Crawl-delay: 5
  7. User-agent: GPTBot
  8. Disallow: /
  10. User-agent: ClaudeBot
  11. Disallow: /
  13. Sitemap: https://example.com/sitemap.xml

A match as a crawler sees it: the file exactly as served, matching lines marked.

What you can find

Crawler policies

Which sites block GPTBot, ClaudeBot, Bytespider or CCBot, and which let them in. Search the user agent, read the exact rules that apply to it.

User-agent: GPTBot

Blocked paths

Disallow lines map what a site keeps out of search: admin panels, staging, exports, internal APIs. Search a path across the whole web at once.

Disallow: /wp-admin/

Sitemaps

Every Sitemap: line in one place. Find the sitemap indexes behind a list of domains, or the ones that point at another host.

Sitemap: https://

How the index is built

Domains
From Certificate Transparency logs, the same feed behind ctlogs.dev: every registrable domain that gets a certificate and resolves is a candidate.
Fetching
One request for /robots.txt per domain, from the domain's own origin, paced per origin IP so no server sees more than one request every few seconds.
Storage
The raw file exactly as served, no parsing, no normalisation. What you search is what a crawler sees. The index keeps the latest copy per domain.
Search
Full text over every line: a user agent, a path, a sitemap host or any phrase. Results list the matching domains and the matching lines.
Access
Free while in beta, with a Google sign-in. The sign-in only opens your workspace and lets us e-mail you when it is ready; nothing else is read from your account.