This robots.txt checker reads a site’s robots.txt and tells you, for each search and AI crawler, whether it may read the home page and which rule decides it. It then asks the home page under each AI crawler’s name, to catch firewalls that turn bots away even when robots.txt allows them. Free, no signup.
What it checks
| Check | What it tells you |
|---|---|
| robots.txt | Whether the file exists, answers without error, and has lines crawlers ignore (typos, rules before any User-agent line) |
| 13 crawlers | Allowed or blocked on the home page, with the exact rule and line: Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent |
| Live answer | What the server answers when asked under each AI crawler’s name, compared with a normal browser request |
| Page tags | A noindex or noai in the robots meta tag or the X-Robots-Tag header |
| llms.txt | Whether the site has one |
AI search crawlers and AI training crawlers are not the same
AI companies run separate crawlers. Search crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch pages to answer a question and link to the source. Training crawlers (GPTBot, ClaudeBot, CCBot and others) collect pages to train models. A site can block training and still appear in AI answers, as long as the search crawlers are allowed. Google-Extended and Applebot-Extended are only switches in robots.txt: blocking them stops AI training use without touching Google or Apple search.
How robots.txt rules are read
- A crawler follows the group with its own name if there is one, and the
User-agent: *group only if there is not. - Inside that group, the longest matching rule wins. On a tie, Allow wins.
*matches any characters and$marks the end of the address.- No robots.txt (a 404) means everything is allowed. A robots.txt that answers with a server error makes crawlers treat the whole site as blocked.
This checker follows those rules, from the robots.txt standard (RFC 9309) and Google’s documentation. For the rest of a domain’s setup, run the Domain Health Check.
Frequently asked questions
Does blocking GPTBot keep my site out of ChatGPT answers?
No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages when a user asks. Block GPTBot and allow those two, and the site can still be cited.
robots.txt allows a crawler, but the live test says blocked. Why?
The server turned the request away before robots.txt mattered: a firewall, a CDN bot setting or a host rule that blocks crawlers by name. That setting is at your host or CDN, not in robots.txt.
Why are Googlebot and Bingbot not tested live?
Sites check that a request claiming to be Googlebot or Bingbot really comes from Google or Microsoft. A request from our server under those names is refused by many sites, so a live result would be wrong. Their robots.txt rules are still checked.
Does every crawler obey robots.txt?
robots.txt is a request, not a lock: a crawler that ignores it can still read the site. To stop a crawler for sure, block it at the server or firewall.