Robots.txt Checker

Free, no signup. Reads robots.txt and asks the home page as each AI crawler.

This robots.txt checker reads a site’s robots.txt and tells you, for each search and AI crawler, whether it may read the home page and which rule decides it. It then asks the home page under each AI crawler’s name, to catch firewalls that turn bots away even when robots.txt allows them. Free, no signup.

What it checks

CheckWhat it tells you
robots.txtWhether the file exists, answers without error, and has lines crawlers ignore (typos, rules before any User-agent line)
13 crawlersAllowed or blocked on the home page, with the exact rule and line: Googlebot, Bingbot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, Meta-ExternalAgent
Live answerWhat the server answers when asked under each AI crawler’s name, compared with a normal browser request
Page tagsA noindex or noai in the robots meta tag or the X-Robots-Tag header
llms.txtWhether the site has one

AI search crawlers and AI training crawlers are not the same

AI companies run separate crawlers. Search crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) fetch pages to answer a question and link to the source. Training crawlers (GPTBot, ClaudeBot, CCBot and others) collect pages to train models. A site can block training and still appear in AI answers, as long as the search crawlers are allowed. Google-Extended and Applebot-Extended are only switches in robots.txt: blocking them stops AI training use without touching Google or Apple search.

How robots.txt rules are read

  • A crawler follows the group with its own name if there is one, and the User-agent: * group only if there is not.
  • Inside that group, the longest matching rule wins. On a tie, Allow wins.
  • * matches any characters and $ marks the end of the address.
  • No robots.txt (a 404) means everything is allowed. A robots.txt that answers with a server error makes crawlers treat the whole site as blocked.

This checker follows those rules, from the robots.txt standard (RFC 9309) and Google’s documentation. For the rest of a domain’s setup, run the Domain Health Check.

Frequently asked questions

Does blocking GPTBot keep my site out of ChatGPT answers?

No. GPTBot collects pages for training. ChatGPT search uses OAI-SearchBot, and ChatGPT-User opens pages when a user asks. Block GPTBot and allow those two, and the site can still be cited.

robots.txt allows a crawler, but the live test says blocked. Why?

The server turned the request away before robots.txt mattered: a firewall, a CDN bot setting or a host rule that blocks crawlers by name. That setting is at your host or CDN, not in robots.txt.

Why are Googlebot and Bingbot not tested live?

Sites check that a request claiming to be Googlebot or Bingbot really comes from Google or Microsoft. A request from our server under those names is refused by many sites, so a live result would be wrong. Their robots.txt rules are still checked.

Does every crawler obey robots.txt?

robots.txt is a request, not a lock: a crawler that ignores it can still read the site. To stop a crawler for sure, block it at the server or firewall.