Free tool
Robots.txt tester: can Google and AI bots read your page?
Enter any page and we fetch its site's robots.txt, then show whether 15 crawlers, from Googlebot to ChatGPT search, Claude and Perplexity, can read it, and which line decides. Edit the file and retest before you change anything live.
Free, no sign-up. We fetch the file from our server the way a crawler would. The tests run in your browser.
Where the numbers come from
A robots.txt tester works out, for each crawler, whether a URL is allowed. A crawler follows the most specific group that names it, or the * group if none does. In that group the longest matching rule wins, and on a tie Google picks allow.
We follow Google's documented rules, which match the robots.txt standard (RFC 9309): * matches anything, a $ at the end marks the end of the URL, and Google ignores anything after the first 500 KiB of the file.
How the server answers matters too. If the file is missing (a 4xx error), Google treats the site as open. If the server errors (5xx), Google stops crawling for the first 12 hours and keeps retrying.
The AI companies split their bots by job. Search bots decide whether you can be shown and cited in their answers, training bots decide whether your pages can train their models. Blocking GPTBot keeps you out of OpenAI's training but not out of ChatGPT search, which uses OAI-SearchBot.
Bots that fetch a page because a user asked work differently. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them.
- Google Search Central: How Google interprets the robots.txt specification
- RFC 9309: Robots Exclusion Protocol
- Google: Common crawlers, including Google-Extended
- OpenAI: Overview of OpenAI crawlers
- Anthropic: How site owners can manage Anthropic's crawlers
- Perplexity: Perplexity crawlers
- Apple: About Applebot
- Meta: Web crawlers
Written and maintained by the SEO Engico team
Two mechanical engineers turned marketers, and the people who do the work. Meet the team
Updated 29 October 2026
What to do with your result
AI search bots are blocked
Allow OAI-SearchBot, Claude-SearchBot and PerplexityBot if you want to be cited in their answers. You can still block the training bots.
See AI search visibilityGooglebot is blocked by mistake
A Disallow: / left over from a staging site is the classic one. Remove it, then ask Google to recrawl the page in Search Console.
Book a search performance auditEverything's open, still not cited
Being readable is the first step. Check whether ChatGPT, Perplexity and Gemini actually cite you, and who they cite instead.
Try the AI Visibility CheckerA worked example
Say your file has a * group with Disallow: /checkout/, and a GPTBot group with Disallow: /.
For /blog/boiler-guide, Googlebot follows the * group. Nothing in it matches the page, so it's allowed. GPTBot has its own group, where Disallow: / matches every page, so it's blocked.
OAI-SearchBot has no group of its own, so it follows * and is allowed. The page stays in ChatGPT search while staying out of OpenAI's training.
Questions, answered
How do I test my robots.txt?
Enter any page on your site above. We fetch the site's robots.txt, show it, and test that page against every crawler in the list. Change the file in the box and the results update straight away, so you can check an edit before it goes live.
Doesn't Search Console do this?
Partly. Its robots.txt report shows the files Google found and any errors, and URL Inspection tells you whether Googlebot can crawl a page on your own site. Neither covers other crawlers or lets you try an edit first.
Should I block AI crawlers?
It depends on what you want. To be cited by ChatGPT, Claude and Perplexity, allow their search bots. To keep your content out of training, block the training bots. They're separate switches, so you can do both.
Does robots.txt stop a page appearing in Google?
No. It stops crawling, not indexing. A blocked page can still show in results if other sites link to it, just without a description. To keep a page out of Google, let it be crawled and add a noindex tag.
Why does a bot show as allowed when I blocked it?
A crawler follows one group only. If you give it its own group, it ignores the * group completely, so rules you put under * no longer apply to it. Check the spelling of the user-agent line too.
Does Google follow crawl-delay?
No, Google ignores it. Bing and some other crawlers, including Anthropic's, do follow it.
More free tools
See all free toolsGoogle review link generator
The link and QR code that open your Google review box, plus texts to send customers.
No sign-upOpen toolSEORedirect checker
Follow every redirect from a URL, see each status code, and check all four versions of your domain.
No sign-upOpen toolSEOSchema markup generator
LocalBusiness and Organization JSON-LD for your site, built to Google's own field list.
No sign-upOpen toolWant AI search to cite you, not just read you?
We make sure the right bots can reach you, then get you onto the pages ChatGPT, Perplexity and Google's AI already trust.