How the check works
We ask the site for a short, fixed list of public files: robots.txt, sitemap.xml and every sitemap that robots.txt lists (a sitemap index is followed into its sitemaps, up to ten files in all), llms.txt, llms-full.txt, security.txt and ai.txt. We never open the pages of the site.
Each file is read against its specification: RFC 9309 for robots.txt, sitemaps.org for sitemaps, RFC 9116 for security.txt and the llmstxt.org proposal for llms.txt. Every finding says what it means in plain words.
The URL test applies the rule Google documents for robots.txt: the longest matching rule wins, and Allow wins a tie. The result names the rule and the group it came from.
Reading the result
An error means a file is broken or will be ignored. A warning is a likely mistake or a risk. A note is only good to know. Fix errors first.
The AI crawler table shows what robots.txt tells each known AI crawler. Blocked means the whole site is blocked for it, partly blocked means some paths are, and allowed means nothing stops it. Several companies run separate crawlers for training, search and user requests, so you can allow one and block another.
A web page served with a success code where a file should be is counted as a missing file, because a crawler cannot use it.
Limits worth knowing
robots.txt is a request that well-behaved crawlers follow. It is not access control, and it does not hide a page from anyone who has the address. To keep a page out of search results, use a noindex tag instead of a Disallow rule.
llms.txt and ai.txt are proposals, not adopted standards, and the large search engines have not said they read them. We report what the file contains, not whether any service uses it.
We see the files as an anonymous visitor does. A site that shows a different robots.txt to some crawlers cannot be checked this way, and a file you just changed may take a little while to show up here.
Frequently asked questions
Can I check a site I do not own?
Yes. Every file here is public and anyone can open it in a browser. The checker is made for site owners and SEOs reviewing their own site or a client's, and it reads only those files, never the pages.
Which AI crawlers are checked?
GPTBot, OAI-SearchBot and ChatGPT-User from OpenAI, ClaudeBot, Claude-SearchBot and Claude-User from Anthropic, PerplexityBot and Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Meta-ExternalFetcher, CCBot, Amazonbot, Bytespider, DuckAssistBot, MistralAI-User, cohere-ai, Diffbot and YouBot.
Should I block AI crawlers?
That is your decision. Blocking a training crawler does not remove you from search results, and blocking a search crawler can lower how often AI assistants cite you. The table shows what your file says today, so you can compare it with what you intend.
Why does it say a file is missing when I can open it in a browser?
We ask for the standard address, such as /robots.txt on the exact host you entered. A file on another host, in another folder, or one that your server answers with a web page instead of the file counts as missing.
Do I need llms.txt?
No. It is a proposal for giving AI agents a short index of your site. It costs little to add, but no search engine has said it uses the file.