---
title: "Free robots.txt, sitemap and llms.txt checker"
url: "https://lookvalid.com/tools/site-files-checker.md"
canonical_url: "https://lookvalid.com/tools/site-files-checker"
last_updated: "2026-10-18"
type: "tool"
summary: "Check a website's robots.txt, sitemap.xml, llms.txt, llms-full.txt, security.txt and ai.txt for free."
related: ["https://lookvalid.com/docs/api/site-files.md"]
---

# Free robots.txt, sitemap and llms.txt checker

Check a website's robots.txt, sitemap.xml, llms.txt, llms-full.txt, security.txt and ai.txt for free.

Enter a website. We read the files it publishes for crawlers, search engines and AI agents, and explain in plain words what each one does, what is wrong with it and which AI crawlers it lets in.

## Facts

- Free, no account needed. Rate limited per visitor.
- Runs the `site-files` product. The same lookup through the API costs 2 credits.
- API reference: [site-files](https://lookvalid.com/docs/api/site-files.md)

We ask the site for a short, fixed list of public files: robots.txt, sitemap.xml and every sitemap that robots.txt lists (a sitemap index is followed into its sitemaps, up to ten files in all), llms.txt, llms-full.txt, security.txt and ai.txt. We never open the pages of the site.

Each file is read against its specification: RFC 9309 for robots.txt, sitemaps.org for sitemaps, RFC 9116 for security.txt and the llmstxt.org proposal for llms.txt. Every finding says what it means in plain words.

The URL test applies the rule Google documents for robots.txt: the longest matching rule wins, and Allow wins a tie. The result names the rule and the group it came from.

## Related

- [Site Files Checker reference](https://lookvalid.com/docs/api/site-files.md)