What robots.txt, sitemaps and llms.txt do

The Robots Exclusion Protocol is RFC 9309. The rules live in a UTF-8 text file named /robots.txt at the top level of the host ¹. Rules are grouped by user agent. A crawler picks the group that matches its product token, falls back to the * group, and applies no rules if neither exists ¹. Inside a group, path matching is case sensitive, the longest matching rule wins, allow beats disallow when two rules are equivalent, and a path that no rule matches is allowed ¹.

How the file is fetched matters as much as what it says. A 4xx answer means the file is unavailable and the crawler may access any resource. A 5xx answer or a network failure means the file is unreachable and the crawler must assume complete disallow ¹. Crawlers should parse at least 500 kibibytes and should not use a cached copy for more than 24 hours unless the file is unreachable ¹. The RFC also states plainly that these rules are not a form of access authorization, so a disallow line is a request, not a lock ¹.

A sitemap is a UTF-8 XML file whose root is urlset. Each url entry needs a loc, and lastmod, changefreq and priority are optional ². One file may hold up to 50,000 URLs and 50 MB uncompressed. Larger sites split the list and point to the pieces from a sitemap index, which can list up to 50,000 sitemaps ². A sitemap may only include URLs under its own path on the same host, and the protocol suggests placing it at the site root ². The usual way to announce it is a Sitemap line in robots.txt, which works for every user agent ². RFC 9309 does not define that line, so crawlers read it as an extension ¹.

The llms.txt proposal is a Markdown file at /llms.txt, or at any path within a site. The only required part is an H1 with the site name. A blockquote summary, some plain text and H2 sections of link lists follow, and a section named Optional marks links an agent can skip when it needs less context ³. It is a convention described on its own site rather than an RFC, so read its findings as guidance for agents, not as a rule that bodies enforce.

security.txt is RFC 9116. Web services must place it under /.well-known/, with a copy at the top level allowed for legacy compatibility, and it must be served over HTTPS as text/plain in UTF-8 ⁴. Two fields are required: Contact, and Expires exactly once. Expires should fall less than a year in the future. Encryption, Acknowledgments, Preferred-Languages, Canonical, Policy and Hiring are optional ⁴.

Our site files checker reads robots.txt, the sitemaps it lists, llms.txt, llms-full.txt, security.txt and ai.txt from a site, explains each in plain words, and can test one URL against robots.txt for a named crawler. It fetches a small fixed set of files and never crawls pages. It sets the robots access field to rules, allow_all, disallow_all_temporarily or unknown depending on what the file returned: a missing file or a 401 or 403 gives allow_all, and a 5xx or 429 gives disallow_all_temporarily.

Every finding carries a severity (error, warning or info) and the line number when there is one. A robots.txt that does not exist is reported as info, because the specification treats a missing file as open access ¹. The request below tests whether GPTBot may fetch a path, and the verified body is the same one the API accepts.

1curl -X POST https://YOUR_API_HOST/v1/site-files \
2  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{"domain":"example.com","test_url":"/private/report","user_agent":"GPTBot"}'

The reply has one block per file, an overall count of files present, errors and warnings, a list of known AI crawlers with allowed or blocked status, and a url_test block that names the deciding rule and its line. You can also try it without code in the site files checker tool.

• Fix errors first: a broken security.txt or an unreadable robots.txt changes how others treat the site.
• Treat warnings as likely mistakes, such as a sitemap listing URLs that robots.txt blocks.
• Treat info lines as notes, such as a missing optional field.

Run the check on your own domain from the quickstart with the request above, and read the list of findings before you change anything.