How to use Site Files Checker

Reads the files a site publishes for crawlers, SEO tools and AI agents (robots.txt, its sitemaps, llms.txt, llms-full.txt, security.txt and ai.txt) and explains each one in plain words: present or missing, parse errors and warnings, robots.txt groups and rules with a URL tester, which known AI crawlers are allowed, sitemap URL counts and families, sitemap URLs that robots.txt blocks, llms.txt structure and security.txt required fields and expiry. A checker for site owners and SEOs: it reads only files any visitor can open.

Send these fields in the JSON body of the request.

No single field is required on its own, but a request with none of them is rejected: Provide url or domain.

The data object of a successful response carries these fields. Every response also lists its sources, the credits_charged and whether it was a cache_hit.

Each field is described in the API reference.

2 credits per site checked (a site that cannot be reached is free).

You can try it without an account in the free tool. The free tools share a limit per visitor.

From code, send a POST request to https://YOUR_API_HOST/v1/site-files with your API key.

1curl -X POST https://YOUR_API_HOST/v1/site-files \
2  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
3  -H "Content-Type: application/json" \
4  -d '{"domain":"cloudflare.com","test_url":"/cdn-cgi/trace","user_agent":"GPTBot"}'

Full parameters and response fields are in the API reference. To run a list of values, use a bulk CSV job.

• A repeat of the same lookup within 1 day is served from your cache and costs 0 credits.
• When the lookup finds nothing it returns found: false and costs 0 credits.
• Invalid input and errors are never charged.
• The API allows 200 requests per 10 seconds per key.