Get API key
Site Files Checker

Site Files Checker

Reads the files a site publishes for crawlers, SEO tools and AI agents (robots.txt, its sitemaps, llms.txt, llms-full.txt, security.txt and ai.txt) and explains each one in plain words: present or missing, parse errors and warnings, robots.txt groups and rules with a URL tester, which known AI crawlers are allowed, sitemap URL counts and families, sitemap URLs that robots.txt blocks, llms.txt structure and security.txt required fields and expiry.

Get API keyRead the docs
POST/v1/site-files
curl -X POST https://YOUR_API_HOST/v1/site-files \
  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"cloudflare.com","test_url":"/cdn-cgi/trace","user_agent":"GPTBot"}'
Read docs
  • 2 credits per site checked (a site that cannot be reached is free)
  • Charged only when it finds something
  • Cached 1 day

2 credits per site checked (a site that cannot be reached is free)

Charged only when it finds something. A repeat inside 1 day is free.

Every LookValid product draws on one shared balance. See pricing.

From key to answer in three steps

One key, one POST to /v1/site-files and a JSON answer back. The same three steps work in any language that can send HTTPS.

  1. 01

    Create a key

    Sign up and copy a key from the dashboard. The same key works for every LookValid product and for the MCP server.

    Terminal
    export LOOKVALID_API_KEY="lv_live_..."
  2. 02

    Send the request

    POST a JSON body to /v1/site-files. 2 credits per site checked (a site that cannot be reached is free).

    request.sh
    curl -X POST https://YOUR_API_HOST/v1/site-files \
  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"cloudflare.com","test_url":"/cdn-cgi/trace","user_agent":"GPTBot"}'
  3. 03

    Read the answer

    A real run of this lookup on 2026-10-18. The data object has 11 named fields.

    response.json
    {
  "product": "site-files",
  "found": true,
  "data": {
    "site": "https://cloudflare.com",
    "checked_at": "2026-10-18T17:16:47.720Z",
    "overall": {
      "summary": "5 of the 6 site files are present. 1 error and 0 warnings to look at.",
      "files_present": 5,
      "errors": 1,
      "warnings": 0
    },
    "robots": {
      "file": "robots.txt",
      "url": "https://www.cloudflare.com/robots.txt",
  ...

A structured answer, not a page of text

The data object has 11 named fields. Every response also lists the sources behind the result, the credits charged and whether it was served from your cache.

  • FieldType
    sitestring
    checked_atstring
    overallobject
    robotsobject
    sitemapsobject

    And 6 more fields in the API reference.

    Named fields. A checker for site owners and SEOs: it reads only files any visitor can open.

    1. Your requestone API key
    2. Your cache1 dayRepeat: 0 credits
    3. Data sourcefirst lookupFound: 2 credits
    4. ResultJSON

    A repeat lookup is free. Results are cached per account for 1 day. Ask the same question again inside that window and the response comes back with cache_hit: true and no charge.

  • Terminal
    # Sent with every request, for every product
Authorization: Bearer $LOOKVALID_API_KEY

    One key for your code and your agents. Send your API key as a Bearer token. The same key and the same credit balance cover every other LookValid product. AI agents can call it as a tool on the MCP server at https://YOUR_API_HOST/v1/mcp.

11

named fields in every answer

1

day a repeat lookup is free

14

lookups on the same key and balance

Latest from the changelog

All changes

Priced in credits

Every LookValid product draws on one shared balance. Site Files Checker uses 2 credits per site checked (a site that cannot be reached is free).

See pricingGet API key

Cost

2credits

Per site checked (a site that cannot be reached is free)

  • Found2 credits
  • Nothing found0 credits
  • Repeat inside 1 day0 credits
  • Errors and invalid requests0 credits

Questions about Site Files Checker

01How much does Site Files Checker cost?

2 credits per site checked (a site that cannot be reached is free). Credits come out of the one balance shared by every LookValid product.

02Am I charged when nothing is found?

No. When the lookup runs and finds nothing, it returns found: false and costs 0 credits. An error costs nothing either.

03How long are results cached?

For 1 day, per account. Repeating the same lookup inside that window returns cache_hit: true and costs 0 credits.

04Which API key do I use?

The same key as every other product. Send it as a Bearer token in the Authorization header. Email hello@lookvalid.com and we set up your account and send the key.