---
title: "How to use Site Files Checker"
url: "https://lookvalid.com/help/products/site-files.md"
canonical_url: "https://lookvalid.com/help/products/site-files"
last_updated: "2026-10-22"
type: "how-to"
summary: "What Site Files Checker takes in, which fields it returns and how credits are charged for each lookup."
related: ["https://lookvalid.com/help/products/domain-authority.md", "https://lookvalid.com/help/products/domain-age.md", "https://lookvalid.com/help/products/website-scraper.md", "https://lookvalid.com/help/billing/plans-and-credits.md"]
---

# How to use Site Files Checker

What Site Files Checker takes in, which fields it returns and how credits are charged for each lookup.

## What it does

Reads the files a site publishes for crawlers, SEO tools and AI agents (robots.txt, its sitemaps, llms.txt, llms-full.txt, security.txt and ai.txt) and explains each one in plain words: present or missing, parse errors and warnings, robots.txt groups and rules with a URL tester, which known AI crawlers are allowed, sitemap URL counts and families, sitemap URLs that robots.txt blocks, llms.txt structure and security.txt required fields and expiry. A checker for site owners and SEOs: it reads only files any visitor can open.

## What you send

Send these fields in the JSON body of the request.

No single field is required on its own, but a request with none of them is rejected: Provide url or domain.

Request fields for Site Files Checker

| Field | Type | Required | Limits |
| --- | --- | --- | --- |
| url | string | Optional | up to 2,048 characters |
| domain | string | Optional | up to 253 characters |
| test_url | string | Optional | up to 2,048 characters |
| user_agent | string | Optional | up to 200 characters, fixed format |

## What you get back

The `data` object of a successful response carries these fields. Every response also lists its `sources`, the `credits_charged` and whether it was a `cache_hit`.

Each field is described in the [API reference](https://lookvalid.com/docs/api/site-files.md).

Response fields for Site Files Checker

| Field | Type |
| --- | --- |
| site | string |
| checked_at | string |
| overall | object |
| robots | object |
| sitemaps | object |
| llms_txt | object |
| llms_full_txt | object |
| security_txt | object |
| ai_txt | object |
| ai_crawlers | list |
| url_test | object or null |

## What it costs

2 credits per site checked (a site that cannot be reached is free).

What each outcome costs

| Case | Charge |
| --- | --- |
| Found | 2 credits |
| Nothing found | 0 credits |
| Repeat inside 1 day | 0 credits |
| Errors and invalid requests | 0 credits |

## Run it

You can try it without an account in the [free tool](https://lookvalid.com/tools.md). The free tools share a limit per visitor.

From code, send a POST request to `https://YOUR_API_HOST/v1/site-files` with your API key.

A request to Site Files Checker

```bash
curl -X POST https://YOUR_API_HOST/v1/site-files \
  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"cloudflare.com","test_url":"/cdn-cgi/trace","user_agent":"GPTBot"}'
```

Full parameters and response fields are in the [API reference](https://lookvalid.com/docs/api/site-files.md). To run a list of values, use a [bulk CSV job](https://lookvalid.com/help/api/bulk-csv-jobs.md).

## Good to know

- A repeat of the same lookup within 1 day is served from your cache and costs 0 credits.
- When the lookup finds nothing it returns `found: false` and costs 0 credits.
- Invalid input and errors are never charged.
- The API allows 200 requests per 10 seconds per key.

## Related

- [How to use Domain Authority](https://lookvalid.com/help/products/domain-authority.md)
- [How to use Domain Age](https://lookvalid.com/help/products/domain-age.md)
- [How to use Website Scraper](https://lookvalid.com/help/products/website-scraper.md)
- [Plans, credits and how they are spent](https://lookvalid.com/help/billing/plans-and-credits.md)