---
title: "Site Files Checker API"
url: "https://lookvalid.com/products/site-files.md"
canonical_url: "https://lookvalid.com/products/site-files"
last_updated: "2026-10-22"
type: "product"
summary: "Reads the files a site publishes for crawlers, SEO tools and AI agents."
related: ["https://lookvalid.com/pricing.md", "https://lookvalid.com/docs/api/site-files.md"]
---

# Site Files Checker API

Reads the files a site publishes for crawlers, SEO tools and AI agents.

Reads the files a site publishes for crawlers, SEO tools and AI agents (robots.txt, its sitemaps, llms.txt, llms-full.txt, security.txt and ai.txt) and explains each one in plain words: present or missing, parse errors and warnings, robots.txt groups and rules with a URL tester, which known AI crawlers are allowed, sitemap URL counts and families, sitemap URLs that robots.txt blocks, llms.txt structure and security.txt required fields and expiry. A checker for site owners and SEOs: it reads only files any visitor can open.

## Facts

- Endpoint: `POST https://YOUR_API_HOST/v1/site-files`
- Price: 2 credits per site checked (a site that cannot be reached is free)
- Charged: Only when a lookup finds something
- Cache: repeating the same lookup within 0 days is free

## Example request

```sh
curl -X POST https://YOUR_API_HOST/v1/site-files \
  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"example.com","test_url":"/private/page","user_agent":"GPTBot"}'
```

## Response

The `data` object has 11 named fields. Every response also lists the sources behind the result, the credits charged and whether it was served from your cache.

| Field | Type | Description |
| --- | --- | --- |
| `site` | string | The origin that was checked |
| `checked_at` | string |  |
| `overall` | object |  |
| `robots` | object |  |
| `sitemaps` | object |  |
| `llms_txt` | object |  |
| `llms_full_txt` | object |  |
| `security_txt` | object |  |
| `ai_txt` | object |  |
| `ai_crawlers` | array of object |  |
| `url_test` | object or null |  |

## Priced in credits

| Case | Charge |
| --- | --- |
| Found | 2 credits |
| Nothing found | 0 credits |
| Repeat inside 1 day | 0 credits |
| Errors and invalid requests | 0 credits |

## Questions

### How much does Site Files Checker cost?

2 credits per site checked (a site that cannot be reached is free). Credits come out of the one balance shared by every LookValid product.

### Am I charged when nothing is found?

No. When the lookup runs and finds nothing, it returns found: false and costs 0 credits. An error costs nothing either.

### How long are results cached?

For 1 day, per account. Repeating the same lookup inside that window returns cache_hit: true and costs 0 credits.

### Which API key do I use?

The same key as every other product. Send it as a Bearer token in the Authorization header. Email hello@lookvalid.com and we set up your account and send the key.

## Related products

- [Domain Authority](https://lookvalid.com/products/domain-authority.md): A 0 to 100 authority score for a domain, with every input shown: its Common Crawl web graph harmonic centrality and PageRank positions and its Majestic Million rank and referring subnets.
- [Domain Age](https://lookvalid.com/products/domain-age.md): Registration, last update and expiry dates, registrar and age in days for a domain, from the registry's public RDAP service.
- [Website Scraper](https://lookvalid.com/products/website-scraper.md): Fetches a public web page and returns its title, metadata, readable text and links.

Full request and response fields: [Site Files Checker reference](https://lookvalid.com/docs/api/site-files.md).

## Related

- [Plans, trial and credit pricing](https://lookvalid.com/pricing.md)
- [Site Files Checker reference](https://lookvalid.com/docs/api/site-files.md)