Site Files Checker

Open in ChatGPT Open in Claude

Reads the files a site publishes for crawlers, SEO tools and AI agents (robots.txt, its sitemaps, llms.txt, llms-full.txt, security.txt and ai.txt) and explains each one in plain words: present or missing, parse errors and warnings, robots.txt groups and rules with a URL tester, which known AI crawlers are allowed, sitemap URL counts and families, sitemap URLs that robots.txt blocks, llms.txt structure and security.txt required fields and expiry. A checker for site owners and SEOs: it reads only files any visitor can open.

POST https://YOUR_API_HOST/v1/site-files

Credits and caching

Success billing. 2 credits per site checked (a site that cannot be reached is free). You are charged only when the lookup finds something; a not-found result is free.
Results are cached per account for 0 days. Repeating the same lookup in that window is free and returns cache_hit: true.
Invalid input, rate limits and errors are never charged.
Read the billing rules for the full picture.

Request

Send a JSON body with Authorization: Bearer <key>. The body can be up to 64 KB.

Field	Type	Required	Notes
`url`	string	no	Up to 2048 characters.
`domain`	string	no	Up to 253 characters.
`test_url`	string	no	A URL or a path such as /private/page to test against robots.txt Up to 2048 characters.
`user_agent`	string	no	The crawler name for the URL tester and the sitemap cross-check. Default: * (any crawler) Pattern `^[\x21-\x7e ]+$`. Up to 200 characters.

Example request

Terminal

curl -X POST https://YOUR_API_HOST/v1/site-files \
  -H "Authorization: Bearer $LOOKVALID_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"domain":"cloudflare.com","test_url":"/cdn-cgi/trace","user_agent":"GPTBot"}'

1const response = await fetch("https://YOUR_API_HOST/v1/site-files", {
2  method: "POST",
3  headers: {
4    Authorization: `Bearer ${process.env.LOOKVALID_API_KEY}`,
5    "Content-Type": "application/json",
6  },
7  body: JSON.stringify({
8    "domain": "cloudflare.com",
9    "test_url": "/cdn-cgi/trace",
10    "user_agent": "GPTBot"
11  }),
12});
13
14const result = await response.json();
15if (!response.ok) throw new Error(`${result.error.code}: ${result.error.fix}`);
16console.log(result.found, result.data, result.credits_charged);

1import os
2import requests
3
4response = requests.post(
5    "https://YOUR_API_HOST/v1/site-files",
6    headers={"Authorization": f"Bearer {os.environ['LOOKVALID_API_KEY']}"},
7    json={
8        "domain": "cloudflare.com",
9        "test_url": "/cdn-cgi/trace",
10        "user_agent": "GPTBot",
11    },
12    timeout=30,
13)
14result = response.json()
15if not response.ok:
16    raise RuntimeError(f"{result['error']['code']}: {result['error']['fix']}")
17print(result["found"], result["data"], result["credits_charged"])

Response

The status is 200 whether or not anything was found. Check found. Every response also carries the X-Request-Id, X-Credits-Charged, X-Credits-Remaining and X-RateLimit-* headers.

Envelope

Field	Type	Notes
`request_id`	string	-
`product`	"site-files"	-
`found`	boolean	False means the lookup ran and found nothing; it is free.
`reason`	string or null	Why nothing was found; null when found.
`sources`	array of string	Where the data came from, including any attribution the licence requires.
`credits_charged`	integer	0 for a cache hit, a not-found result or an error.
`credits_remaining`	integer	-
`cache_hit`	boolean	True when the account already paid for this lookup within the cache window.
`cache`	object	-

The `data` object

Present when found is true, otherwise null.

Field	Type	Notes
`site`	string	The origin that was checked
`checked_at`	string	-
`overall`	object	-
`overall.summary`	string	-
`overall.files_present`	integer	-
`overall.errors`	integer	-
`overall.warnings`	integer	-
`robots`	object	-
`robots.file`	string	The file name, such as robots.txt
`robots.url`	string	The address that was read, after redirects
`robots.status`	string	present: a file was served; missing: the site has no such file; error: the answer could not be used One of `present`, `missing`, `error`.
`robots.http_status`	integer or null	-
`robots.bytes`	integer or null	-
`robots.content_type`	string or null	-
`robots.summary`	string	One plain sentence on what this file is doing for the site
`robots.findings`	array of object	-
`robots.findings[].severity`	string	error: the file is broken or ignored; warning: likely a mistake or a risk; info: good to know One of `error`, `warning`, `info`.
`robots.findings[].message`	string	What was found and what it means, in plain words
`robots.findings[].line`	integer or null	The line of the file the finding is about, when there is one
`robots.access`	string	How crawlers treat the site given what robots.txt returned: follow its rules, treat everything as allowed, treat everything as blocked for now, or unknown One of `rules`, `allow_all`, `disallow_all_temporarily`, `unknown`.
`robots.groups`	array of object	-
`robots.groups[].agents`	array of string	-
`robots.groups[].rules`	array of object	-
`robots.groups[].rules[].type`	string	One of `allow`, `disallow`.
`robots.groups[].rules[].pattern`	string	-
`robots.groups[].rules[].line`	integer	-
`robots.groups[].rules_truncated`	boolean	-
`robots.groups[].crawl_delay_seconds`	number or null	-
`robots.groups[].content_signals`	object or null	The Content-Signal preferences for AI services, when the group sets them
`robots.groups[].content_signals.search`	string or null	-
`robots.groups[].content_signals.ai_input`	string or null	-
`robots.groups[].content_signals.ai_train`	string or null	-
`robots.groups[].content_signals.line`	integer	-
`robots.groups[].line`	integer	-
`robots.groups[].explanation`	string	-
`robots.groups_truncated`	boolean	-
`robots.rule_count`	integer	-
`robots.sitemaps_listed`	array of string	-
`sitemaps`	object	-
`sitemaps.summary`	string	-
`sitemaps.files`	array of object	-
`sitemaps.files[].file`	string	The file name, such as robots.txt
`sitemaps.files[].url`	string	The address that was read, after redirects
`sitemaps.files[].status`	string	present: a file was served; missing: the site has no such file; error: the answer could not be used One of `present`, `missing`, `error`.
`sitemaps.files[].http_status`	integer or null	-
`sitemaps.files[].bytes`	integer or null	-
`sitemaps.files[].content_type`	string or null	-
`sitemaps.files[].summary`	string	One plain sentence on what this file is doing for the site
`sitemaps.files[].findings`	array of object	-
`sitemaps.files[].findings[].severity`	string	error: the file is broken or ignored; warning: likely a mistake or a risk; info: good to know One of `error`, `warning`, `info`.
`sitemaps.files[].findings[].message`	string	What was found and what it means, in plain words
`sitemaps.files[].findings[].line`	integer or null	The line of the file the finding is about, when there is one
`sitemaps.files[].kind`	string or null	-
`sitemaps.files[].entry_count`	integer or null	URLs in a sitemap, or sitemaps in an index
`sitemaps.files[].listed_in_robots`	boolean	-
`sitemaps.files[].found_in`	string or null	The sitemap index this file was found through
`sitemaps.files[].lastmod_count`	integer or null	-
`sitemaps.files[].extensions`	array of string	-
`sitemaps.total_urls`	integer	URLs in every sitemap that was read
`sitemaps.complete`	boolean	False when some sitemaps were not read, so total_urls is a lower bound
`sitemaps.families`	array of object	-
`sitemaps.families[].prefix`	string	-
`sitemaps.families[].urls`	integer	-
`sitemaps.families[].example`	string	-
`sitemaps.families_sampled`	boolean	True when the families count only the first URLs of large sitemaps
`sitemaps.blocked_by_robots`	object or null	-
`sitemaps.blocked_by_robots.user_agent`	string	-
`sitemaps.blocked_by_robots.checked`	integer	-
`sitemaps.blocked_by_robots.blocked`	integer	-
`sitemaps.blocked_by_robots.examples`	array of object	-
`sitemaps.blocked_by_robots.examples[].url`	string	-
`sitemaps.blocked_by_robots.examples[].rule`	string	-
`sitemaps.blocked_by_robots.explanation`	string	-
`sitemaps.findings`	array of object	-
`sitemaps.findings[].severity`	string	error: the file is broken or ignored; warning: likely a mistake or a risk; info: good to know One of `error`, `warning`, `info`.
`sitemaps.findings[].message`	string	What was found and what it means, in plain words
`sitemaps.findings[].line`	integer or null	The line of the file the finding is about, when there is one
`llms_txt`	object	-
`llms_txt.file`	string	The file name, such as robots.txt
`llms_txt.url`	string	The address that was read, after redirects
`llms_txt.status`	string	present: a file was served; missing: the site has no such file; error: the answer could not be used One of `present`, `missing`, `error`.
`llms_txt.http_status`	integer or null	-
`llms_txt.bytes`	integer or null	-
`llms_txt.content_type`	string or null	-
`llms_txt.summary`	string	One plain sentence on what this file is doing for the site
`llms_txt.findings`	array of object	-
`llms_txt.findings[].severity`	string	error: the file is broken or ignored; warning: likely a mistake or a risk; info: good to know One of `error`, `warning`, `info`.
`llms_txt.findings[].message`	string	What was found and what it means, in plain words
`llms_txt.findings[].line`	integer or null	The line of the file the finding is about, when there is one
`llms_txt.title`	string or null	-
`llms_txt.has_summary`	boolean	-
`llms_txt.sections`	array of object	-
`llms_txt.sections[].heading`	string	-
`llms_txt.sections[].links`	integer	-
`llms_txt.link_count`	integer	-
`llms_full_txt`	object	-
`llms_full_txt.file`	string	The file name, such as robots.txt
`llms_full_txt.url`	string	The address that was read, after redirects
`llms_full_txt.status`	string	present: a file was served; missing: the site has no such file; error: the answer could not be used One of `present`, `missing`, `error`.
`llms_full_txt.http_status`	integer or null	-
`llms_full_txt.bytes`	integer or null	-
`llms_full_txt.content_type`	string or null	-
`llms_full_txt.summary`	string	One plain sentence on what this file is doing for the site
`llms_full_txt.findings`	array of object	-
`llms_full_txt.findings[].severity`	string	error: the file is broken or ignored; warning: likely a mistake or a risk; info: good to know One of `error`, `warning`, `info`.
`llms_full_txt.findings[].message`	string	What was found and what it means, in plain words
`llms_full_txt.findings[].line`	integer or null	The line of the file the finding is about, when there is one
`security_txt`	object	-
`security_txt.file`	string	The file name, such as robots.txt
`security_txt.url`	string	The address that was read, after redirects
`security_txt.status`	string	present: a file was served; missing: the site has no such file; error: the answer could not be used One of `present`, `missing`, `error`.
`security_txt.http_status`	integer or null	-
`security_txt.bytes`	integer or null	-
`security_txt.content_type`	string or null	-
`security_txt.summary`	string	One plain sentence on what this file is doing for the site
`security_txt.findings`	array of object	-
`security_txt.findings[].severity`	string	error: the file is broken or ignored; warning: likely a mistake or a risk; info: good to know One of `error`, `warning`, `info`.
`security_txt.findings[].message`	string	What was found and what it means, in plain words
`security_txt.findings[].line`	integer or null	The line of the file the finding is about, when there is one
`security_txt.contacts`	array of string	-
`security_txt.expires`	string or null	-
`security_txt.days_until_expiry`	integer or null	-
`security_txt.signed`	boolean	-
`security_txt.fields`	array of string	-
`ai_txt`	object	-
`ai_txt.file`	string	The file name, such as robots.txt
`ai_txt.url`	string	The address that was read, after redirects
`ai_txt.status`	string	present: a file was served; missing: the site has no such file; error: the answer could not be used One of `present`, `missing`, `error`.
`ai_txt.http_status`	integer or null	-
`ai_txt.bytes`	integer or null	-
`ai_txt.content_type`	string or null	-
`ai_txt.summary`	string	One plain sentence on what this file is doing for the site
`ai_txt.findings`	array of object	-
`ai_txt.findings[].severity`	string	error: the file is broken or ignored; warning: likely a mistake or a risk; info: good to know One of `error`, `warning`, `info`.
`ai_txt.findings[].message`	string	What was found and what it means, in plain words
`ai_txt.findings[].line`	integer or null	The line of the file the finding is about, when there is one
`ai_crawlers`	array of object	-
`ai_crawlers[].token`	string	The name to put in a User-agent line of robots.txt
`ai_crawlers[].operator`	string	-
`ai_crawlers[].purpose`	string	One of `training`, `search`, `user_request`, `other`.
`ai_crawlers[].status`	string	allowed: nothing blocks it; partly_blocked: some paths are blocked; blocked: the whole site is blocked; unknown: robots.txt could not be read One of `allowed`, `partly_blocked`, `blocked`, `unknown`.
`ai_crawlers[].rule_source`	string	Which part of robots.txt decided One of `own_group`, `wildcard_group`, `no_group`, `no_robots`.
`ai_crawlers[].explanation`	string	-
`url_test`	object or null	-
`url_test.url`	string	-
`url_test.user_agent`	string	-
`url_test.applies`	boolean	False when the URL is on another host, which this robots.txt does not govern
`url_test.allowed`	boolean or null	Null when the answer cannot be known
`url_test.matched_rule`	string or null	-
`url_test.rule_group`	string or null	-
`url_test.explanation`	string	-

Captured example (2026-10-18)

A real run of this lookup on 2026-10-18, using the request above, which is a public input. The envelope fields are left out.

1{
2  "product": "site-files",
3  "found": true,
4  "data": {
5    "site": "https://cloudflare.com",
6    "checked_at": "2026-10-18T17:16:47.720Z",
7    "overall": {
8      "summary": "5 of the 6 site files are present. 1 error and 0 warnings to look at.",
9      "files_present": 5,
10      "errors": 1,
11      "warnings": 0
12    },
13    "robots": {
14      "file": "robots.txt",
15      "url": "https://www.cloudflare.com/robots.txt",
16      "status": "present",
17      "http_status": 200,
18      "bytes": 1099,
19      "content_type": "text/plain",
20      "summary": "robots.txt has 9 groups, 9 rules and lists 1 sitemap.",
21      "findings": [
22        {
23          "severity": "info",
24          "message": "Line 40: Content signals: search indexing allowed, use in AI answers allowed, AI training allowed. These are preferences you state for AI services. They are not enforced, and each service decides whether to honor them.",
25          "line": 40
26        }
27      ],
28      "access": "rules",
29      "groups": [
30        {
31          "agents": [
32            "*"
33          ],
34          "rules": [
35            {
36              "type": "allow",
37              "pattern": "/",
38              "line": 4
39            }
40          ],
41          "rules_truncated": false,
42          "crawl_delay_seconds": null,
43          "content_signals": null,
44          "line": 3,
45          "explanation": "For every crawler that has no group of its own: nothing is blocked."
46        },
47        {
48          "agents": [
49            "GPTBot"
50          ],
51          "rules": [
52            {
53              "type": "allow",
54              "pattern": "/",
55              "line": 15
56            }
57          ],
58          "rules_truncated": false,
59          "crawl_delay_seconds": null,
60          "content_signals": null,
61          "line": 14,
62          "explanation": "For GPTBot: nothing is blocked."
63        },
64        {
65          "agents": [
66            "ChatGPT-User"
67          ],
68          "rules": [
69            {
70              "type": "allow",
71              "pattern": "/",
72              "line": 18
73            }
74          ],
75          "rules_truncated": false,
76          "crawl_delay_seconds": null,
77          "content_signals": null,
78          "line": 17,
79          "explanation": "For ChatGPT-User: nothing is blocked."
80        },
81        {
82          "agents": [
83            "Google-Extended"
84          ],
85          "rules": [
86            {
87              "type": "allow",
88              "pattern": "/",
89              "line": 21
90            }
91          ],
92          "rules_truncated": false,
93          "crawl_delay_seconds": null,
94          "content_signals": null,
95          "line": 20,
96          "explanation": "For Google-Extended: nothing is blocked."
97        },
98        {
99          "agents": [
100            "Anthropic-AI"
101          ],
102          "rules": [
103            {
104              "type": "allow",
105              "pattern": "/",
106              "line": 24
107            }
108          ],
109          "rules_truncated": false,
110          "crawl_delay_seconds": null,
111          "content_signals": null,
112          "line": 23,
113          "explanation": "For Anthropic-AI: nothing is blocked."
114        },
115        {
116          "agents": [
117            "Claude-Web"
118          ],
119          "rules": [
120            {
121              "type": "allow",
122              "pattern": "/",
123              "line": 27
124            }
125          ],
126          "rules_truncated": false,
127          "crawl_delay_seconds": null,
128          "content_signals": null,
129          "line": 26,
130          "explanation": "For Claude-Web: nothing is blocked."
131        },
132        {
133          "agents": [
134            "CCBot"
135          ],
136          "rules": [
137            {
138              "type": "allow",
139              "pattern": "/",
140              "line": 30
141            }
142          ],
143          "rules_truncated": false,
144          "crawl_delay_seconds": null,
145          "content_signals": null,
146          "line": 29,
147          "explanation": "For CCBot: nothing is blocked."
148        },
149        {
150          "agents": [
151            "PerplexityBot"
152          ],
153          "rules": [
154            {
155              "type": "allow",
156              "pattern": "/",
157              "line": 33
158            }
159          ],
160          "rules_truncated": false,
161          "crawl_delay_seconds": null,
162          "content_signals": null,
163          "line": 32,
164          "explanation": "For PerplexityBot: nothing is blocked."
165        },
166        {
167          "agents": [
168            "Cohere-ai"
169          ],
170          "rules": [
171            {
172              "type": "allow",
173              "pattern": "/",
174              "line": 36
175            }
176          ],
177          "rules_truncated": false,
178          "crawl_delay_seconds": null,
179          "content_signals": {
180            "search": "yes",
181            "ai_input": "yes",
182            "ai_train": "yes",
183            "line": 40
184          },
185          "line": 35,
186          "explanation": "For Cohere-ai: nothing is blocked. Content signals: search indexing allowed, use in AI answers allowed, AI training allowed."
187        }
188      ],
189      "groups_truncated": false,
190      "rule_count": 9,
191      "sitemaps_listed": [
192        "https://www.cloudflare.com/sitemap.xml"
193      ]
194    },
195    "sitemaps": {
196      "summary": "1 sitemap file read, listing 1,010 URLs.",
197      "files": [
198        {
199          "file": "sitemap.xml",
200          "url": "https://www.cloudflare.com/sitemap.xml",
201          "status": "present",
202          "http_status": 200,
203          "bytes": 1628250,
204          "content_type": "application/xml",
205          "summary": "sitemap.xml lists 1,010 URLs.",
206          "findings": [],
207          "kind": "urlset",
208          "entry_count": 1010,
209          "listed_in_robots": true,
210          "found_in": null,
211          "lastmod_count": 1010,
212          "extensions": [
213            "alternate languages"
214          ]
215        }
216      ],
217      "total_urls": 1010,
218      "complete": true,
219      "families": [
220        {
221          "prefix": "/learning/",
222          "urls": 524,
223          "example": "https://www.cloudflare.com/learning/security/account-takeover"
224        },
225        {
226          "prefix": "/case-studies/",
227          "urls": 265,
228          "example": "https://www.cloudflare.com/case-studies/123milhas"
229        },
230        {
231          "prefix": "/perspectives/",
232          "urls": 100,
233          "example": "https://www.cloudflare.com/perspectives/whats-the-best-edge-platform-for-running-a-headless-cms-backend"
234        },
235        {
236          "prefix": "/products/",
237          "urls": 70,
238          "example": "https://www.cloudflare.com/products/artifacts"
239        },
240        {
241          "prefix": "/",
242          "urls": 27,
243          "example": "https://www.cloudflare.com/"
244        },
245        {
246          "prefix": "/resource/",
247          "urls": 13,
248          "example": "https://www.cloudflare.com/resource/app-innovation-report"
249        },
250        {
251          "prefix": "/solutions/",
252          "urls": 7,
253          "example": "https://www.cloudflare.com/solutions/ai"
254        },
255        {
256          "prefix": "/policies/",
257          "urls": 2,
258          "example": "https://www.cloudflare.com/policies/privacy"
259        },
260        {
261          "prefix": "/careers/",
262          "urls": 1,
263          "example": "https://www.cloudflare.com/careers/jobs"
264        },
265        {
266          "prefix": "/partners/",
267          "urls": 1,
268          "example": "https://www.cloudflare.com/partners/cloud-and-platform/oracle"
269        }
270      ],
271      "families_sampled": false,
272      "blocked_by_robots": {
273        "user_agent": "GPTBot",
274        "checked": 1010,
275        "blocked": 0,
276        "examples": [],
277        "explanation": "None of the 1,010 sitemap URLs tested is blocked by robots.txt for GPTBot."
278      },
279      "findings": []
280    },
281    "llms_txt": {
282      "file": "llms.txt",
283      "url": "https://www.cloudflare.com/llms.txt",
284      "status": "present",
285      "http_status": 200,
286      "bytes": 17173,
287      "content_type": "text/plain",
288      "summary": "llms.txt gives AI agents 90 links in 13 sections.",
289      "findings": [
290        {
291          "severity": "info",
292          "message": "The \"Optional\" section marks links an agent may skip when its context window is short. That is what the proposal intends.",
293          "line": null
294        }
295      ],
296      "title": "Cloudflare",
297      "has_summary": true,
298      "sections": [
299        {
300          "heading": "About Cloudflare",
301          "links": 0
302        },
303        {
304          "heading": "Developer Platform - Compute",
305          "links": 11
306        },
307        {
308          "heading": "Developer Platform - Storage",
309          "links": 7
310        },
311        {
312          "heading": "Developer Platform - AI",
313          "links": 6
314        },
315        {
316          "heading": "Developer Platform - Media",
317          "links": 4
318        },
319        {
320          "heading": "Application Security",
321          "links": 14
322        },
323        {
324          "heading": "Zero Trust / SASE (Cloudflare One)",
325          "links": 11
326        },
327        {
328          "heading": "Network & Performance",
329          "links": 14
330        },
331        {
332          "heading": "Solutions",
333          "links": 10
334        },
335        {
336          "heading": "Pricing",
337          "links": 3
338        },
339        {
340          "heading": "Other Pages",
341          "links": 2
342        },
343        {
344          "heading": "External Resources",
345          "links": 7
346        },
347        {
348          "heading": "Optional",
349          "links": 1
350        }
351      ],
352      "link_count": 90
353    },
354    "llms_full_txt": {
355      "file": "llms-full.txt",
356      "url": "https://www.cloudflare.com/llms-full.txt",
357      "status": "present",
358      "http_status": 200,
359      "bytes": 169547,
360      "content_type": "text/plain",
361      "summary": "llms-full.txt holds 166 KB of text with 781 headings.",
362      "findings": []
363    },
364    "security_txt": {
365      "file": "security.txt",
366      "url": "https://www.cloudflare.com/.well-known/security.txt",
367      "status": "present",
368      "http_status": 200,
369      "bytes": 368,
370      "content_type": "text/plain",
371      "summary": "security.txt exists but has 1 error, so researchers may not be able to use it.",
372      "findings": [
373        {
374          "severity": "error",
375          "message": "There is no Expires field. It is required, so that readers can tell when the contact details went stale.",
376          "line": null
377        }
378      ],
379      "contacts": [
380        "https://hackerone.com/cloudflare",
381        "https://www.cloudflare.com/abuse/"
382      ],
383      "expires": null,
384      "days_until_expiry": null,
385      "signed": false,
386      "fields": [
387        "Contact",
388        "Policy",
389        "Hiring",
390        "Preferred-Languages",
391        "Canonical"
392      ]
393    },
394    "ai_txt": {
395      "file": "ai.txt",
396      "url": "https://www.cloudflare.com/ai.txt",
397      "status": "missing",
398      "http_status": 404,
399      "bytes": null,
400      "content_type": null,
401      "summary": "ai.txt is missing. It is not a standard and nothing depends on it, so having none is normal.",
402      "findings": []
403    },
404    "ai_crawlers": [
405      {
406        "token": "GPTBot",
407        "operator": "OpenAI",
408        "purpose": "training",
409        "status": "allowed",
410        "rule_source": "own_group",
411        "explanation": "OpenAI's GPTBot (AI training) collects pages that may be used to train OpenAI models. It has its own group in robots.txt, and nothing in it is blocked."
412      },
413      {
414        "token": "OAI-SearchBot",
415        "operator": "OpenAI",
416        "purpose": "search",
417        "status": "allowed",
418        "rule_source": "wildcard_group",
419        "explanation": "OpenAI's OAI-SearchBot (AI search) finds pages to show in ChatGPT search results. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
420      },
421      {
422        "token": "ChatGPT-User",
423        "operator": "OpenAI",
424        "purpose": "user_request",
425        "status": "allowed",
426        "rule_source": "own_group",
427        "explanation": "OpenAI's ChatGPT-User (a user request) opens a page when a ChatGPT user asks about it. It has its own group in robots.txt, and nothing in it is blocked."
428      },
429      {
430        "token": "ClaudeBot",
431        "operator": "Anthropic",
432        "purpose": "training",
433        "status": "allowed",
434        "rule_source": "wildcard_group",
435        "explanation": "Anthropic's ClaudeBot (AI training) collects pages that may be used to train Claude models. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
436      },
437      {
438        "token": "Claude-SearchBot",
439        "operator": "Anthropic",
440        "purpose": "search",
441        "status": "allowed",
442        "rule_source": "wildcard_group",
443        "explanation": "Anthropic's Claude-SearchBot (AI search) finds pages to improve search results in Claude. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
444      },
445      {
446        "token": "Claude-User",
447        "operator": "Anthropic",
448        "purpose": "user_request",
449        "status": "allowed",
450        "rule_source": "wildcard_group",
451        "explanation": "Anthropic's Claude-User (a user request) opens a page when a Claude user asks about it. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
452      },
453      {
454        "token": "PerplexityBot",
455        "operator": "Perplexity",
456        "purpose": "search",
457        "status": "allowed",
458        "rule_source": "own_group",
459        "explanation": "Perplexity's PerplexityBot (AI search) finds pages to show in Perplexity answers. It has its own group in robots.txt, and nothing in it is blocked."
460      },
461      {
462        "token": "Perplexity-User",
463        "operator": "Perplexity",
464        "purpose": "user_request",
465        "status": "allowed",
466        "rule_source": "wildcard_group",
467        "explanation": "Perplexity's Perplexity-User (a user request) opens a page when a Perplexity user asks about it. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
468      },
469      {
470        "token": "Google-Extended",
471        "operator": "Google",
472        "purpose": "training",
473        "status": "allowed",
474        "rule_source": "own_group",
475        "explanation": "Google's Google-Extended (AI training) a switch that controls whether Google may use your pages for Gemini training and grounding. It has its own group in robots.txt, and nothing in it is blocked."
476      },
477      {
478        "token": "Applebot-Extended",
479        "operator": "Apple",
480        "purpose": "training",
481        "status": "allowed",
482        "rule_source": "wildcard_group",
483        "explanation": "Apple's Applebot-Extended (AI training) a switch that controls whether Apple may use your pages to train its AI models. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
484      },
485      {
486        "token": "Meta-ExternalAgent",
487        "operator": "Meta",
488        "purpose": "training",
489        "status": "allowed",
490        "rule_source": "wildcard_group",
491        "explanation": "Meta's Meta-ExternalAgent (AI training) collects pages that may be used to train Meta AI models. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
492      },
493      {
494        "token": "Meta-ExternalFetcher",
495        "operator": "Meta",
496        "purpose": "user_request",
497        "status": "allowed",
498        "rule_source": "wildcard_group",
499        "explanation": "Meta's Meta-ExternalFetcher (a user request) opens a page when a Meta AI user asks about it. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
500      },
501      {
502        "token": "CCBot",
503        "operator": "Common Crawl",
504        "purpose": "training",
505        "status": "allowed",
506        "rule_source": "own_group",
507        "explanation": "Common Crawl's CCBot (AI training) builds the open Common Crawl archive that many AI models train on. It has its own group in robots.txt, and nothing in it is blocked."
508      },
509      {
510        "token": "Amazonbot",
511        "operator": "Amazon",
512        "purpose": "other",
513        "status": "allowed",
514        "rule_source": "wildcard_group",
515        "explanation": "Amazon's Amazonbot (other AI use) collects pages for Amazon services such as Alexa answers. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
516      },
517      {
518        "token": "Bytespider",
519        "operator": "ByteDance",
520        "purpose": "training",
521        "status": "allowed",
522        "rule_source": "wildcard_group",
523        "explanation": "ByteDance's Bytespider (AI training) collects pages for ByteDance AI models. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
524      },
525      {
526        "token": "DuckAssistBot",
527        "operator": "DuckDuckGo",
528        "purpose": "search",
529        "status": "allowed",
530        "rule_source": "wildcard_group",
531        "explanation": "DuckDuckGo's DuckAssistBot (AI search) finds pages for DuckDuckGo AI-assisted answers. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
532      },
533      {
534        "token": "MistralAI-User",
535        "operator": "Mistral",
536        "purpose": "user_request",
537        "status": "allowed",
538        "rule_source": "wildcard_group",
539        "explanation": "Mistral's MistralAI-User (a user request) opens a page when a Le Chat user asks about it. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
540      },
541      {
542        "token": "cohere-ai",
543        "operator": "Cohere",
544        "purpose": "training",
545        "status": "allowed",
546        "rule_source": "own_group",
547        "explanation": "Cohere's cohere-ai (AI training) collects pages for Cohere models. It has its own group in robots.txt, and nothing in it is blocked."
548      },
549      {
550        "token": "Diffbot",
551        "operator": "Diffbot",
552        "purpose": "training",
553        "status": "allowed",
554        "rule_source": "wildcard_group",
555        "explanation": "Diffbot's Diffbot (AI training) collects pages for the Diffbot knowledge graph. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
556      },
557      {
558        "token": "YouBot",
559        "operator": "You.com",
560        "purpose": "search",
561        "status": "allowed",
562        "rule_source": "wildcard_group",
563        "explanation": "You.com's YouBot (AI search) finds pages for You.com AI search. It has no group of its own, so it follows the \"*\" group, and nothing in it is blocked."
564      }
565    ],
566    "url_test": {
567      "url": "https://cloudflare.com/cdn-cgi/trace",
568      "user_agent": "GPTBot",
569      "applies": true,
570      "allowed": true,
571      "matched_rule": "Allow: /",
572      "rule_group": "the group for GPTBot",
573      "explanation": "Crawling this URL is allowed for GPTBot. The deciding rule is \"Allow: /\" (line 15) in the group for GPTBot. When several rules match, the longest one wins, and Allow wins a tie."
574    }
575  },
576  "sources": [
577    "Public site files served by cloudflare.com, read on 2026-10-18"
578  ]
579}

Errors

400 invalid_input: The request input is invalid.
401 unauthorized: The API key is missing, malformed, revoked or unknown.
402 insufficient_credits: The account does not have enough credits for this lookup. Error code: insufficient_credits. During a trial whose allowance is used up the code is trial_allowance_exhausted: nothing is charged and the customer starts the plan from the billing page.
405 method_not_allowed: This endpoint does not accept that HTTP method.
413 payload_too_large: The request body is too large.
429 rate_limited: Too many requests for this API key.
500 internal_error: An unexpected error occurred.
503 provider_unavailable: The data source for this product is unavailable right now.