--- title: "Find hidden pages from robots.txt" url: "https://lookvalid.com/blog/technical/find-hidden-pages-from-robots-txt.md" canonical_url: "https://lookvalid.com/blog/technical/find-hidden-pages-from-robots-txt" last_updated: "2026-10-24" type: "conceptual" summary: "Every Disallow line in a robots.txt names a path. Open each one once, see which still answer any visitor, and learn what to fix on your own site." related: ["https://lookvalid.com/tools/site-files-checker.md", "https://lookvalid.com/blog/technical/reading-robots-txt-sitemaps-and-llms-txt.md", "https://lookvalid.com/docs/api/site-files.md", "https://lookvalid.com/products/site-files.md"] --- # Find hidden pages from robots.txt > **For AI agents:** the complete index of LookValid pages is [llms.txt](https://lookvalid.com/llms.txt). Every page is also served as markdown at its URL plus .md. Every Disallow line in a robots.txt names a path. Open each one once, see which still answer any visitor, and learn what to fix on your own site. A robots.txt file asks crawlers to stay out of certain paths, and in doing so it prints a list of those paths for anyone to read [1]. The check below opens each plain Disallow path once, the way any visitor would, and shows which ones answer with a page. Use it on your own site to find pages you assumed were private, or on a site you audit to see what its owner chose to point at. Run it here: [Site files checker](https://lookvalid.com/tools/site-files-checker.md), a free tool. ## Why a Disallow line is a lead RFC 9309 says plainly that robots.txt rules are not a form of access authorization, and that listing paths in the file exposes them publicly and makes them discoverable [1]. Google says the same from the search side: robots.txt is not a mechanism for keeping a web page out of Google, and the two ways to do that are a noindex rule or password protection [2]. So each Disallow line is a request to well-behaved crawlers and a pointer for everyone else. A staging folder, an old admin panel or a forgotten export directory shows up in the file because someone wanted crawlers to skip it, not because the server hides it. Whether the path really answers a visitor is a separate question, and it takes one request to find out. ## What the check does with each path Enter a site in the tool above. It reads robots.txt, collects every distinct Disallow path in file order, and opens each plain one once with a GET request. The request carries the LookValid crawler user agent, which names the product and a contact address, and it sends no login, no form data and no parameters. Redirects are followed up to 5 times, and at most 20 paths are opened per site, so one check never turns into a crawl. Rules that are not a single plain address are listed and left unopened: wildcards such as `/*.pdf$`, rules with a query string, and a bare `/` that covers the whole site. The [site files checker](https://lookvalid.com/tools/site-files-checker.md) shows the same list, and the result of each opened path has four parts. What the result shows for each opened path | Part of the result | What it tells you | | --- | --- | | Status | The HTTP status of the final answer after redirects, such as 200, 403 or 404 | | Final address | Where the answer came from, which differs from the path when the server redirects | | Page title | The text of the page title when the answer is a web page, which often names the tool behind it | | Public or not | Public when any visitor gets a page. Not public when the path is missing, protected, or sends visitors to a sign-in page, the home page or another site | ## Reading what comes back - A 200 answer with a page title means the path is open to every visitor. If you wanted it private, a Disallow line did nothing for you. - A 401 or 403 answer means the server refuses visitors without access. That is the protection working, and the path is only listed. - A 404 or 410 answer means the path is gone. The robots.txt line is left over and can be removed. - A redirect to a sign-in page means the content sits behind a login. The sign-in page is public, the page itself is not. A public path is not automatically a problem. Many sites disallow search result pages, tag archives or print views on purpose, because they want them kept out of search results and are happy for visitors to open them. The finding that matters is a public path you meant to keep private. ## Fix what the check finds 1. Run the check on your own domain and read the table of paths robots.txt asks crawlers to skip. 2. For each public path that should be private, put it behind a login or remove it from the server. A Disallow line does not protect it [1]. 3. For each public path that should simply stay out of search results, serve a noindex rule and let crawlers read it, because a crawler that is blocked by robots.txt never sees the rule [2]. 4. Delete the Disallow lines for paths that return 404, so the file stops advertising pages that no longer exist. You can run the same check from code. The request below returns the full list in the `hidden_paths` block of the answer, next to the robots.txt, sitemap and llms.txt reports. Check a domain and read the hidden paths block ```bash curl -X POST https://YOUR_API_HOST/v1/site-files \ -H "Authorization: Bearer $LOOKVALID_API_KEY" \ -H "Content-Type: application/json" \ -d '{"domain":"example.com"}' ``` For what the other files do, read [what robots.txt, sitemaps and llms.txt do](https://lookvalid.com/blog/technical/reading-robots-txt-sitemaps-and-llms-txt.md), and see the [site files API reference](https://lookvalid.com/docs/api/site-files.md) for every field. ## Sources 1. [RFC 9309: Robots Exclusion Protocol](https://www.rfc-editor.org/rfc/rfc9309), read 2026-10-23 2. [Google Search Central: Introduction to robots.txt](https://developers.google.com/search/docs/crawling-indexing/robots/intro), read 2026-10-23 By LookValid engineering. Published 2026-10-23, updated 2026-10-23. ## Related - [Free robots.txt, sitemap and llms.txt checker](https://lookvalid.com/tools/site-files-checker.md) - [What robots.txt, sitemaps and llms.txt do](https://lookvalid.com/blog/technical/reading-robots-txt-sitemaps-and-llms-txt.md) - [Site Files Checker reference](https://lookvalid.com/docs/api/site-files.md) - [Site Files Checker API](https://lookvalid.com/products/site-files.md)