Robots.txt Security Checker

How it works

robots.txt is meant for crawlers, but attackers read it too. Disallow lines that mention /admin, /api, /backup, .env, staging, or debug paths effectively advertise where sensitive surfaces might live. We fetch /robots.txt for your public origin using SSRF-safe networking, then flag Disallow entries that match common sensitive patterns. A clean robots file does not mean the paths are gone - and a noisy one does not prove they are exposed. This is recon hygiene: reduce accidental disclosure, and treat flagged paths as candidates to lock down, auth-wall, or remove from public routing. We also note when robots.txt is missing or has no Disallow rules. We do not crawl the Disallow targets, fuzz for backups, or authenticate. Different environments (preview vs production) may publish different robots files. Use findings as a checklist before launch marketing drives bots and curious humans to your domain. Follow up with security.txt (so researchers know how to reach you) and a full AtRisk URL scan for live findings beyond robots.txt. This free tool is a teaser, not a penetration test.

This is a free teaser check - not a penetration test, vulnerability assessment, or compliance audit. AtRisk full scans go deeper across more detectors and unlock fix prompts on Starter.