Free Tool
Robots.txt Analyzer
Enter a URL to fetch and analyze any site's robots.txt file.
What is robots.txt?
- Purpose: Tells search engine crawlers which pages they can and can't access on your site.
- Location: Always at
yoursite.com/robots.txt— it must be in the root directory. - Not security: robots.txt is a suggestion, not a wall. Don't use it to hide sensitive pages.
- AI crawlers: Bots like GPTBot, CCBot, and Google-Extended can be blocked to prevent AI training on your content.
- Sitemap: Include a
Sitemap:directive pointing to your XML sitemap.
This is just one of 90+ checks. Get your full Growth Score.
Run Free Audit →