SEO check
robots.txt
Fetches /robots.txt and tests the audited URL against its rules for Googlebot.
- Category
- SEO
- Importance
- High importance
- Check id
- robots-txt
How the result is decided
- Passrobots.txt exists and allows Googlebot to crawl this page.
- WarningThere is no robots.txt (a 4xx status), or it returned an unexpected status.
- Failrobots.txt blocks Googlebot from this page, or it returns a server error (5xx), which Google treats as “don’t crawl”.
- Inforobots.txt couldn’t be loaded.
“Info” results don’t count toward the score.
Why it matters
robots.txt controls what crawlers may visit. A rule that blocks an important page keeps it from ranking properly, and a robots.txt that errors can make Google pause crawling the whole site.
How to fix it
- If the report names a blocking rule, remove or narrow it.
- If there is no robots.txt, add a small one that lists your sitemap.
- If it returns a 5xx error, fix the server so it returns the file (or a 404).
User-agent: *
Disallow:
Sitemap: https://www.example.com/sitemap.xmlAfter the fix, open the page’s report and choose Run again to confirm it passes.