Robots.txt Tester

Fetch or paste a robots.txt file and test whether Googlebot or any crawler can access a URL, using Google's matching rules.

Blocked

/admin/settings for Googlebot

Matched line 2: Disallow: /admin/

Using group(s) for: *

Sitemaps

  • https://example.com/sitemap.xml
Show matched line in context
1User-agent: *
2Disallow: /admin/
3Disallow: /*?sort=
4Allow: /admin/help/
5
6User-agent: GPTBot
7Disallow: /
8
9Sitemap: https://example.com/sitemap.xml

How to use the robots.txt tester

Fetch the live file from any domain, or paste a draft to test changes before you deploy them. Then enter a URL or path and pick a crawler. The tester shows whether the URL is allowed and highlights the exact line that decided it.

Matching rules this tester follows

  • Groups for the same user agent are combined.
  • * matches any sequence of characters, and $ marks the end of the URL.
  • Rules match against the path and query string, case-sensitively.
  • An empty Disallow: allows everything, and /robots.txt itself is always allowed.
  • A missing robots.txt (HTTP 4xx) means everything is allowed. A server error (5xx) makes Google pause crawling.

Frequently asked questions

How does Google decide which robots.txt rule applies?

Google uses the group that names its crawler (for example Googlebot), or the * group if none does. Within that group, the most specific (longest) matching rule wins. If an Allow and a Disallow rule are equally long, Allow wins.

Does Disallow remove a page from Google?

No. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.

Can I block AI crawlers with robots.txt?

Yes. Add a group for the crawler's user agent, such as GPTBot, ClaudeBot or PerplexityBot, with Disallow: /. Google-Extended controls whether Google uses your content for Gemini training, separately from Googlebot.