How does Google decide which robots.txt rule applies?

Google uses the group that names its crawler (for example Googlebot), or the * group if none does. Within that group, the most specific (longest) matching rule wins. If an Allow and a Disallow rule are equally long, Allow wins.

Does Disallow remove a page from Google?

No. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.

Can I block AI crawlers with robots.txt?

Yes. Add a group for the crawler's user agent, such as GPTBot, ClaudeBot or PerplexityBot, with Disallow: /. Google-Extended controls whether Google uses your content for Gemini training, separately from Googlebot.

Free SEO tool

Robots.txt Tester

Fetch or paste a robots.txt file and test whether Googlebot or any crawler can access a URL, using Google's matching rules.

robots.txtor paste your own

URL or path to test

Crawler

Blocked

/admin/settings for Googlebot

Matched line 2: Disallow: /admin/

Using group(s) for: *

Sitemaps

https://example.com/sitemap.xml

Show matched line in context

1User-agent: *
2Disallow: /admin/
3Disallow: /*?sort=
4Allow: /admin/help/
5
6User-agent: GPTBot
7Disallow: /
8
9Sitemap: https://example.com/sitemap.xml

How to use the robots.txt tester

Fetch the live file from any domain, or paste a draft to test changes before you deploy them. Then enter a URL or path and pick a crawler. The tester shows whether the URL is allowed and highlights the exact line that decided it.

Matching rules this tester follows

Groups for the same user agent are combined.
* matches any sequence of characters, and $ marks the end of the URL.
Rules match against the path and query string, case-sensitively.
An empty Disallow: allows everything, and /robots.txt itself is always allowed.
A missing robots.txt (HTTP 4xx) means everything is allowed. A server error (5xx) makes Google pause crawling.

Frequently asked questions

How does Google decide which robots.txt rule applies?: Google uses the group that names its crawler (for example Googlebot), or the * group if none does. Within that group, the most specific (longest) matching rule wins. If an Allow and a Disallow rule are equally long, Allow wins.
Does Disallow remove a page from Google?: No. Disallow stops crawling, not indexing. A blocked URL can still appear in results if other pages link to it. To keep a page out of the index, allow crawling and use a noindex meta tag instead.
Can I block AI crawlers with robots.txt?: Yes. Add a group for the crawler's user agent, such as GPTBot, ClaudeBot or PerplexityBot, with Disallow: /. Google-Extended controls whether Google uses your content for Gemini training, separately from Googlebot.