How to test robots.txt
- Paste the contents of your robots.txt file. You can open it at
https://yoursite.com/robots.txtand copy everything. - Choose the crawler to test: Googlebot, Bingbot, GPTBot, ClaudeBot, Google-Extended or the wildcard
*. - Enter the URLs or paths you care about, one per line. Each one is marked Allowed or Blocked, with the exact rule that decided it.
- Edit the robots.txt text and watch the results update, then publish the fixed version.
How the matching works
The tester follows the rules Google documents and that RFC 9309 standardises. A crawler uses the group whose User-agent line matches its name most specifically, falling back to the * group, and ignores all other groups. Within that group, the longest matching path rule wins; if an Allow and a Disallow rule match with equal length, Allow wins. * matches any sequence of characters and $ anchors the end of the URL.
That is why a common mistake slips through: adding a User-agent: Googlebot group for one rule means Googlebot no longer reads any of the rules in the * group. Testing several important URLs, such as your home page, key product pages, CSS and JavaScript files, and pages you meant to block, catches these problems before they cost traffic.
The test runs entirely in your browser. To create a new file, switch to the Generate tab.
FAQ
Does a blocked URL disappear from Google?
Not necessarily. Blocking stops crawling, but a URL can still be indexed without content if other pages link to it. Use noindex to remove pages from results.
Is the test the same as Google Search Console?
It uses the same matching rules. Search Console’s robots.txt report also shows the version Google last fetched and any fetch errors.
Are paths case-sensitive?
Yes. /Admin and /admin are different paths in robots.txt rules.