How to create a robots.txt file
- Choose whether crawlers may access everything (normal for live sites) or nothing (for staging and development sites).
- List paths to keep crawlers out of, such as admin areas, carts, internal search results and filter or sort URLs that create endless duplicates. Use
*as a wildcard and$to match the end of a URL. - Add your sitemap URL so search engines find all your pages, and optionally block AI training crawlers.
- Copy or download the file and upload it to the root of your domain. Then use the Test URLs tab to check that important pages are allowed and private ones are blocked.
What robots.txt can and cannot do
robots.txt tells well-behaved crawlers which URLs they may fetch. It saves crawl budget on large sites and keeps bots out of areas with no search value. It is not a security measure: the file is public, and bad bots ignore it. It also does not reliably keep pages out of Google's index: a blocked URL can still be indexed from links, without its content. To remove a page from search results, allow crawling and add a noindex meta tag instead.
Search engines pick the group that most specifically matches their user agent, then apply the longest matching rule; when an Allow and a Disallow rule are equally long, Allow wins. The tester follows these rules, as documented by Google and RFC 9309. Never block your CSS and JavaScript files, because Google needs them to render pages.
The AI crawler options add groups for bots used to train or power AI models, including GPTBot, ClaudeBot, Google-Extended and CCBot. Blocking them does not affect normal search rankings.
Everything runs in your browser.
Where do I put robots.txt?
At the root of each host: https://example.com/robots.txt. Subdomains such as shop.example.com need their own file.
Does Google support Crawl-delay?
No. Googlebot ignores Crawl-delay; Bing and Yandex respect it. Google adjusts its crawl rate automatically based on your server's response.
How do I block all bots from a staging site?
Use "User-agent: *" followed by "Disallow: /". Better still, protect staging with a password, since robots.txt is only a request.