Robots.txt Generator
Build a robots.txt file with rules for all crawlers or specific bots, plus your sitemap location.
Runs in your browser — nothing is uploadedStart from a preset
Upload this file to the root of your site, e.g. https://example.com/robots.txt. robots.txt controls crawling, not indexing — use a noindex meta tag to keep a page out of search results. Google ignores Crawl-delay; Bing and Yandex respect it.
About Robots.txt Generator
A robots.txt file sits at the root of your website and tells search engine crawlers which parts of the site they may crawl. It can block admin areas, internal search results and duplicate pages, point crawlers to your sitemap and set rules for specific bots — including AI crawlers.
Start from a preset, add Allow and Disallow rules for all crawlers or specific bots, add your sitemap URL and download a ready-to-upload file.
When to use it
- Setting up a new website for search engines.
- Stopping crawlers from wasting time on admin, cart or search pages.
- Blocking or allowing AI training crawlers.
- Adding your sitemap location for search engines.
Tips for better results
- Never block CSS and JavaScript files that pages need to render.
- Test changes in Google Search Console before relying on them.
- Remember that robots.txt is public — do not list secret URLs in it.
How to use the Robots.txt Generator
- Start from a preset or the default.
- Add Disallow and Allow paths for each crawler.
- Add your sitemap URL.
- Download robots.txt and upload it to your site root.
Frequently asked questions
Does Disallow remove a page from Google?
No. robots.txt controls crawling, not indexing. A blocked page can still appear in results if other sites link to it. Use a noindex meta tag to keep a page out of search results.
Will blocking AI crawlers stop all AI tools?
It asks the listed crawlers not to crawl your site. Reputable crawlers respect robots.txt, but it is a request, not an enforcement mechanism.
Where do I upload robots.txt?
In the root of your domain, so it is available at https://yourdomain.com/robots.txt. Each subdomain needs its own file.
What happens if I have no robots.txt?
Crawlers assume they can crawl everything. That is fine for many small sites, but adding a file with your sitemap location is still good practice.