Select what search engines should be allowed to access — allow all, block all, or custom rules.
If using custom mode, enter the directory paths to block (one per line, e.g. /admin/).
Add your sitemap URL to help crawlers discover your pages efficiently.
Click Generate robots.txt and then Copy robots.txt to upload to your site root.
All processing happens locally. Your text/data never reaches any server.
Results appear instantly as you type or paste. No page reloads needed.
No sign-ups, no limits, no watermarks. Use it as much as you want.
Responsive design works on desktop, tablet, and mobile browsers.
Robots.txt tells search engine crawlers which parts of your site they are allowed or not allowed to access and index. It is a standard protocol for crawl control.
Only block everything if your site is in development or staging. For live sites, allow all crawlers and only block specific private or admin directories.
The robots.txt file must be placed in the root directory of your website (e.g., yourdomain.com/robots.txt). It cannot be placed in subdirectories.