robots.txt is a file that tells search engine crawlers which pages to crawl and which to skip.

Where should I place robots.txt?

Place robots.txt in the root directory of your website (e.g., example.com/robots.txt).

Are my data uploaded to a server?

No. All processing happens directly in your browser.

Last updated: July 2026 • 5 min read

robots.txt guides search engine crawlers through your site. Generate a properly formatted robots.txt to control indexing and crawling.

Key Features: Custom rules • Sitemap reference • Crawl delays • Download file • 100% private • No registration required

Go to the SEO Tools page and click on the "Robots.txt" tab.

Set which directories to allow or disallow for crawlers.

Download the generated robots.txt file.

robots.txt tells search engine crawlers which pages to crawl and which to skip.