Where should robots.txt live?

At the root of your domain: https://example.com/robots.txt. Crawlers look only at that exact URL.

Does Disallow block indexing?

No — Disallow blocks crawling, not indexing. A disallowed page can still appear in search results if linked from elsewhere. To fully block indexing, use the noindex meta tag or X-Robots-Tag header.

What does crawl-delay do?

It asks compliant crawlers to wait N seconds between requests. Google ignores it (use Search Console settings instead), but Bing and Yandex honor it. Useful for low-traffic servers.

Can I have multiple User-agent blocks?

Yes — add as many as you need. Each block applies only to its named bot. Use * as a catch-all at the end.

Should my sitemap be in robots.txt?

It's a good practice — adding `Sitemap: https://example.com/sitemap.xml` helps crawlers discover it. Major search engines also accept direct sitemap submission via their webmaster consoles.

Robots.txt Generator

Build a robots.txt with allow/disallow rules, sitemap & crawl-delay.

100% private — no uploads SEO ToolsFree & unlimited Read tutorial

See how it works

Watch a quick simulation of this tool in action — no file needed.

How to use Robots.txt Generator

Follow these steps — everything happens instantly in your browser.

1
Step 1. Add rule blocks — one per user-agent (use * for all bots).
2
Step 2. For each block, list Allow and Disallow paths (e.g. /, /admin/).
3
Step 3. Optionally set a crawl-delay and a Host directive for mirror sites.
4
Step 4. Enter your sitemap URL, then copy or download the generated robots.txt.