🤖 SEO & Developer Tools
robots.txt Generator & Tester
Build a robots.txt visually with real-world presets, then test any URL against it using the actual Google-spec matching algorithm — see exactly which rule wins and why.
Quick presets
Sitemaps (global, placed at the end)
Host directive (Yandex only — largely obsolete)
Paste an existing robots.txt below, or click "Use generated robots.txt" to test what you just built.
Single URL test
Batch test
Directive reference
| Directive | What it does | Support |
|---|---|---|
User-agent | Starts a group targeting one or more crawlers by name. | All major crawlers |
Disallow | Blocks crawling of a path (empty value = allow all). | All major crawlers |
Allow | Explicitly permits a path, usually to carve out an exception inside a disallowed folder. | Google, Bing, most modern crawlers |
Sitemap | Points crawlers to your XML sitemap. Global — applies regardless of which group it's under. | All major search engines |
Crawl-delay | Requests a minimum delay (seconds) between requests. | Bing, Yandex — ignored by Google |
Host | Historically told Yandex your preferred mirror/domain. | Yandex only, largely obsolete today |
Noindex | Was an unofficial attempt to keep pages out of the index via robots.txt. | Not supported by Google since Sept 2019 |
Where the file must live
robots.txt must be placed at the root of your site — https://example.com/robots.txt — never in a subfolder like /pages/robots.txt, which crawlers will not look for. Each subdomain needs its own file: shop.example.com and www.example.com are treated as separate hosts. Paths in robots.txt are case-sensitive (/Folder and /folder are different), and the file must be saved as plain UTF-8 text.
robots.txt (crawl) vs. meta robots noindex (index)
robots.txt tells crawlers which URLs they may fetch. It does not reliably keep a URL out of search results — if other pages link to a blocked URL, Google can still show that URL in results (typically without a snippet), because it never crawled the page to see any noindex instruction.
A meta robots noindex tag (or X-Robots-Tag: noindex header) tells crawlers not to index a page they're allowed to crawl. This is the correct tool when you want a page to disappear from search results entirely. Use robots.txt to save crawl budget on low-value URLs (like faceted search or admin areas); use noindex when the goal is specifically to keep a page out of search.
Frequently asked questions
Where do I put robots.txt?
It must live at the root of your domain, e.g. https://example.com/robots.txt — not in a subfolder. Each subdomain (like shop.example.com) needs its own separate robots.txt file at its own root.
Does robots.txt stop pages being indexed?
Not reliably. robots.txt only asks crawlers not to crawl a URL — if the page is linked from elsewhere, Google can still index the URL (usually without a description) since it never fetched the content to see a noindex tag. To truly keep a page out of search results, use a meta robots noindex tag or password protection instead.
How do I block AI bots?
Add a User-agent group listing the AI crawler names you want to block (like GPTBot, CCBot, Google-Extended) with Disallow: / underneath. Compliance is voluntary, though — only well-behaved crawlers respect robots.txt, so this won't stop bots that ignore it entirely.
Why is Google ignoring my crawl-delay?
Googlebot does not support the Crawl-delay directive at all — it's simply ignored. Bing and Yandex do honour it. To control Google's crawl rate, use the crawl rate settings in Google Search Console instead.