robots.txt Generator & Tester

A robots.txt file is a plain-text file placed at the root of a website that tells search engine crawlers and other bots which parts of the site they may or may not request. It uses simple directives like User-agent, Disallow, and Allow, and it's the first thing well-behaved crawlers check before crawling a site.

It must be placed at the root of your domain — https://example.com/robots.txt — not inside a subfolder. A robots.txt at /blog/robots.txt or /pages/robots.txt will be ignored.

No. A website with no robots.txt is treated as fully crawlable by default. You only need one if you want to restrict crawling of specific paths, add a sitemap reference, or manage crawler-specific behavior.

Add "User-agent: Googlebot" followed by "Disallow: /" in its own group. Note this only stops crawling — pages that are already indexed or linked from elsewhere may still appear in search results without a snippet.

Indirectly. robots.txt doesn't boost rankings by itself, but it helps SEO by directing crawl budget away from low-value pages toward the pages that matter, and by preventing crawlers from wasting resources on duplicate or thin content.

Not reliably. robots.txt only blocks crawling, not indexing. If a blocked URL is linked from elsewhere, Google can still list it in search results (usually without a description), because it never crawled the page to see any noindex instruction.

Nothing breaks — crawlers assume everything is allowed and crawl normally. Missing robots.txt is not an error and doesn't hurt SEO by itself; it's simply the default "open" state.

Add a line like "Sitemap: https://example.com/sitemap.xml" anywhere in the file — it's global and applies regardless of which User-agent group it's placed under, and you can list multiple sitemap lines.

Use a dedicated tester like the one on this page — paste your file, pick a crawler, and test individual URLs or a batch of paths to see exactly which rule matches and why. Google Search Console also has a robots.txt report for your verified property.

Yes — Disallow rules match any path, including image files and folders, e.g. "Disallow: /images/private/" or "Disallow: /*.jpg$" to block a file type.

"*" matches any sequence of characters (including none), and "$" anchors the match to the end of the URL. For example, "Disallow: /*.pdf$" blocks any URL ending in ".pdf", while "Disallow: /private*" blocks anything starting with "/private".

Crawl-delay asks a crawler to wait a given number of seconds between requests. Bing and Yandex support it; Googlebot ignores it completely — use Google Search Console's crawl rate settings instead.

Yes, Google is one of the crawlers that most strictly follows the published robots.txt specification (it even helped standardize it as RFC 9309), matching rules using longest-match-wins logic.

Bing generally follows the same core directives as Google but, unlike Google, it does honor Crawl-delay. It's worth testing rules specifically for Bingbot if precise control matters.

Some do, some don't. Reputable AI companies like OpenAI (GPTBot) and Anthropic (ClaudeBot) publish crawler names and generally respect robots.txt, but compliance is entirely voluntary.

Accidentally disallowing the entire site with "Disallow: /" under "User-agent: *", blocking CSS/JS files search engines need to render the page, forgetting the sitemap URL, trying to hide sensitive pages (it reveals the path publicly instead), and putting rules before any User-agent line so they're ignored.

Yes. Paths in robots.txt are case-sensitive, so "/Folder/" and "/folder/" are treated as two different paths. The directive names themselves are not case-sensitive.

No — a site should have exactly one robots.txt file at its root. Each subdomain, though, is treated as a completely separate host and needs its own robots.txt.

Disallow (in robots.txt) tells crawlers not to request a URL at all. Noindex (a meta tag or HTTP header) tells crawlers they may request the page but should not include it in search results.

Not directly — robots.txt controls crawling, not indexing, which is a common and important distinction in technical SEO.

Yes — create a separate User-agent group naming just that bot (e.g. "User-agent: Bingbot") with its own rules. Bots look for a group naming them specifically before falling back to the "*" (all-crawlers) group.

It carves out an exception. For example, "Disallow: /wp-admin/" followed by "Allow: /wp-admin/admin-ajax.php" blocks the admin area but still lets that one specific file be crawled.

Yes — Google enforces a 500 KiB limit and ignores content beyond that. Well-organized files are almost always well under this limit.

Yes — any line starting with "#" is a comment and ignored by crawlers, useful for documenting why a rule exists.

If it's already live, remove or fix the rule and resubmit the file — Google typically re-fetches robots.txt within about a day, though you can request a recrawl via Search Console.

Yes. Each subdomain is a distinct host as far as crawlers are concerned, so blog.example.com needs its own robots.txt separate from www.example.com's.

Yes, using wildcards — e.g. "Disallow: /*?sort=" blocks any URL containing a "sort=" query parameter, a common way to stop crawlers wasting budget on faceted navigation.

A canonical tag tells search engines which URL is the "master" version among duplicates, while robots.txt controls whether a URL gets crawled at all. A page can be canonicalized and still fully crawlable.

Google typically re-checks robots.txt roughly once a day, and caches the last successfully fetched version for up to 24 hours if a later fetch fails.

Yes, but robots.txt alone won't reliably keep a staging site out of search results — combine "Disallow: /" with password protection or a noindex header for reliable exclusion.

Yes, meaningfully on large sites. Disallowing low-value URL patterns frees up crawl budget for pages that actually matter.

It's the default group that applies to any crawler not specifically named elsewhere in the file — the fallback most sites rely on for their general rules.

You technically can, but it's strongly discouraged — blocking CSS or JS can prevent Google from rendering your page the way users see it, which can hurt ranking. Google explicitly recommends against this.

Add a User-agent group naming the crawler (e.g. "User-agent: GPTBot") with "Disallow: /" underneath. This tool has a one-click "Block AI crawlers only" preset covering the major named AI bots.

robots.txt should be served as plain text with the MIME type "text/plain" — serving it as HTML or with an incorrect content type can cause some crawlers to misinterpret or ignore it.

No — robots.txt can only allow or disallow crawling of specific paths. Redirection is a separate, server-level concern.

Yes, this generator and tester is completely free, with no signup, no usage limits, and no watermark on the output.

No — everything runs locally in your browser. Your robots.txt content, rules, and any URLs you test are never sent to a server.

Yes — paste or type a robots.txt into the Tester tab and click "Import this into visual builder" to load it into the Generator tab for editing.

Yes — use "Export JSON" to save your configuration as a file you can re-import later, "Save to browser" to store it locally, or "Copy shareable link" to generate a URL that reconstructs your configuration when opened.