Generate robots.txt crawler directives with allow/disallow rules and sitemap entries.
Updated
Quick start
Configure inputs and generate output.
Enter required fields.
Create output instantly.
Copy and apply in your workflow.
In-depth guide
A robots.txt file sits at the root of your domain and tells well-behaved crawlers which paths they may or may not request. It is the first file Googlebot fetches on a site. This tool assembles valid User-agent, Allow, Disallow and Sitemap directives in your browser, ready to paste.
User-agent names the crawler a block applies to (* means all). Disallow blocks a path prefix; Allow carves out an exception inside a blocked path. Sitemap points crawlers at your XML sitemap. Rules are matched by longest path, so a specific Allow can override a broad Disallow.
robots.txt at your domain root, e.g. https://example.com/robots.txt.Use it to manage crawl budget and keep low-value paths out of the queue. Do not use it to hide a page from search results — a disallowed URL can still be indexed if linked elsewhere. To keep a page out of the index, allow crawling and use a noindex meta tag or header instead.
Use this tool for quick browser-based work when you need an answer or output immediately. Use a dedicated application or automated workflow when you need bulk processing, approvals, or repeatable production rules.
Yes. Provide your sitemap URL and the tool appends a Sitemap line, which helps crawlers discover all your pages.
Major crawlers support * as a wildcard and $ to anchor the end of a path. You can enter these patterns as plain lines.
Yes. Copy it into a file named robots.txt placed at your domain root, for example https://example.com/robots.txt.
No. A disallowed URL can still be indexed if other sites link to it. To keep a page out of the index, allow crawling and add a noindex meta tag or header instead.
No. Google needs to fetch CSS and JS to render your page the way users see it. Blocking them can hurt how your pages are understood and ranked.
No, it is publicly readable by anyone. Do not list secret paths there — use authentication for anything that must stay hidden.
Keep exploring
Hand-picked utilities that pair well with the one you're on — all free, client-side, and zero-signup.
Generate a deduplicated .gitignore by combining language, framework, OS and editor templates.
Convert URL lists into valid sitemap.xml output with optional lastmod, changefreq and priority fields.
Generate SEO, Open Graph and Twitter tags from page metadata fields.
Shrink PDF file size without uploading to a server.
Body Mass Index with metric / imperial inputs and WHO category bands.
Combine multiple PDFs into one in your browser.
Split a PDF by pages or page ranges, download as zip.