SEO & MARKETING TOOLS / DEVTOOLS
Robots.txt Generator
Create a robots.txt file with crawl rules and a sitemap URL.
Values are generated locally in your browser; they are not uploaded.
ABOUT THIS TOOL
What Robots.txt Generator does
Draft robots.txt user-agent, allow, disallow, and sitemap directives as plain text.
Useful for drafting crawler directives. Robots rules are public and do not secure private pages.
How to use Robots.txt Generator
- Choose a user-agent or use the wildcard for all crawlers.
- Enter allow and disallow paths, one per line.
- Add the absolute sitemap URL and copy or download the text.
ANSWERS AND DETAILS
Frequently asked questions
What does this tool do?
Draft robots.txt user-agent, allow, disallow, and sitemap directives as plain text.
When is this tool useful?
Useful for drafting crawler directives. Robots rules are public and do not secure private pages.
Is my input uploaded?
No. This tool runs in your browser and does not send the value you enter to a processing server.