Home / Tools / robots.txt generator
robots.txt generator
Generate a downloadable robots.txt with crawler, path and sitemap rules.
Content is processed in this browser, without upload or storage.
How it works & common questions
Does robots.txt protect private content?
No. It is a crawling directive for compliant crawlers, not authentication or access control, and it does not guarantee removal from search results.
How do I deploy it?
Download and publish at /robots.txt on your domain. Review paths before deployment. Public pages are usually allowed; private consoles and APIs may be excluded. Use an absolute HTTP or HTTPS sitemap URL.