Robots.txt Generator
Build a robots.txt file for common crawler directives and sitemap discovery.
Rules
robots.txt
Create a robots.txt file
Robots.txt provides crawl directives to compliant bots. It is not an access-control mechanism and should not be used to protect sensitive content.