Robots.txt Generator
Create robots.txt files with user-agent rules, sitemaps, and crawl directives
Template
Rules
Directive
Ctrl+Enter to runCtrl+Shift+C to copy
Learn More
robots.txt Generator: Control Search Engine Crawling for Your Site
Learn robots.txt syntax, create rules for search engines and AI crawlers, and avoid common SEO mistakes.
What Is robots.txt?
robots.txt is a plain text file placed at the root of a website that tells web crawlers which pages or sections they should or should not access. It follows the Robots Exclusion Protocol, a standard that has been used since 1994 to communicate crawling preferences to search engine bots, AI crawlers, and other automated agents.
6 min readRead full guide