// seo · crawlers · search engines · web scraping
██████╗ ██████╗ ██████╗ ██████╗ ████████╗ ███████╗
██╔══██╗ ██╔═══██╗ ██╔══██╗ ██╔═══██╗ ╚══██╔══╝ ██╔════╝
██████╔╝ ██║ ██║ ██████╔╝ ██║ ██║ ██║ ███████╗
██╔══██╗ ██║ ██║ ██╔══██╗ ██║ ██║ ██║ ╚════██║
██║ ██║ ╚██████╔╝ ██║ ██║ ╚██████╔╝ ██║ ███████║
╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═════╝ ╚═╝ ╚══════╝
robots.txt generator_
Build a robots.txt file visually — add rules per bot, set allow/disallow paths, sitemaps and crawl delays.
Live preview updates as you type. Copy and paste directly into your site root. 100% client-side.
// quick presets
// global settings
// live preview
robots.txt
// known crawlers — click to add as rule