~/robots ☀ LIGHT apps ← about me

🌐 Language

// seo · crawlers · search engines · web scraping
██████╗ ██████╗ ██████╗ ██████╗ ████████╗ ███████╗ ██╔══██╗ ██╔═══██╗ ██╔══██╗ ██╔═══██╗ ╚══██╔══╝ ██╔════╝ ██████╔╝ ██║ ██║ ██████╔╝ ██║ ██║ ██║ ███████╗ ██╔══██╗ ██║ ██║ ██╔══██╗ ██║ ██║ ██║ ╚════██║ ██║ ██║ ╚██████╔╝ ██║ ██║ ╚██████╔╝ ██║ ███████║ ╚═╝ ╚═╝ ╚═════╝ ╚═════╝ ╚═════╝ ╚═╝ ╚══════╝

robots.txt generator_

Build a robots.txt file visually — add rules per bot, set allow/disallow paths, sitemaps and crawl delays.
Live preview updates as you type. Copy and paste directly into your site root. 100% client-side.

// import existing robots.txt
// quick presets
// global settings
// live preview robots.txt 

      

      

        
        
        
      

      

        
// checks

        

      

      

        
// test a url

        

          
          
          
          
        

        

      

    


  


  
  

    
// known crawlers — click to add as rule

    

  












Frequently Asked Questions — robots.txt Generator



grep






Robots.txt Basics


What is a robots.txt file?
A plain-text file placed at the root of your domain that tells web crawlers which parts of your site they're allowed to request. It's the first thing most well-behaved crawlers check before fetching any other page.


Where does robots.txt need to be placed?
At the root of your domain, exactly at yoursite.com/robots.txt — not in a subfolder. A file at yoursite.com/blog/robots.txt is ignored, and each subdomain needs its own copy at its own root.


Does robots.txt prevent a page from being indexed?
No. robots.txt only asks crawlers not to fetch certain paths — it doesn't stop a page from appearing in search results if it was already discovered some other way. To actually keep a page out of the index, use a <meta name="robots" content="noindex"> tag or the X-Robots-Tag HTTP header, and don't Disallow that page — or the crawler can't fetch it to see the noindex instruction in the first place.


What does User-agent: * mean?
It targets every crawler that doesn't have its own more specific User-agent block elsewhere in the file. Use a specific name, like User-agent: Googlebot, to write rules for one crawler only — major search engines use the most specific matching block instead of the wildcard one.


Is this tool free, and is my data uploaded anywhere?
Yes, completely free — and everything runs client-side in your browser. The rules you build, and anything you paste into the import box, never leave your machine; there's no server involved in generating your robots.txt.



Directives & Syntax


What does Disallow do?
Disallow tells the matched crawler not to request the given path. Disallow: /admin/ blocks that one folder; Disallow: / blocks the entire site for that crawler.


What does Allow do, and when do I need it?
Allow explicitly permits a path, which matters when it sits inside a broader Disallow. For example, Disallow: /private/ followed by Allow: /private/public-file.pdf lets that one file through even though its parent folder is blocked.


When Allow and Disallow overlap for the same path, which one wins?
The longer, more specific path wins, regardless of which directive appears first in the file. If an Allow and a Disallow match with exactly the same length, Allow wins — this is documented behavior for Google and most major crawlers, and it's exactly how the URL tester on this page evaluates a match.


What do * and $ mean inside a path?
* matches any sequence of characters, so Disallow: /*.pdf$ blocks any path ending in .pdf. $ anchors the match to the end of the URL — without it, Disallow: /page also matches /page2 and /page/anything. Both are a widely supported extension beyond the original robots.txt spec.


What does an empty Disallow: value mean?
It means no restriction at all for that block. Disallow: with nothing after the colon is equivalent to not having a Disallow rule for that crawler.


What is Crawl-delay, and does Google respect it?
Crawl-delay asks a crawler to wait a number of seconds between requests. Bing, Yandex and several others honor it — Google does not and ignores Crawl-delay entirely; use the crawl rate settings in Google Search Console instead.


What is the Sitemap directive for?
Sitemap points crawlers to your sitemap.xml so they can discover pages more efficiently. You can list as many as you need, one per line, and unlike other directives a Sitemap line applies to every crawler regardless of which User-agent block it appears near.


What is the Host directive — do I need it?
Host was a Yandex-specific directive for stating a preferred domain when a site was mirrored under multiple hostnames. Yandex deprecated it in 2018 and no major crawler relies on it today — leave it blank unless you have a specific legacy reason to set it.


Does the order of rules in the file matter?
Not within a single block — Google and most major crawlers always compare the longest matching Allow vs Disallow path, not which line came first. Order does matter between blocks though: a specific User-agent block is used instead of the wildcard block entirely for that crawler, not merged with it.



Bots, Presets & AI Crawlers


What do the quick presets do?
Each preset replaces your current rules with a ready-made set for a common situation — allowing or blocking every bot, a standard SEO-friendly setup, blocking known AI-training crawlers, blocking known low-value SEO/scraper bots, or starter rules for e-commerce and blog sites. Applying one overwrites what's currently in the builder, so copy or export your existing rules first if you want to keep them.


What happens when I click a bot in the "known crawlers" grid?
It adds a new rule block pre-filled with that bot's exact user-agent string and a Disallow: / directive, ready for you to edit or delete. If a rule for that bot already exists, it won't add a duplicate — you'll get a notice instead.


What's the difference between "block AI crawlers" and "block bad bots"?
Both produce Disallow: / blocks for different bot lists. "Block AI crawlers" targets bots used to gather AI training data — GPTBot, CCBot, Google-Extended and similar — while still allowing normal search crawlers. "Block bad bots" instead targets third-party SEO/scraper tools like SemrushBot, AhrefsBot, MJ12bot and DotBot, which many sites block to save crawl bandwidth rather than because they're malicious.


Does blocking GPTBot or CCBot stop my content from appearing in that AI's answers?
Not necessarily. GPTBot and CCBot are generally used for training-data collection, not for the separate crawler some AI products use to fetch a page while answering a live user question — those often use a different user-agent string. Check each provider's current documentation for their live-browsing crawler's name if you want to block that separately.


Are there crawlers that ignore robots.txt entirely?
Yes. robots.txt relies on voluntary compliance — reputable crawlers honor it, but nothing technically stops a scraper from ignoring it. Treat robots.txt as guidance for well-behaved bots, not a security or access-control mechanism; use authentication or server-side blocking for anything that actually needs to be kept out.



Testing, Validation & Import


What does the "test a url" panel do?
You give it a path and a user-agent (default *), and it tells you whether your current rules would allow or block that exact combination — including which specific Allow or Disallow line decided it. It re-evaluates live as you edit your rules.


What do the ✓ / ⚠ / ✗ checks mean?
They flag common mistakes as you build: a red ✗ is a rule that fully blocks a major search engine (or all crawlers) from your whole site; a yellow ⚠ is something worth double-checking, like a Crawl-delay Google ignores, a path missing its leading /, or no sitemap declared; a green ✓ means no issues were found.


Can I import my site's existing robots.txt?
Yes — paste it into the import box (or upload the .txt file) and click "import parsed rules." It reads User-agent, Allow, Disallow, Crawl-delay, Sitemap and Host lines and rebuilds them as editable rule blocks, replacing whatever is currently in the builder.


What happens if I paste a robots.txt with several User-agent lines in a row?
Consecutive User-agent lines are treated as sharing the same block, per the robots.txt spec — so User-agent: A / User-agent: B / Disallow: /x is imported as two separate rule blocks, one for A and one for B, each starting with an identical copy of that Disallow rule, which you can then edit independently.


Where can I double-check against how Google actually interprets my file?
This tool checks your logic client-side and doesn't fetch your live site. For a definitive read on how Google specifically sees your actual deployed file, use the robots.txt report in Google Search Console, which tests against Google's real parser.



Using This Tool


Does this tool save my work if I close the tab?
Yes. Your rules, sitemaps and host are saved to your browser's local storage as you edit, and a resume banner offers to restore them the next time you open this page on the same browser. Nothing is sent anywhere — it never leaves your device.


Does the download button produce a real, ready-to-upload file?
Yes — clicking download saves an actual robots.txt file, not a renamed .txt, ready to upload to your site's root exactly as generated.


What languages does this tool support?
The interface is available in English, Dutch, Spanish, Arabic, Indonesian, French, Portuguese, Russian, Japanese and German — switch via the 🌐 globe icon in the top bar.