Build a robots.txt file without learning the syntax. Choose your site type, toggle the sections you want hidden from crawlers, and export a clean file in seconds. No command line, no guesswork.
Toggle sections you want hidden from search engines. Active = blocked from crawlers.
One path per line
One path per line
# Toggle the sections you want to block on the left # Your robots.txt will appear here once you click Generate
Every search engine bot that visits your site spends a slice of your crawl budget, and that budget is finite. Robots.txt is the file that decides how it gets spent. A bot that wastes requests on login pages, admin panels, and internal search results has fewer requests left for the content you actually want ranked. A well built file points crawlers at your best pages and keeps thin or private areas out of the way.
The opposite mistake is just as common. One bad rule can hide your entire site from Google and Bing, and the traffic disappears within days. That risk keeps many site owners from touching the file at all, which is understandable but costly. The syntax is simple once you see it, yet most people never get the chance because nobody shows them how. This generator removes the barrier by translating plain language choices into correct directives.
There is a second reason this file matters more than it used to. AI crawlers now roam the web at scale, collecting pages for model training and answer engines. GPTBot, CCBot, ClaudeWeb, PerplexityBot, and Bytespider all read robots.txt, so the file is the standard place to state your preference about AI access. A modern robots.txt is a traffic policy for every bot, search engines included.
Start with your site type. The presets handle the common cases, so a blog blocks admin, login, search, tags, and API endpoints by default, while an online store adds cart, checkout, and account pages. You can switch any section on or off with one click, and the preview updates as you go.
Use the crawl settings to control behaviour for the main bots. The crawl delay slider tells crawlers how long to wait between requests, which helps when server load is a concern. The option to remove the delay keeps the file clean when you do not need it. Toggle the AI crawler block to restrict known AI bots with a single switch, or add custom paths for anything the presets miss.
When the file looks right, generate it, then upload it to the root of your domain so it sits at example.com/robots.txt. Confirm in Google Search Console that crawlers can reach your sitemap and that nothing important is blocked. Small mistakes are easy to catch when you can see the final file before it goes live.