Skip to content
DevUtil

Robots.txt Generator - Build Robots.txt Online

Presets
Rules
Rule 1
Sitemaps
Output
User-agent: *
Allow: /
Disallow: /admin
Disallow: /api
Disallow: /private
Disallow: /tmp

About the Robots.txt Generator

Every website needs a robots.txt file to tell crawlers which paths are open and which are off-limits. Writing the syntax by hand is simple for one or two rules, but gets tedious when you need to manage multiple user-agents, block AI training bots, and include sitemaps.

This visual builder lets you create robots.txt files through a form interface. Start with a preset, then customise individual rules. The live preview updates as you type so you always see exactly what will be generated.

Presets

  • Allow All. Opens your entire site to all crawlers.
  • Block All. Disallows all crawlers from all paths. Useful for staging sites.
  • Block AI Crawlers. Allows regular search engines but blocks 10 known AI training crawlers including GPTBot, CCBot, and Claude-Web.
  • Standard SEO. Allows everything except common private directories like /admin, /api, /private, and /tmp.

How to Use

Click a preset to start, then add or edit rules as needed. Each rule block has a user-agent field, allow paths, disallow paths, and an optional crawl-delay. Add sitemap URLs at the bottom. Copy the output from the preview pane on the right and save it as robots.txt at the root of your site.

Frequently Asked Questions

What is a robots.txt file?
A robots.txt file tells search engine crawlers which pages on your site they can and cannot request. It sits at the root of your domain (e.g. example.com/robots.txt) and uses a simple text format with User-agent, Allow, and Disallow directives.
Can robots.txt block AI crawlers?
Yes. AI training crawlers like GPTBot, CCBot, Claude-Web, and Google-Extended respect robots.txt directives. This tool includes a 'Block AI Crawlers' preset that adds Disallow rules for all known AI training bots while keeping your site open to regular search engines.
What AI crawlers does the tool know about?
The preset covers GPTBot (OpenAI), CCBot (Common Crawl), Claude-Web (Anthropic), Google-Extended, Bytespider (ByteDance), anthropic-ai, Applebot-Extended, cohere-ai, PerplexityBot, and Amazonbot.
Does robots.txt guarantee pages won't be indexed?
No. Robots.txt controls crawling, not indexing. A page blocked by robots.txt can still appear in search results if other pages link to it. To prevent indexing, use a 'noindex' meta tag or X-Robots-Tag HTTP header.
Where should I put my robots.txt file?
Place it at the root of your domain so it is accessible at https://yourdomain.com/robots.txt. It must be at the root path to be discovered by crawlers.