Robots.txt Generator - Build Robots.txt Online
Import existing robots.txt
User-agent: * Allow: / Disallow: /admin Disallow: /api Disallow: /private Disallow: /tmp
Disallow: /adminUsing the "User-agent: *" group.Follows Google's rules: the most specific user-agent group applies, the longest matching path wins, and Allow wins a tie. Supports * and $.
Build a robots.txt file
Start from a preset, then edit the rule groups: each has a user agent, Allow and Disallow paths, and an optional
Crawl-delay. Add your sitemap URLs at the bottom. The preview updates as you type. Save the result as
robots.txt at the root of the host, such as https://example.com/robots.txt. Each subdomain needs
its own file; shop.example.com does not read the one on example.com.
Presets
- Allow all. Every crawler can fetch every path.
- Block all.
Disallow: /for everyone. For staging sites, but password-protect them as well, since a blocked staging URL can still be indexed from links. - Block AI crawlers. Keeps search engines in and asks 12 AI crawlers to stay out.
- Standard SEO. Allows everything except
/admin,/api,/privateand/tmp.
Example
User-agent: *
Disallow: /admin
Allow: /admin/public/
User-agent: GPTBot
Disallow: /
Sitemap: https://example.com/sitemap.xml
Here /admin/settings is blocked for every crawler, but /admin/public/logo.png is allowed, because
/admin/public/ is the longer match. GPTBot has its own group, so it ignores the * rules and is
blocked everywhere. A crawler only ever follows one group, which catches people out: rules in * do not also
apply to a bot that has a group of its own.
Common mistakes
- Blocking CSS and JavaScript folders. Google renders pages, and a page it cannot render can rank worse.
- Using robots.txt to hide private URLs. The file is public, so it lists exactly where to look.
- Paths are case-sensitive:
Disallow: /Admindoes not block/admin.