robots txt generator

How to use the robots txt generator

  1. Pick a preset: allow all, block AI crawlers, block everything, or custom.
  2. Add user-agent blocks one at a time — specify Googlebot, GPTBot, ClaudeBot, Bytespider, etc. — and pick allow/disallow per path.
  3. Add sitemap URLs — the generator appends Sitemap: lines at the end.
  4. Copy the output to /robots.txt at your site root.

When to use it

Ship a robots.txt when you want to block specific crawlers (aggressive scrapers, training-data bots) or limit crawl to specific paths. Don't use robots.txt to hide sensitive pages — it tells bots the path exists. Use noindex meta tags and authentication for actual secrecy. Alternative: a blank or missing robots.txt is equivalent to 'allow everything' — that's the right default for most sites.

Frequently asked questions

Does robots.txt block bots that ignore it?
No. Well-behaved crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot) respect it. Scrapers and malicious bots ignore it entirely — use server-level blocking (fail2ban, Cloudflare rules) for those.
Should I block GPTBot and ClaudeBot?
Depends on your monetization. Allow them if you want AI citations in ChatGPT/Claude search. Block them if you worry about your content being used as training data. Many publishers allow OAI-SearchBot (citations) but block GPTBot (training).
Do I need Sitemap: in robots.txt?
It's optional but recommended. Search engines discover your sitemap faster when it's listed there. Alternatively, submit it directly via Google Search Console or Bing Webmaster Tools.
What's the difference between Disallow: / and Disallow: /admin/?
Disallow: / blocks the whole site. Disallow: /admin/ blocks only /admin/ and deeper paths. Trailing slashes matter: /admin matches /admin-page too, /admin/ only matches /admin/ and below.

Related tools

Last updated: 2026-04-22