How to use the Robots.txt Generator
- Pick how you want crawling handled: allow all pages (the normal default), block the whole site (staging only), or block specific paths.
- If you chose specific paths, list them one per line, like
/admin/or/cart/. - Decide whether to allow or block AI crawlers such as GPTBot, ClaudeBot and Google-Extended.
- Add your sitemap URL so search engines can find every page.
- Copy the generated file, save it as plain text named exactly
robots.txt, and upload it to your domain root so it answers athttps://yourdomain.com/robots.txt.
Understanding your results
The file. This is your complete robots.txt, ready to copy. It always starts with a User-agent: * block that applies to every crawler, followed by any extra blocks you asked for.
Allow all vs block all. An empty Disallow: means "crawl everything" — correct for a normal public site. Disallow: / blocks the entire site and, on a live site, removes it from search over time, so the tool flags it clearly.
AI crawler rules. If you choose to block them, the file adds named User-agent blocks for the main AI crawlers (OpenAI's GPTBot, Anthropic's ClaudeBot, Google-Extended, Common Crawl's CCBot and more), each with Disallow: /. This is a polite request that well-behaved bots honour — it is not enforcement.
The sitemap line. Listing your sitemap helps crawlers discover every URL. If you leave it out, the tool reminds you to add it.
How this tool works
The generator builds the file purely from your choices — it fetches nothing and stores nothing. It normalizes the paths you enter (stripping any pasted domain and adding a leading slash) and assembles valid directives in the order crawlers expect.
Remember what robots.txt does and does not do: it controls crawling, not indexing or access. A page blocked here can still appear in search if other sites link to it, and anyone can read your robots.txt. Once you have uploaded the file, test it with the Robots.txt Checker, and validate the sitemap you linked with the Sitemap Checker.
Tips
- Put it in the root. A robots.txt in a subfolder is ignored. It must be at
/robots.txt. - Never hide private data with it. robots.txt is public and only requests crawlers stay away. Use real authentication for anything sensitive.
- To remove a page from Google, use noindex, not a Disallow. A disallowed page can still be indexed without its content.
- Do not block your CSS or JS. Google needs them to render and rank your pages.
- Keep the sitemap line accurate — a full
httpsURL that actually resolves.
Frequently asked questions
Where do I put the robots.txt file?
In the root of your domain, so it is reachable at https://yourdomain.com/robots.txt. Crawlers only look there; a file anywhere else is ignored.
Does robots.txt stop a page from being indexed?
No. It stops crawling, not indexing. If other sites link to a blocked page, Google can still list the URL. To keep a page out of search, use a noindex meta tag and allow it to be crawled.
Should I block AI crawlers like GPTBot?
That is your call. Block them if you do not want your content used to train or answer inside AI tools; allow them if the referral visibility is worth it. The generator makes either choice a single click, and you can change it any time.
Is blocking AI bots in robots.txt actually enforced?
Only by bots that choose to obey it. Reputable crawlers (GPTBot, Google-Extended, ClaudeBot, CCBot) document that they honour robots.txt, but it is a request, not a technical block.
What is the difference between this and llms.txt?
robots.txt controls crawling for all bots. llms.txt is a newer, separate file that points AI tools at your most important content. They complement each other — many sites publish both.