Robots.txt Checker

How to use the Robots.txt Checker#

1. Enter the site's URL, for example https://example.com/. We fetch /robots.txt from that host.
2. Optionally enter a path to test, such as /blog/ or /search?q=shoes.
3. Run the check.
4. Read the verdict for each crawler: allowed or blocked, with the rule and line number that decided it.
5. Review the full file, shown with line numbers, along with any warnings and the Sitemap lines found.

Understanding your results#

Crawler verdicts#

We test your path for Googlebot, Bingbot, GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot and a few others. GPTBot and OAI-SearchBot are OpenAI's crawlers, ClaudeBot is Anthropic's, PerplexityBot is Perplexity's and CCBot is Common Crawl's. Google-Extended is different: it is a control token that Google checks, not a separate crawler, so blocking it does not stop Googlebot from crawling for Search.

How the rules are matched#

We follow RFC 9309 and Google's documented behaviour:

• Which group applies. A crawler obeys the group with the most specific user-agent that matches it. If none matches, it uses the User-agent: * group. If there is no such group either, no rules apply to it.
• Which rule wins. Within the group, the longest matching Allow or Disallow path wins. If an Allow and a Disallow rule match with equal length, Allow wins.
• Wildcards. * matches any sequence of characters, and $ marks the end of the URL. For example, Disallow: /*.pdf$ blocks URLs ending in .pdf.

Warnings#

• Syntax problems: lines we cannot parse, rules outside any user-agent group, or missing colons.
• Unknown directives: lines that crawlers will ignore, often typos such as Dissallow.
• Crawl-delay: Google does not support it. Other crawlers may honor it.
• File size over 500 KiB: Google ignores any content after that limit.

Sitemap lines#

Every Sitemap: line is listed. These must be full URLs and help crawlers find your sitemaps. You can validate them with the Sitemap Checker.

Status codes#

• 4xx (such as 404): there is no usable file, so crawlers may access everything. Google treats a 429 as a server error rather than a 4xx.
• 5xx: RFC 9309 says crawlers must assume everything is disallowed. Google stops crawling the site for 12 hours while it retries, then may use a cached copy for up to 30 days. A broken server response for robots.txt can therefore pause crawling of the whole site.

How this tool works#

We fetch /robots.txt from the host you entered, record the status code, parse the file with the rules above and test your path for each crawler. Rules apply only to the host and protocol they are served from, so blog.example.com needs its own file.

Robots.txt controls crawling, not indexing. A blocked URL can still appear in search results if other pages link to it, usually without a description. To keep a page out of results, use a noindex rule and make sure the page is not blocked, because a crawler that cannot fetch the page will never see the noindex. Google explains this in its guide to blocking indexing with noindex.

Compliance is voluntary. The file tells well-behaved crawlers what you want; it does not technically prevent access. Our results show what the rules say, not whether a given bot obeys them.

Tips#

• Do not block the CSS and JavaScript files your pages need; search engines use them to render pages.
• To remove a page from search, use noindex instead of Disallow.
• List every sitemap with a Sitemap: line using its full URL.
• Test important paths after every edit. One stray Disallow: / blocks the whole site.
• Decide your policy for AI crawlers deliberately, then set rules for each user-agent you want to allow or block.
• Consider publishing an llms.txt file too; the llms.txt Generator can draft one.
• For examples of common setups, read our robots.txt guide.

Frequently asked questions#

Does robots.txt stop a page from appearing in Google?#

No. It stops crawling, not indexing. A blocked page can still be indexed from links elsewhere. Use a noindex rule on a crawlable page to keep it out of search results.

What happens if my robots.txt returns a 404?#

Crawlers treat it as if there is no robots.txt and may crawl everything. That is fine if you do not need any restrictions.

Which rule wins when Allow and Disallow both match?#

The rule with the longest matching path wins. If both match with the same length, Allow wins. This is how RFC 9309 and Google handle conflicts.

Why does Google ignore crawl-delay?#

Google does not support the crawl-delay field and adjusts its crawl rate automatically based on how your server responds. Other crawlers may still honor the field.

How do I block AI crawlers?#

Add a group for each crawler's user-agent, such as User-agent: GPTBot followed by Disallow: /. Test the result here to confirm each one is blocked while Googlebot is still allowed.

Does each subdomain need its own robots.txt?#

Yes. A robots.txt file applies only to the host and protocol it is served from, so shop.example.com is not covered by the file at example.com.