Robots.txt Guide: Syntax, Rules and Common Mistakes

A robots.txt file is a plain text file at the root of your site that tells crawlers which parts of the site they may and may not request. It's one of the oldest conventions on the web, and since 2022 it has had a formal standard: RFC 9309, the Robots Exclusion Protocol.

It's also one of the easiest files to get wrong. A single misplaced slash can stop search engines from crawling your whole site. This guide explains the syntax, how crawlers decide which rule applies, how Google handles edge cases, and the mistakes worth checking for.

Where robots.txt lives#

The file must be named robots.txt, in lowercase, and sit at the top level of a host:

https://www.example.com/robots.txt

A few rules follow from this:

• It applies per host and protocol. https://www.example.com/robots.txt doesn't cover https://shop.example.com/. Each subdomain needs its own file.
• It can't live in a subfolder. A file at /blog/robots.txt is ignored.
• It should be plain text, UTF-8 encoded. Serve it with the text/plain media type.

The basic syntax#

A robots.txt file is made of groups. Each group starts with one or more User-agent lines, followed by one or more rules.

# Rules for all crawlers
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Allow: /

# A separate group for one specific crawler
User-agent: ExampleBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

The fields you'll use:

• User-agent names the crawler a group applies to. * matches any crawler that doesn't have a more specific group.
• Disallow gives a path the crawler shouldn't request.
• Allow gives a path the crawler may request, typically used to open up part of a disallowed section.
• Sitemap gives the full URL of a sitemap. It isn't tied to any group.
• # starts a comment.

Two details trip people up:

• User-agent matching is case-insensitive, so googlebot and Googlebot are the same.
• Paths are case-sensitive. Disallow: /Private/ doesn't block /private/.

An empty Disallow: line means nothing is disallowed for that group.

How a crawler picks its group#

A crawler reads the file and looks for the group that matches its own product token most specifically. It follows only that group and ignores the rest.

That has an important consequence. In the example above, ExampleBot follows only its own group, so it's blocked from everything. It does not also inherit the * rules. If you create a group for a specific crawler, repeat any rules from the * group that you want it to follow.

If no group matches a crawler specifically, it uses the * group. If there's no * group either, the crawler can fetch everything. RFC 9309 also says that if several groups name the same user agent, their rules are combined.

Wildcards: * and $#

RFC 9309 defines two special characters for paths:

• * matches zero or more of any character.
• $ marks the end of the URL path.

Some examples:

User-agent: *
# Block any URL containing a session parameter
Disallow: /*?sessionid=

# Block all PDF files anywhere on the site
Disallow: /*.pdf$

# Block /search but not /search-tips/
Disallow: /search$

Without the $, Disallow: /*.pdf would also block a URL like /files/report.pdf?version=2. With it, only paths ending exactly in .pdf are matched.

Rules are prefix matches by default. Disallow: /admin blocks /admin, /admin/, /admin/users and also /administrator-guide/. If you only mean the folder, write Disallow: /admin/.

The longest-match rule#

When both an Allow and a Disallow rule match a URL, which one wins? RFC 9309 says crawlers must use the most specific match, meaning the rule with the longest matching path. If an Allow and a Disallow rule are equally specific, the Allow rule should be used.

User-agent: *
Disallow: /shop/
Allow: /shop/sale/

For /shop/sale/boots/, both rules match. /shop/sale/ is longer than /shop/, so the Allow wins and the page can be crawled. For /shop/boots/, only the Disallow matches, so it's blocked.

The order of lines in the file doesn't matter for this. Specificity does.

How Google handles robots.txt#

Google's robots.txt documentation adds some practical details:

• Size limit: Google processes up to 500 KiB. Content after that is ignored.
• Caching: Google generally caches the file for up to 24 hours, sometimes longer if the file can't be refreshed.
• Redirects: Google follows up to five redirect hops to reach the file.
• 4xx errors: a 404 or other client error is treated as if there's no robots.txt, so everything can be crawled. The exception is 429 Too Many Requests.
• 5xx errors: Google treats server errors as a temporary full block and pauses crawling. If the errors continue for a long time, it falls back to its last cached copy for a while, and eventually treats the site as having no restrictions.
• Supported fields: user-agent, allow, disallow and sitemap. Google doesn't support crawl-delay (some other crawlers do) and doesn't support noindex inside robots.txt.

The server error behavior matters more than most people realize. If your robots.txt starts returning 500 errors because of a misconfigured server, Google may stop crawling your site until it's fixed.

The Sitemap line#

Adding your sitemap to robots.txt is an easy way to help crawlers find it:

Sitemap: https://www.example.com/sitemap.xml
Sitemap: https://www.example.com/sitemap-products.xml

Use full, absolute URLs. You can list several sitemaps. The line can go anywhere in the file, because it isn't part of any user-agent group. For more on building one, see how to create a sitemap, and use our Sitemap Checker to confirm it's valid.

Common mistakes#

Blocking the whole site by accident#

User-agent: *
Disallow: /

This is usually left over from a staging site that went live. Check your robots.txt after every launch or migration.

Using robots.txt to hide pages from search#

This is the most common misunderstanding. Google's introduction to robots.txt says plainly that robots.txt isn't a mechanism for keeping a page out of Google. A blocked URL can still appear in search results, without a description, if other pages link to it.

To keep a page out of search results, use a noindex robots meta tag or X-Robots-Tag HTTP header. And here's the catch: Google's guide to blocking indexing with noindex points out that the page must not be blocked by robots.txt, or crawlers will never see the noindex instruction.

<meta name="robots" content="noindex">

Blocking CSS and JavaScript#

Google renders pages much like a browser. If you block the CSS and JavaScript files your pages need, Google may not be able to understand the page layout or content. Only block resources you're sure aren't needed to render the page.

Treating robots.txt as security#

Robots.txt is public. Anyone can read it. Listing /secret-admin-panel/ in a Disallow line tells everyone where it is. Protect private areas with authentication, not robots.txt.

Forgetting that specific groups don't inherit#

As covered above, a crawler follows only its most specific group. Adding a group for one bot and assuming it still follows the * rules is a common source of surprises.

Returning HTML instead of text#

Some servers return a normal HTML page, or a soft error page with a 200 status, at /robots.txt. Crawlers then try to parse HTML as rules. Make sure the URL returns real plain text, or a proper 404 if you don't want a file.

A solid starting template#

For a typical content site or small store, a simple file like this is often enough:

User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/
Disallow: /*?sort=
Disallow: /*?filter=

Sitemap: https://www.example.com/sitemap.xml

It keeps crawlers out of user-specific pages and endless sorted or filtered versions of listing pages, while leaving everything else open. Adjust the paths to match your platform.

If you also want to control access for AI crawlers specifically, see how to allow or block AI crawlers.

Test before and after every change#

Before you publish a change, run your file through our Robots.txt Checker and test the URLs that matter most, such as your homepage, a key category page and a product page, to confirm they're allowed. After publishing, fetch https://yourdomain.com/robots.txt in a browser to make sure the live file is the one you expect.

A robots.txt file only needs a few lines to do its job. Keep it short, test it, and use noindex rather than Disallow whenever your real goal is keeping a page out of search results.