What Is Crawl Budget and How to Optimize It
Updated 11 September 2026 · 4 min read
Crawl budget is the number of URLs a search engine is willing to crawl on your site in a given period. It is a mix of two things: how much crawling your server can handle without slowing down (the crawl rate), and how much Google wants to crawl based on how useful and fresh your pages are (the crawl demand). When budget is spent on the wrong URLs, your important pages get crawled — and therefore indexed and updated — more slowly.
Do you even need to worry about it?
For most sites, no. Google has said crawl budget is mainly a concern for:
- Large sites (roughly tens of thousands of URLs or more), or
- Sites with many auto-generated URLs — faceted navigation, search-result pages, endless parameter combinations, or
- Sites where a lot of pages return errors or redirects.
If you run a small blog or a few dozen pages, focus on good content and internal links; crawl budget will take care of itself. If you have a big or fast-growing site, read on.
How to see where your budget goes
You cannot guess this — you have to look at what crawlers actually did. Two free sources:
- Google Search Console → Settings → Crawl Stats. Google's own record of Googlebot's requests, by response code and file type.
- Your server access log. Paste a week of it into the Log File Analyzer to see exactly which URLs every crawler hit, with status codes, and — if you add your sitemap — which of your pages were never crawled.
The number to watch is the share of crawler requests that ended in a redirect (3xx) or error (4xx/5xx) instead of a live page. Every one of those is budget that did not reach a real page.
The optimisation checklist
1. Fix the errors crawlers keep hitting
Repeated 5xx responses are the worst: they make Google slow down crawling of the whole site while it backs off. Repeated 404s waste budget on dead URLs. Find the internal links that point to them with the Broken Link Checker and fix or remove them.
2. Shorten redirect chains
An internal link to an old URL that 301s to a new one costs an extra request every time it is crawled. Multiply that across a big site and it adds up. Update internal links to point straight at the final URL — the Redirect Checker shows you the full chain for any URL.
3. Stop crawlers reaching low-value URLs
Faceted-search links, sort orders, session parameters and print versions can generate thousands of near-duplicate URLs. Keep crawlers off them with a Disallow rule in robots.txt, and avoid linking to them internally in the first place. Note that robots.txt stops crawling, not indexing — for a page you want kept out of results, use a noindex rule on a crawlable page instead.
4. Keep one URL per piece of content
Duplicate URLs split crawl budget across copies of the same page. Pick one canonical URL, link to it consistently, and set a self-referencing rel="canonical". Consistent trailing slashes and a single protocol (all HTTPS) matter here too.
5. Make important pages easy to reach
Crawl demand follows links. A page buried ten clicks deep, with no internal links pointing to it, will be crawled rarely. Link to your important pages from pages that are crawled often — your home page, category hubs and popular posts — so they sit near the surface of your site.
6. Keep your sitemap clean
List only canonical, indexable, 200-status URLs in your XML sitemap, and keep it up to date. A sitemap full of redirects, 404s or noindex URLs sends crawlers to dead ends. Validate yours with the Sitemap Checker.
7. Serve pages quickly
The faster your server responds, the more URLs a crawler can fetch in the time it allocates. A slow server directly lowers your crawl rate, so page speed is a crawl-budget lever as well as a ranking and user-experience one.
What not to do
- Don't add random internal links just to "spread" budget — link where it helps users.
- Don't rely on the
crawl-delaydirective for Google; Google ignores it and adjusts its rate automatically from how your server responds. - Don't remove your sitemap to "save" budget — it helps discovery, it doesn't cost budget.
Frequently asked questions
How do I know if crawl budget is actually a problem for me?
Check Crawl Stats and your logs. If Googlebot is crawling all your important pages within a reasonable time and few requests end in errors or redirects, budget is fine. If new pages take weeks to be crawled and a big share of requests hit 3xx/4xx/5xx, it is worth optimising.
Does blocking a URL in robots.txt save crawl budget?
Yes — Google won't crawl URLs disallowed in robots.txt, so it frees budget for the pages you do want crawled. Just remember it blocks crawling, not indexing.
Will improving crawl budget improve my rankings?
Not directly. Crawl budget affects how quickly and thoroughly your pages are discovered and refreshed. Better, faster discovery helps new and updated content get into the index sooner, but ranking still comes down to content quality and links.
How often should I check this?
For a large or fast-changing site, review Crawl Stats monthly and run a log analysis after any major change (a migration, a new section, a template change). For a small static site, a check every few months is plenty.