Crawl budget is one of the most over-invoked concepts in SEO. For a 300-page marketing site it is essentially irrelevant. For a storefront where filters generate combinatorial URLs, it is the difference between new products appearing in search within a day and appearing in six weeks.
This guide is about telling those two situations apart, and fixing the second one.
What crawl budget actually is
Two forces decide how much of your site gets crawled:
Crawl rate limit — how much load the crawler is willing to put on your server. Slow responses and 5xx errors push this down; fast, stable responses let it rise.
Crawl demand — how much the crawler wants your URLs. Driven by popularity, internal linking, and how often content genuinely changes.
The effective budget is the smaller of the two. A fast server with no demand gets crawled lightly; a popular site on a slow server gets throttled.
Does your site need to care?
Be honest about this before spending a sprint on it:
| Situation | Crawl budget concern |
|---|---|
| Under ~1,000 URLs, stable | No. Fix content and links instead |
| 1,000–10,000 URLs | Marginal. Worth a check, not a project |
| Over 10,000 URLs | Yes |
| Faceted navigation or search result URLs | Yes, regardless of size |
| Frequently updated catalogue or feed | Yes |
| Slow server (TTFB over ~600 ms) | Yes |
If you are in the first row, close this tab and go work on internal links — it will pay back more.
Where the budget goes: read your logs
Every diagnosis starts with server logs, because they show what the crawler actually requested rather than what you assume it requested. Filter to verified search engine user agents and aggregate by URL pattern:
- Which path patterns absorb the most requests?
- What share of requests return 200 vs 3xx vs 4xx vs 5xx?
- How many requests hit URLs that carry query parameters?
- When was each important page last crawled?
The classic finding is that 60–80% of crawl requests land on parameterised URLs that nobody wants indexed, while genuinely important category pages are visited once a month.
The five biggest drains, and their fixes
1. Faceted navigation
?color=black&size=m&sort=price&page=3 — a handful of filters produces thousands of URL combinations, all with near-identical content.
Fix, in order of preference:
- Make filters that should not be indexed use a non-crawlable mechanism (POST, or
#fragments handled client-side). - Canonicalise filtered views to the unfiltered category page.
- For filter combinations that genuinely deserve indexing — a real "black dresses" demand pattern — build a clean, linked, canonical URL for exactly those and let the rest be uncrawlable.
robots.txtdisallow as a last resort. It stops the crawl but leaves any already-indexed URLs stuck, because the crawler can no longer read thenoindex.
2. Redirect chains
Every hop is a request. A → B → C → D spends four requests to deliver one page, and link equity degrades along the way. Crawl your site, export every 3xx, and collapse each chain to a single hop. Then fix the sources: internal links pointing at the old URL should be updated to point at the destination directly. A redirect is a safety net, not a linking strategy.
3. Soft 404s and thin pages
Pages returning 200 with "no results found", empty category pages, and expired campaign pages all consume crawl and dilute quality signals. Return a real 404 or 410 for genuinely gone content, and consolidate empty categories.
4. Duplicate URL forms
The same document reachable as /page, /page/, /Page, /page?utm_source=x, http:// and https://, www and non-www is six crawls for one page. Enforce one canonical form with server-side redirects, keep internal links consistent with it, and strip tracking parameters before they enter your link graph.
5. Slow responses
Crawl rate is directly sensitive to response time. Cutting TTFB from 800 ms to 200 ms often produces a visible increase in daily crawl volume within weeks, with no other change. See Core Web Vitals for product teams for where that time usually hides.
Spending the budget on the right pages
Reducing waste is half the job; the other half is directing attention.
Flatten depth. A page four clicks from the homepage is crawled far less than one two clicks away. Hub pages, well-designed category structures, and contextual links inside content all pull important pages up. Your IA is the lever here — if the information architecture is deep and narrow, no amount of sitemap tuning compensates.
Kill orphans. A URL in the sitemap with no internal links pointing at it gets discovered and then largely ignored. Every page worth indexing should be reachable by clicking.
Keep lastmod honest. Accurate change timestamps let a crawler skip unchanged pages and revisit changed ones. Falsifying them wastes the mechanism, as covered in the sitemap guide.
Segment your sitemaps. Typed sitemap files let you see, per content type, whether crawling is reaching what matters.
A 30-day plan
- Week 1 — Pull 30 days of server logs. Classify crawl requests by path pattern and status code. Crawl the site with DeepAnalyzer and export the full URL inventory with status codes and canonicals.
- Week 2 — Diff the two. Everything crawled but not in your intended inventory is waste; everything in the inventory but never crawled is a discovery problem.
- Week 3 — Fix the top three waste patterns. Usually: parameter handling, redirect chains, and one runaway faceted section.
- Week 4 — Re-pull logs. The share of requests hitting 200-status canonical pages should be measurably higher.
Measure the ratio, not the absolute request count. Total crawl volume moves for reasons outside your control; the proportion landing on pages you want indexed is the number you actually influence.
FAQ
How do I know if crawl budget is my problem?
Compare "last crawled" dates in Search Console for important pages against how often you publish. If pages you updated three weeks ago have not been recrawled and your logs show heavy crawling of parameter URLs, budget is being misallocated. If important pages are crawled daily and still not indexed, the problem is quality, not budget.
Does blocking URLs in robots.txt save crawl budget?
Yes, but it is blunt. Blocked URLs already in the index stay there without content, because the crawler can no longer see a noindex. Prefer canonicalisation or removing the links entirely, and use robots.txt for sections that should never have been crawlable at all.
Does nofollow on internal links preserve budget?
Not meaningfully. nofollow no longer conserves flow the way it did historically, and the URL can still be discovered elsewhere. Structural fixes — not linking to it, or not generating the URL — work better.
Do large sitemaps help crawl budget?
They help discovery, not budget. Listing 50,000 URLs does not earn more crawling; it just makes the crawler aware of more URLs it may choose to ignore. Fix depth and internal linking first.
How long until changes show up?
Log-level changes appear within days. Index coverage changes typically take two to six weeks, longer on large sites. Resist the urge to change five things in one week — you lose the ability to attribute the result.