Crawl Budget

Crawl budget is Google's allocation of crawling resources dedicated to a specific domain. Google determines this budget based on two factors: crawl capacity (server speed, responsiveness, and available bandwidth) and crawl demand (how frequently Google thinks the site updates and how valuable its pages are). A site with slow server response times or excessive crawl waste gets a smaller budget; a fast, well-structured site gets more crawl.Because Google's crawlers are finite, sites compete implicitly for crawl resources. A site with a small crawl budget may not get all its pages indexed quickly, or important updates may take longer to discover. This matters especially for large sites, sites with poor internal link structure, or those that serve crawlers unnecessary content like duplicate pages or massive pagination.

In practice, SEOs optimize crawl budget by improving site speed, fixing redirect chains, removing crawl traps (infinite parameter combinations), blocking low-value pages in robots.txt, and ensuring high-priority pages are easily reachable from the homepage. Monitoring crawl statistics in Google Search Console—especially crawl requests over time—helps identify whether budget constraints are limiting indexation or freshness.