Sign up

Advanced Concepts

Crawl Budget

The number of pages Googlebot will crawl on your site in a given time.

On this page

What is Crawl Budget

Crawl budget is the number of pages Google's crawler will visit on your website within a certain timeframe. For large sites, optimizing crawl budget ensures important pages get crawled frequently while low-value pages don't waste resources.

Example

A site with 1 million pages but crawl budget for 10,000/day means it takes 100 days to crawl everything once.

How to apply it

Block low-value pages in robots.txt, fix crawl errors, improve page speed, remove duplicate content, optimize internal linking to important pages.

Tip:Crawl budget only matters for large sites (100K+ pages). Small sites will have everything crawled regardless of optimization.

Track it in GSC Boost

GSC Boost reads your Search Console data and tells you what to fix first, with the clicks each fix is likely to bring. Start with the guide or sign up free.

Related terms

CrawlingThe process by which search engine bots discover and read web pages.Robots.txtA file that instructs search engine crawlers which pages to crawl or avoid.IndexingThe process of storing and organizing crawled pages in search engine databases.

Related resources

Was this page helpful?

Sign up freeGet a quote