Technical SEO
Crawling
The process by which search engine bots discover and read web pages.
On this page
What is Crawling
Crawling is the process where search engine bots (like Googlebot) systematically browse the web to discover new and updated content. Bots follow links, read content, and send data back to be indexed.
Example
Googlebot discovers your new blog post by following an internal link from your homepage, then reads and stores the content.
How to apply it
Ensure a clear site structure, submit an XML sitemap, use internal linking, avoid crawl traps (infinite calendars, faceted navigation), and monitor crawl stats in GSC.
Tip:Check Server Logs to see exactly what Googlebot is crawling. GSC's crawl stats only show aggregates.
Track it in GSC Boost
GSC Boost reads your Search Console data and tells you what to fix first, with the clicks each fix is likely to bring. Start with the guide or sign up free.
Related terms
IndexingThe process of storing and organizing crawled pages in search engine databases.Robots.txtA file that instructs search engine crawlers which pages to crawl or avoid.XML SitemapA file listing important URLs to help search engines discover and crawl content.Crawl BudgetThe number of pages Googlebot will crawl on your site in a given time.
Related resources
Was this page helpful?