Technical SEO
Robots.txt
A file that instructs search engine crawlers which pages to crawl or avoid.
On this page
What is Robots.txt
Robots.txt is a text file placed in your site's root directory that tells search engine crawlers which URLs they can and cannot access. It's used to manage crawl budget and prevent indexing of certain content.
Example
User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /public/ Sitemap: https://example.com/sitemap.xml
How to apply it
Block admin pages, staging environments, search results pages, and low-value filter/sort URLs. Always allow CSS/JS files. Include sitemap location.
Tip:Robots.txt blocks crawling, not indexing. If pages have external links pointing to them, they can still appear in search results.
Track it in GSC Boost
GSC Boost reads your Search Console data and tells you what to fix first, with the clicks each fix is likely to bring. Start with the guide or sign up free.
Related terms
CrawlingThe process by which search engine bots discover and read web pages.Noindex TagA directive telling search engines not to include a page in their index.Crawl BudgetThe number of pages Googlebot will crawl on your site in a given time.
Related resources
Was this page helpful?