Technical SEO

Robots.txt

A file that instructs search engine crawlers which pages to crawl or avoid.

On this page

What is Robots.txt

Robots.txt is a text file placed in your site's root directory that tells search engine crawlers which URLs they can and cannot access. It's used to manage crawl budget and prevent indexing of certain content.

Example

User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /public/ Sitemap: https://example.com/sitemap.xml

How to apply it

Block admin pages, staging environments, search results pages, and low-value filter/sort URLs. Always allow CSS/JS files. Include sitemap location.

Tip:Robots.txt blocks crawling, not indexing. If pages have external links pointing to them, they can still appear in search results.

Explore your own search data

Connect Search Console to AnotherSEOGuru App to see which searches bring people to a page and how its clicks have changed. Start with the guide or sign up free.

Related resources

Was this page helpful?

Need help? support@anotherseoguru.com

Sign up freeLive demo