What is Robots.txt
Robots.txt is a text file placed in your site's root directory that tells search engine crawlers which URLs they can and cannot access. It's used to manage crawl budget and prevent indexing of certain content.
Example
User-agent: * Disallow: /admin/ Disallow: /private/ Allow: /public/ Sitemap: https://example.com/sitemap.xml
How to apply it
Block admin pages, staging environments, search results pages, and low-value filter/sort URLs. Always allow CSS/JS files. Include sitemap location.
Explore your own search data
Connect Search Console to AnotherSEOGuru App to see which searches bring people to a page and how its clicks have changed. Start with the guide or sign up free.