# Robots.txt

> A file that instructs search engine crawlers which pages to crawl or avoid.

Source: https://anotherseoguru.com/en/glossary/robots-txt

Robots.txt is a text file placed in your site's root directory that tells search engine crawlers which URLs they can and cannot access. It's used to manage crawl budget and prevent indexing of certain content.

## Example

User-agent: *
Disallow: /admin/
Disallow: /private/
Allow: /public/
Sitemap: https://example.com/sitemap.xml

## How to apply it

Block admin pages, staging environments, search results pages, and low-value filter/sort URLs. Always allow CSS/JS files. Include sitemap location.

> **Pro tip:** Robots.txt blocks crawling, not indexing. If pages have external links pointing to them, they can still appear in search results.

## Related terms

- [Crawling](https://anotherseoguru.com/en/glossary/crawling)
- [Noindex Tag](https://anotherseoguru.com/en/glossary/noindex)
- [Crawl Budget](https://anotherseoguru.com/en/glossary/crawl-budget)