Robots.txt is a small file that tells crawlers which parts of your site they can and cannot visit.
It lives at yoursite.com/robots.txt. It is a request, not a lock. Well-behaved bots like Google and AI crawlers follow it, but it does not secure private pages.
In short: A doorman for crawlers, pointing them away from areas you do not want crawled.
What it does for you
- Saves crawl budget. Keeps bots off low-value URLs.
- Tidier crawling. Stops admin, cart and search pages being crawled.
- High stakes. One wrong line can hide your whole site from Google.
Common rules
- Disallow junk. Block admin, cart and internal search URLs.
- Never block CSS/JS. Google needs them to render your pages.
- Link your sitemap. Add a Sitemap line so crawlers find it.
See Google's robots.txt guide.
Frequently asked questions
Does robots.txt stop a page being indexed?
No. It controls crawling, not indexing. To keep a page out of search use a noindex tag or password protection, not robots.txt.
Where is my robots.txt file?
At yoursite.com/robots.txt. Every site has one path for it, whether or not you have created the file.
What should I never block in robots.txt?
Your CSS and JavaScript. Google needs them to render the page, so blocking them can hurt how your pages are understood and ranked.