If you are setting up a new website or auditing an existing one, robots.txt is one of the first technical SEO files worth understanding. It is small, simple, and easy to get wrong: a single misplaced rule can accidentally block search engines from your entire site.
This guide walks through what robots.txt actually does, how to create one, and the mistakes to avoid.
What Is a Robots.txt File?
A robots.txt file is a plain text file that lives at the root of your domain, for example https://yoursite.com/robots.txt. It gives instructions to search engine crawlers, like Googlebot, about which parts of your site they are allowed to crawl and which parts they should avoid.
It is one of the first things a well-behaved crawler checks before indexing a site, which is why getting it right matters even for small websites.
Why Robots.txt Matters for SEO
Robots.txt will not directly boost your rankings, but it plays a supporting role in a few important ways:
- Managing crawl budget. Large sites can guide crawlers away from low-value pages, like admin areas, duplicate filtered pages, or internal search results, so crawl activity focuses on the content that matters.
- Avoiding accidental exposure. You can keep crawlers out of staging areas, admin panels, or private directories, while remembering that robots.txt is not a security tool.
- Pointing to your sitemap. Robots.txt commonly includes a reference to your XML sitemap, helping search engines discover your pages faster.
Basic Robots.txt Syntax
A robots.txt file is made up of a small set of directives:
- User-agent specifies which crawler the following rules apply to. Use
*to apply rules to all crawlers, or name a specific one likeGooglebot. - Disallow tells the specified crawler not to access a given path.
- Allow creates an exception within a disallowed path, letting a specific subfolder or file through.
- Sitemap provides the full URL to your XML sitemap.
A simple example:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Allow: /admin/help-page.html
Sitemap: https://yoursite.com/sitemap.xmlThis tells all crawlers to avoid the /admin/ and /cart/ directories, except for one specific help page, and points them to the sitemap.
How to Create a Robots.txt File Online
Writing a robots.txt file by hand is straightforward for simple sites, but using an online robots.txt generator is faster and reduces the risk of a syntax mistake. Typically the process looks like this:
- Decide which folders or pages, if any, need to be blocked from crawlers.
- Choose whether the rules should apply to all crawlers or specific ones.
- Add your sitemap URL so crawlers can find it easily.
- Generate the file and copy it.
- Upload it to the root directory of your website, not a subfolder.
How to Check If Your Robots.txt File Is Working
Once uploaded, visit https://yoursite.com/robots.txt directly in a browser to confirm it is live and displaying the rules you expect. Google Search Console also includes a robots.txt report that can show whether Google can read your file and which URLs are blocked.
After creating your crawler rules, use the free XML sitemap generator to prepare a sitemap URL and reference it from robots.txt.
Common Robots.txt Mistakes to Avoid
Blocking the entire site by accident. A stray Disallow: / at the top level tells every crawler to avoid your entire website. This is one of the most common and most damaging robots.txt mistakes, especially after a site migration when a staging rule sometimes accidentally makes it into the live file.
Confusing Disallow with noindex. Robots.txt controls crawling, not indexing. A page blocked in robots.txt can technically still appear in search results, usually without a description, if other pages link to it. To reliably keep a page out of search results, use a noindex meta tag on the page itself instead.
Forgetting the sitemap reference. It is a small addition, but including your sitemap URL in robots.txt is a simple way to help search engines discover new and updated pages faster.
Placing the file in the wrong location. Robots.txt must sit at the root of the domain. A file placed inside a subfolder, like /blog/robots.txt, will be ignored by crawlers.
Final Thoughts
Robots.txt is a small file with an outsized ability to help, or accidentally hurt, your site's visibility in search. Use an online robots.txt generator to create one quickly and correctly, double-check it in Google Search Console once it is live, and revisit it whenever you restructure your site.