robots.txt for ecommerce stores: what to allow, block and link
A robots.txt file tells crawlers what they may visit and where your sitemap is. See what a healthy store file contains and how to check yours.
2 min read
Area
Store discovery
Severity
Medium
Fix effort
Small
Stores failing
6%
Of 35 stores we scanned where this check applies, 6% fail it and 80% pass. See all benchmarks.
The robots.txt file sits at the root of your site and tells search engine crawlers which paths they may visit. A good one lets them reach products and collections, keeps them out of carts and accounts, and points to your sitemap. This check looks for all three.
Why it matters
A single wrong line can block the whole store from search. It happens more often than it should, usually when a staging rule is copied to production.
It is also where you tell crawlers about the sitemap, which helps new products get found sooner.
What the audit looks at
The audit checks that your robots.txt exists, lets search engines reach the store and points to your sitemap.
What usually causes it
- A leftover "Disallow: /" from a staging or password-protected phase
- No robots.txt because the platform default was removed
- Rules that block filter and search URLs too broadly and catch real collections
- A missing Sitemap line, so crawlers have to find pages the slow way
How to fix it
Publish a robots.txt that blocks only private areas (cart, checkout, account) and lists your sitemap. The effort is small, usually a setting or a paste and under 30 minutes.
- Serve a file at /robots.txt with User-agent: *, Disallow lines for cart, checkout and account pages only, and a Sitemap: line.
- Make sure nothing says Disallow: / for all agents.
On Shopify
- Shopify writes a sound robots.txt for you. If it is missing or blocking everything, a theme or app has overridden it: check Online Store > Themes > Edit code for a robots.txt.liquid and restore the default.
On WooCommerce
- Settings > Reading: make sure "Discourage search engines from indexing this site" is off.
- In Yoast or Rank Math, open the robots.txt editor and add the Sitemap line and the cart, checkout and my-account Disallow lines.
How to check it yourself
- Open yourstore.com/robots.txt in a browser and read it.
- Use the robots.txt report in Google Search Console to see how Google reads it.
- Test a few product and collection URLs against the rules to confirm they are allowed.
Frequently asked questions
Does robots.txt hide a page from Google?
It stops crawling, but a blocked page can still appear in results if other sites link to it. To keep a page out, use a noindex tag on a page that is allowed to be crawled.
Can I edit robots.txt on Shopify?
Shopify generates it, and lets you customise it through a robots.txt.liquid template in the theme.
Related checks
- XML sitemaps for online stores: how to check and submit yours
- www vs non-www: how to make your store open at one address
- Custom 404 pages for stores: return a real 404 and help shoppers back in
Further reading
Check your own store
See whether your store passes this check
Paste your store URL. The audit loads your pages in a real browser and shows the result for this check with the evidence behind it.