robots.txt is the one file that decides which crawlers get into your store, and a single stray line in it can turn away the exact AI agents you want shopping your catalog. Most merchants have never opened theirs. In the AI era, that one file is the difference between being read and being skipped.
- robots.txt is the one file that decides which crawlers get into your store, a single stray line can turn away the AI agents you want.
- Amazon blocks the ChatGPT crawlers in robots.txt, so its listings can’t surface in ChatGPT shopping; a reachable Shopify store can appear where Amazon can’t.
- Shopify manages much of your robots.txt, but you can still add rules that silently block the wrong user-agent.
- Read yours now: a misplaced
Disallowunder an AI user-agent is enough to make you invisible to that engine.
robots.txt is the bouncer at your front door. It is the first thing a crawler checks, and it answers one question: am I allowed in, and where. For two decades the only bot anyone worried about was Googlebot. Now the same file quietly governs whether GPTBot, ClaudeBot, and PerplexityBot get to read the store at all, and almost nobody has reread it with the new guests in mind.
What it does, in one breath
The file lives at yourstore.com/robots.txt. It lists user-agents and the paths they may or may not crawl. A line like Disallow: / under a bot’s name slams the door entirely. The danger is not malice. It is inheritance: a rule added years ago, or shipped by an app or theme, that now blocks an agent you would happily welcome.
The Shopify-specific part
Shopify generates a sensible default robots.txt for you, and for a long time you could not touch it. You can now, through the robots.txt.liquid template. That is power and risk in the same lever: edit it carefully and you can welcome the AI crawlers explicitly; edit it carelessly and you can lock out the ones driving discovery. If you have apps that inject their own rules, this is where their fingerprints show up.
| robots.txt line | What it means | Effect on AI |
|---|---|---|
User-agent: GPTBot / Disallow: /
|
Block OpenAI’s crawler entirely | Invisible to ChatGPT, the Amazon move |
User-agent: * / Disallow: /checkout
|
Block a path for all crawlers | Fine. Keeps private paths out |
User-agent: * / Disallow: /
|
Block everything | Catastrophic. Nobody can read you |
Allow: / (default) |
Let well-behaved crawlers in | Reachable, the readable layer still has to deliver |
How to read yours right now
Open yourstore.com/robots.txt in a browser. It is public, on every store, no tools required. Look for any Disallow lines, and for named user-agents like GPTBot or CCBot. If you see a bot you meant to allow sitting under a block, you just found why it is not describing you. Then make sure the doors that are open lead somewhere a machine can read, that is where your sitemap and the broader crawler-access picture come in. And know exactly who the named crawlers are before you decide who to let through.
A scan is a snapshotlegibility drifts
A correct robots.txt is never settled. A theme update rewrites your robots or templates, an app injects a script, a migration spawns redirects, and the layer a crawler reads regresses silently while the storefront still looks perfect to you. Your store changes weekly, so being reachable and readable is a moving target, not a one-time pass. That is why serious stores measure, fix, and re-measure, and why we re-scan our own store on a schedule, in public.
Questions people actually ask
Where is my robots.txt file?
At yourstore.com/robots.txt. It is public on every store, so you can open it in any browser right now without logging in. If you want to change it on Shopify, that is done through the robots.txt.liquid template in your theme.
Should robots.txt block AI crawlers?
For most ecommerce stores, you want it to allow the crawlers that feed AI shopping answers, not block them. Blocking is a deliberate choice for specific cases, not a safe default. The common error is blocking a bot by accident through an old or app-added rule.
Can a robots.txt mistake really make my store invisible to AI?
Yes. If a crawler is disallowed, it does not enter, so nothing else you do on the page can be seen by it. That is what makes this file the front door: every other improvement depends on the door being open first.
See what a machine sees
You can't tell from your browser whether AI can read your store. You can find out in a few minutes. Run a free scan and see the exact layer the machine reads, and where you're losing the shortlist.
Sources: Similarweb (2025) on AI-referral traffic and Amazon’s crawler blocking; Gartner (2024) on traditional search decline. Figures are third-party and current as of mid-2026; we publish our own benchmark data as our scan volume grows.