robots.txt for the AI Era: The Crawlers You Block Without Knowing It

robots.txt for the AI Era: Who You're Blocking Without Knowing It

robots.txt is the public file at yourstore.com/robots.txt that tells each crawler which parts of a store it may read. One rule under the wrong name can turn away an AI crawler you want, since OpenAI says sites that block OAI-SearchBot will not be shown in ChatGPT search answers.

By Margareta Petrovic, CISA, CISSP, founder of Visibility Mesh. Updated November 9, 2026.

The same file speaks to Google, OpenAI, Perplexity, Anthropic and many others, each with crawlers that do different jobs, and many files were written before those names existed.

What does each line in robots.txt mean?

The file is a list of groups. Each group starts with a User-agent line that names a crawler, or an asterisk for every crawler, and then lists the paths that crawler may not fetch. Disallow: / under a name closes the whole store to that crawler. Disallow: /checkout keeps one private path out and leaves everything else open. A crawler follows the group that names it most specifically, so a named group overrides the asterisk group for that crawler.

robots.txt line What it means Effect on AI
User-agent: GPTBot / Disallow: / Keep OpenAI’s training crawler out It tells OpenAI your content should not be used in training. The separate OAI-SearchBot line decides whether ChatGPT search can show your pages.
User-agent: * / Disallow: /checkout Block a path for all crawlers Fine. Keeps private paths out
User-agent: * / Disallow: / Block everything Catastrophic. Nobody can read you
Allow: / (default) Let well-behaved crawlers in Reachable, the readable layer still has to deliver
The difference between appearing in AI shopping and being invisible can be one line.

Which AI crawlers do different jobs?

A rule only blocks the job of the crawler it names, so read the name before you judge the rule.

  • OAI-SearchBot finds pages for ChatGPT search. OpenAI says blocked sites will not be shown in ChatGPT search answers, although they can still appear as navigational links.
  • GPTBot collects content for training. OpenAI says disallowing it means a site's content should not be used to train its foundation models.
  • ChatGPT-User fetches a page when a person asks ChatGPT to, and OpenAI says robots.txt rules may not apply to it.
  • PerplexityBot surfaces and links websites in Perplexity's search results. Perplexity says it is not used for training and asks sites to allow it.
  • Perplexity-User fetches a page because a person asked, and Perplexity says it generally ignores robots.txt rules.

The longer list of names, including Anthropic's crawlers, is in the AI crawler user agents guide.

How do I read my store's robots.txt?

  1. Open yourstore.com/robots.txt in any browser. The file is public on every store, so no login is needed.
  2. Search the page for each crawler name you care about: OAI-SearchBot, GPTBot, PerplexityBot, ClaudeBot, Google-Extended and AdsBot-Google.
  3. Under each name you find, read the Disallow lines. Disallow: / closes the whole store to that crawler.
  4. Read the asterisk group last. Its rules apply to every crawler that has no group of its own.
  5. Write down each rule that blocks a crawler you want, and find out where it came from: your theme, an app, or someone on your team.

The knowledge base guide to checking robots.txt has screenshots of each step.

How do I change robots.txt on Shopify?

Shopify's help page says the default robots.txt works for most stores, and that editing it is an unsupported customization that can lose you traffic if done wrong. If you do need a change, Shopify gives these steps.

  1. In your Shopify admin, go to Online Store > Themes.
  2. Next to the live theme, click the three dots and choose Edit code.
  3. Click Add a new template, select robots, and click Create template.
  4. Add your rule to the robots.txt.liquid template, keeping Shopify's default rules in place, and save.
  5. Open yourstore.com/robots.txt again and confirm the change. OpenAI says its systems take about 24 hours to adjust after an update.

If your theme already has an old robots.txt.liquid template that prints outdated rules, Shopify says that deleting the template restores the current default.

How does a kitchenware store undo a robots.txt rule an old app added?

A kitchenware store on Shopify installed a content protection app two years ago. The app added a robots.txt.liquid template with User-agent: OAI-SearchBot followed by Disallow: /, next to a GPTBot group with the same rule. The owner wanted to keep the store out of training data, which the GPTBot rule does. The OAI-SearchBot rule went further than anyone intended and kept the store's pages out of ChatGPT search answers.

The owner deletes the OAI-SearchBot group, keeps the GPTBot group, and saves. The app had been guarding the front door against the one visitor who came to recommend the store. OpenAI says to allow about a day for the change to register.

What should I check after robots.txt?

Once the file lets crawlers in, check your sitemap next, then whether AI crawlers can access your store past any firewall. A launching store should read the new store launch checklist, and a slow store should read page speed and AI crawlers. The free scan reads your robots.txt the way each crawler does and tells you which rules block which names.

Questions owners ask

Where is my robots.txt file?

Your robots.txt file is at yourstore.com/robots.txt, and it is public on every store. On Shopify you change it through the robots.txt.liquid template in Online Store > Themes > Edit code.

Should robots.txt block AI crawlers?

Most stores that want to appear in AI answers should allow the search crawlers, such as OAI-SearchBot and PerplexityBot. Blocking a training crawler such as GPTBot is a separate choice that does not decide whether ChatGPT search shows your pages.

Can one robots.txt mistake make my store invisible to an AI assistant?

A rule that disallows the whole store under one crawler's name keeps that crawler out. A rule against OAI-SearchBot keeps your pages out of ChatGPT search answers, while a rule against GPTBot only tells OpenAI not to train on them.

How long does a robots.txt change take to work?

Shopify says the file changes as soon as you save it. OpenAI says its systems take about 24 hours to adjust, and Perplexity says to allow up to 24 hours.

Sources

We opened every source below on November 9, 2026.
See whether this applies to your site

This article is about whether AI can find you; the first category of the scan checks exactly that on your site.

The free Visibility Mesh scan checks whether AI crawlers can reach your website and reads the structured data and text of 5 key pages. The scorecard has 3 complete findings, and each one names the page and the fix. No install, no call, no card.

Run the free scan