AI crawlers can read your site only if four gates let them through. Your robots.txt and CDN have to allow them, they have to reach the page, the content has to be in the HTML they receive, and they have to understand what they find. Each gate takes minutes to check, and a closed gate sends you no error message.
- In our June 2026 study, 41% of 216 Shopify storefronts failed Front Door, the part of our score that tests whether a crawler can retrieve and read a page.
- RFC 9309 says robots.txt rules are not a form of access authorization, so robots.txt is a policy and not a lock.
- Google renders JavaScript. OpenAI, Anthropic and Perplexity do not document whether their crawlers do, so put the facts that matter in the initial HTML.
- Each vendor documents different rules for its crawlers, so check what blocking a name does before you keep or remove it.
Why is crawler access the first thing to check?
A page a crawler cannot fetch cannot be considered for an answer. In our June 2026 study of 216 Shopify storefronts, 41% failed Front Door, the metric that tests whether an AI crawler can retrieve and read the page. It was the best performing of our five areas, and two stores in five still failed it.
What are the four gates a crawler has to pass?
A crawler meets the gates in the order a visit happens. A closed first gate ends the visit. The later gates decide how much of the site the crawler reaches and understands.
| Gate | The question | What fails it | Where to check it |
|---|---|---|---|
| 1. Access | Are you letting the crawler in? | A Disallow line in robots.txt, often added by an app or a theme, or a CDN rule that blocks the bot | yourdomain.com/robots.txt, then your CDN's bot settings |
| 2. Reach | Can it find every page? | Orphan pages, dead URLs, an outdated sitemap, contradictory canonicals | yourdomain.com/sitemap.xml and a few of its URLs |
| 3. Render | Can it read the page when it arrives? | Content that appears only after JavaScript runs | The page source, searched for a sentence you can see on screen |
| 4. Parse | Can it understand what it found? | Facts that live only in images, no structured data, ambiguous copy | Images and structured data on your key pages |
How do I check all four gates on my site?
- Open yourdomain.com/robots.txt in a browser. Search the file for GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended, and note every Disallow line under those names. On Shopify the file comes from the
robots.txt.liquidtemplate under Online Store › Themes › Edit code if your theme has one. - If you use Cloudflare or another CDN, open its bot settings and check whether AI crawlers are blocked or challenged. Is Cloudflare blocking AI from your site walks through the Cloudflare screens.
- Open yourdomain.com/sitemap.xml and open three of the product URLs it lists. Each should load the right page without a redirect.
- Open a product page, view the source with Ctrl+U on Windows or Option+Command+U on a Mac, and search for the price and the first sentence of the description. If either is missing from the source, it loads by script.
- Test the same page in Google's Rich Results Test and check that the product's structured data is detected.
OpenAI says a robots.txt change takes about 24 hours to reach its search systems, and Perplexity says up to 24 hours. Check again the next day before you conclude anything.
Is robots.txt a lock or a policy?
It is a policy. RFC 9309, the standard behind the file, says its rules are not a form of access authorization (RFC 9309, September 2022). I hold CISA and CISSP certifications, and in an audit I would file robots.txt under policy, not under controls. A crawler that follows the standard obeys it. A crawler that ignores the standard may read your pages anyway, so private material belongs behind a login.
Amazon shows what a deliberate policy looks like. On September 26, 2026, amazon.com/robots.txt disallowed GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, CCBot and Google-Extended from the whole site. That is a business decision, and I respect it. The same lines copied into a small store's file by an app or a theme are a finding to fix.
What does blocking each crawler name do?
Each purpose below is quoted or summarized from the vendor's own page, read September 26, 2026. The crawler guide covers each name in more depth, and robots.txt for the AI era shows how to write the rules.
| Name in robots.txt | Vendor | What it does | What blocking it means |
|---|---|---|---|
| OAI-SearchBot | OpenAI | Surfaces websites in ChatGPT search | Your pages are not shown in ChatGPT search answers, though they can still appear as navigational links |
| GPTBot | OpenAI | Collects content for training OpenAI's foundation models | A signal that your content should not be used for training |
| ChatGPT-User | OpenAI | Visits a page when a user asks ChatGPT to | OpenAI says robots.txt rules may not apply to these user actions |
| ClaudeBot | Anthropic | Collects content that could contribute to model training | A signal that your future materials should be left out of training |
| Claude-SearchBot | Anthropic | Indexes content to improve search results | Anthropic says it may reduce your visibility in user search results |
| Claude-User | Anthropic | Retrieves pages when a Claude user asks | Anthropic says it stops retrieval of your content in response to a user query |
| PerplexityBot | Perplexity | Surfaces and links websites in Perplexity search | Your pages leave Perplexity's search results |
| Perplexity-User | Perplexity | Fetches pages a user asked for | Perplexity says this fetcher generally ignores robots.txt rules |
| Google-Extended | A control token with no crawler of its own, used for AI training and grounding in some Google systems | Google says it does not affect inclusion or ranking in Google Search | |
| Applebot-Extended | Apple | An opt out from training Apple's generative foundation models | Apple says the pages can still be included in search results |
| CCBot | Common Crawl | Builds Common Crawl's open repository of web crawl data | Common Crawl stops adding your pages to that repository |
What happens at gates two to four?
A crawler reaches pages by following links and your sitemap. Redirect chains and contradictory canonicals make it harder to tell which URL is the real one.
Google renders JavaScript with an evergreen version of Chromium (JavaScript SEO basics). OpenAI, Anthropic and Perplexity say nothing in their crawler documentation about whether their crawlers do. So I treat anything that appears only after a script runs as content those crawlers may never see.
Gate four is about understanding. A fact that lives only in an image with no alt text is not in the text a crawler reads.
You may also come across llms.txt. Google says Google Search does not use it and that keeping one for other services is fine (AI optimization guide).
How does an app end up blocking ChatGPT search?
Consider a home fragrance brand on Shopify with 90 products. Two years ago it installed an app that promised to protect its product photos. The app added a Disallow line for GPTBot and CCBot, and a later update added OAI-SearchBot as well. Nobody on the team ever opened robots.txt. The store looks fine in a browser and cannot appear in ChatGPT search answers, since OpenAI says OAI-SearchBot is the crawler behind them.
Training is a business choice, so the brand can keep GPTBot and CCBot blocked if it prefers. Search is a different choice, so it removes the OAI-SearchBot line, waits a day and checks again.
What does our scan check, and what does it not?
Since September 26, 2026, our scan has checked eight crawler names one by one against the robots.txt rule for each page it reads. Three are from OpenAI (GPTBot, OAI-SearchBot and ChatGPT-User), three from Anthropic (ClaudeBot, Claude-SearchBot and Claude-User) and two from Perplexity (PerplexityBot and Perplexity-User). It reports Google-Extended on its own as a control for Gemini training and grounding, and never counts it as a block. The scan does not test Bingbot or Applebot. A firewall can also answer our request differently from the way it answers the real crawler, so check your CDN logs too.
A theme or app update can rewrite robots.txt with nobody on your team touching it. AI visibility drift explains why we scan again after every theme or app change. A free scan checks the eight crawler names against your robots.txt and reads the structured data and text of five of your pages.
Questions people ask
How do I know if AI crawlers can access my store?
Open yourdomain.com/robots.txt and look for rules that name AI crawlers. Check that your key content appears in the page source, not only after JavaScript runs, and check your CDN or firewall for bot rules. A free scan checks eight crawlers from OpenAI, Anthropic and Perplexity, but not Bingbot or Applebot.
Do AI crawlers read JavaScript?
Google does. Its documentation says Google Search runs JavaScript with an evergreen version of Chromium. OpenAI, Anthropic and Perplexity do not document whether their crawlers run JavaScript, so put prices, product names and descriptions in the initial HTML.
Can a bot fake the GPTBot or Googlebot user agent?
A user agent string is text any client can send, and Google and Common Crawl both warn about spoofed crawler names. Verify the source instead. Google and Apple support a reverse DNS lookup, Microsoft offers a Verify Bingbot tool, and OpenAI, Anthropic, Perplexity, Google, Apple and Common Crawl publish their IP ranges.
Will blocking AI crawlers stop my content from being used to train AI models?
It works only for crawlers that honor the rule, and only for content collected after the rule is in place. OpenAI, Anthropic, Google, Apple and Common Crawl each document a robots.txt name for this: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended and CCBot. None of those pages says the rule removes content collected earlier.
Can crawlers read content behind a login or a consent wall?
They cannot read content that requires signing in, and Anthropic says its bots will not try to bypass CAPTCHAs. A consent banner that only overlays the page leaves the HTML readable. A wall that withholds content until someone clicks withholds it from crawlers too.
How do I stop Microsoft Copilot from using my content without leaving Bing search?
Microsoft documented two robots meta values on September 22, 2023, in a post about Bing Chat. NOARCHIVE keeps a page out of chat answers and out of training for Microsoft's generative AI models. NOCACHE lets answers use only the URL, title and snippet. Microsoft says the page stays in Bing search either way.
Sources: OpenAI, Overview of OpenAI Crawlers; Anthropic, Does Anthropic crawl data from the web, and how can site owners block the crawler?; Perplexity, Perplexity Crawlers; Google, Google common crawlers, Verify requests from Google crawlers and fetchers, Understand the JavaScript SEO basics and the AI optimization guide; Apple, About Applebot; Common Crawl, CCBot; IETF, RFC 9309, September 2022; Microsoft, Announcing new options for webmasters to control usage of their content in Bing Chat, September 22, 2023; Amazon, robots.txt, read with curl on September 26, 2026 at 13:58 UTC. Every vendor source was read on September 26, 2026. Visibility Mesh, The State of AI Visibility 2026, 216 Shopify storefronts scanned June 28, 2026. The home fragrance brand is an illustration, not a client.