To check if AI can read your website, run six checks in order: your robots.txt, your page source, your indexing, your structured data, whether an assistant knows who you are and whether it recommends you. The six take about 20 minutes, and the first two rule out the most common reasons an AI crawler cannot read a site at all.
A browser shows you the page after every script has run. An AI crawler often reads only the HTML your server sends, and it has to be allowed in before it reads anything. The checks below test what the crawler gets, in the order a crawler meets it.
Check 1: are AI crawlers allowed in?
- Open yourdomain.com/robots.txt in your browser.
- Look for lines that disallow OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, Google-Extended or Bingbot, either by name or under a rule for every crawler.
- If you use a firewall or bot protection, such as a CDN setting or a security plugin, check that it does not block the same crawlers. Our guide to AI crawler user agents lists who to look for.
Blocking a crawler can be a deliberate choice. It is also easy to do by accident with a file copied from another site. On Shopify, the file is generated by the platform and changed only through a robots.txt.liquid template, so a rule you did not write usually came from an app or a developer. We read our own robots.txt again on November 9, 2026. It names 15 AI crawlers, including OAI-SearchBot, ClaudeBot and PerplexityBot, and allows each of them everywhere except the admin, cart, checkout, order and account paths.
Check 2: is your content in the page source?
- Open a key page, such as your best selling product or main service page.
- Choose View page source in your browser. This shows the HTML the server sent, before scripts run.
- Search the source for a sentence you can see on the page, and for the price if it is a product.
If the sentence is missing, scripts build the content after the page loads, and a crawler that does not run scripts never sees it. Our guide to inspecting the raw HTML shows the check in each browser.
Check 3: is the page indexed?
- Copy a distinctive sentence from the page and search for it on Google in quotation marks.
- If your page does not come back, open Google Search Console, paste the page address into URL Inspection and read the status.
- Do the same in Bing Webmaster Tools with its URL Inspection tool, since Copilot answers from Bing search.
A page that neither search engine has indexed is unlikely to reach an assistant through search.
Check 4: can a machine tell what the page is?
- Paste the page address into Google's Rich Results Test or the Schema Markup Validator.
- For a product, look for one Product with a name, a price and an availability value.
- Compare those values with what the page shows a shopper.
Markup that contradicts the page can do more harm than having none, which is the subject of when schema and page disagree.
Check 5: does an assistant know who you are?
Ask ChatGPT, Gemini and Copilot to describe your business, by its name plus its category, in a Temporary Chat or a signed out session. A wrong or empty answer often means that nothing outside your site confirms who you are.
Check 6: would it recommend you?
Ask the buying question a customer would type, such as "best waterproof hiking boots for wide feet", and see whether your business is named. Ask it three times, since answers vary. This is the outcome, and checks 1 to 5 are the reasons it may not happen. Our guide to checking what AI says about your brand explains how to record the runs.
How often do real websites fail these checks?
The failures start early. In our 2026 study of 216 Shopify storefronts, 41 percent failed Front Door, the test of whether an AI crawler can read the page at all. That is checks 1 and 2 in this list.
In our July 2026 study of 303 non ecommerce websites, 97.7 percent of reachable homepages served a robots.txt. Only 28.4 percent named any AI crawler, so on most of those sites AI access was never addressed in robots.txt.
Checks 2 and 4 show up in the same study. Of the 350 sites it set out to measure, 47 produced no score, most of them because of heavy bot protection or a homepage built only in JavaScript. Among the homepages we could reach, 52.2 percent carried any schema markup in a script tag, and 1.3 percent carried FAQ schema.
What does a failed check look like on a furniture store?
Take a furniture store on Shopify that installed a bot protection app after a wave of spam orders. Check 1 shows a clean robots.txt, but the app returns a challenge page to anything that is not a browser, so OAI-SearchBot and Bingbot get the challenge instead of the sofa. Checks 2 to 6 cannot pass until the app's settings allow the verified search crawlers. The owner adds them to the app's allow list, confirms each address with the crawler's published verification method, and runs check 2 again the next day.
Where can I get the full checklist?
The six checks cover the most common failures. The full checklist holds 45 checks in nine sections, each naming the vendor page it rests on or saying plainly that it is our own practice. You can download the checklist as a PDF or as an Excel working copy. Neither one asks for an email address.
- Section A checks whether crawlers can reach the site and whether the CDN lets them in.
- Section B checks whether they can read what matters in the HTML the server sends.
- Section C covers indexing and canonical URLs in Google and in Bing.
- Section D checks that every page names the company in the same words.
- Section E checks that the structured data matches the page.
- Section F looks for pages that answer the questions buyers ask.
- Section G covers internal links.
- Section H covers the evidence outside the website, such as profiles and reviews.
- Section I sets up measurement in Search Console, GA4 and Bing Webmaster Tools.
An appendix shows where to change each item on eight platforms, including Shopify, WordPress, Wix and Next.js, and the Launch column marks the checks that matter before a new site goes live. We check every cited vendor page each quarter, and within two weeks of a change to the Google Search Central updates page or to the OpenAI, Anthropic or Perplexity crawler pages. Version 1.0 was published on October 1, 2026.
Common questions
How do I check if AI can read my website?
Open your robots.txt and confirm that AI crawlers are not blocked, then view the source of a key page and search it for a sentence you can see on screen. If both pass, confirm that the page is indexed and that its structured data matches what the page says.
Does the checklist score my website?
The checklist records a status for each check and does not score anything. The free scan reads five of your key pages the way an AI crawler does, scores them across five categories and emails you the scorecard.
How often should I run these checks?
Monthly covers most sites. Run them again after a theme change, a new app, a migration or a new security setting, since those are the changes that close the door to crawlers.
What if everything passes and I am still not recommended?
Then the gap is usually outside your site. Being readable makes a site eligible, and mentions, reviews and listings elsewhere help an assistant choose it. Our guide to your citation footprint covers that work.
Sources
- Visibility Mesh, The State of AI Visibility 2026, 216 Shopify storefronts scored June 2026.
- Visibility Mesh, The State of AI Visibility on the Non Ecommerce Web, 303 websites, July 2026.
- Our own robots.txt, read November 9, 2026.
If several checks fail at once, the Full Assessment and Roadmap reads up to 1,000 pages and puts every finding in the order to fix it. If you would rather describe the problem to a person first, use Tell us what is not working.