How do I test whether crawlers see the same content as visitors?

Compare three copies of the same page: the raw HTML your server sends, the rendered page after scripts run, and the response your site gives to a crawler's user agent. Every fact a visitor can see should appear in all three. A fact that exists only after rendering, or a different answer for a crawler, goes on the repair list.

By Margareta Petrovic, founder of Visibility Mesh. We checked every statement in this guide against its primary source on September 30, 2026.

A visitor and a crawler can receive different pages from the same address, and the owner usually sees only the visitor's. So test the page as each of them.

Before you start

  • Save the raw HTML of the page first.
  • Open a browser with developer tools.
  • Get Search Console access for the live test in step 4.
  • Write the fact list. It holds the title, the main heading, the main text, prices and availability, contact details, internal links, the canonical link and the structured data.

Steps

Step 1. Copy the rendered page after it settles

Load the page in a private window with no cookies. Consent walls, logins and location redirects each change what you get. Note what the page shows before any click, and leave the consent banner alone. If the site redirects by market, note where the request ended up.

Then wait until the page stops changing. Open every tab and accordion, and scroll to the bottom so lazy loaded sections appear. In Chrome's developer tools, open the Elements panel, right click the html element and choose Copy, then Copy outerHTML. Save it next to the raw file as rendered.html.

Step 2. Search both copies for every fact

Search the raw file and the rendered file for each fact on the list. Mark each fact as present in both, present only after rendering, or missing. Keep the three columns in one table per page. That table becomes the repair list in step 6.

Step 3. Find content that appears only after a click

Tabs, accordions and load more buttons hide content in one of two ways. Some keep the text in the raw HTML and only hide it with styling. A crawler that reads HTML still receives that text. Others fetch the text from the server when someone clicks. Crawlers do not click, so that text never reaches them. No load more button has ever persuaded one. Search the raw file for a sentence from each tab to see which kind you have.

Google's lazy loading guide says content should load when it becomes visible in the viewport, without relying on a scroll or a click. Google Search does not interact with the page.

Step 4. Run the URL Inspection live test

In Search Console, inspect the URL and choose Test live URL. Open View tested page. It shows the rendered HTML, a screenshot and more information about the resources the page loaded. Compare Google's rendered HTML with your fact list. For Google, this is the copy to trust. When a fact is in your rendered copy but missing from Google's, it often loads too late or depends on a resource Google could not fetch. The resource list in the live test shows what Google could not load.

Step 5. Request the page with a crawler's user agent

Fetch the page twice with curl. Send a browser user agent the first time and the crawler's full user agent string the second time. Each operator publishes its strings, and OpenAI's example for GPTBot is below. Compare the status codes and search both files for your facts.

curl -sL -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64)" -o browser.html -w "%{http_code}\n" https://www.example.com/page
curl -sL -A "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot" -o crawler.html -w "%{http_code}\n" https://www.example.com/page

A firewall can allow verified crawlers by their network addresses and challenge everyone else. Perplexity's crawler page gives the allow list steps for Cloudflare and AWS WAF. So a user agent test from your laptop shows only how your server treats a request that claims to be a crawler.

Step 6. List what needs to move into the first response

Every fact marked present only after rendering is a candidate for server side output. So is every fact behind a click. Hand the list to whoever builds the templates. Give each fact one line that names the page, the fact and the copy where you found it.

Check that it worked

After the fix, repeat steps 2, 4 and 5 on the same page. The raw HTML should hold every fact on the list, and Google's rendered HTML should match it. The response to the crawler user agent should carry the same status and the same facts as the browser response.

If it did not work

  • If the crawler user agent gets a challenge page, your firewall answered your test. That does not prove the real crawler is blocked. Check your server or CDN logs for requests from the crawler's verified addresses. The log guide shows how to see what verified crawlers received. Google documents a reverse and forward DNS check for Googlebot, and OpenAI and Perplexity publish the IP address lists of their crawlers.
  • If the crawler user agent gets a normal page, the real crawler may still be treated differently. A firewall rule can match network addresses as well as the user agent.
  • If the copies differ between two runs, a cache may be serving an older version. Add a query string such as ?cb=1 to the address and fetch again.
  • If your rendered copy has content that the raw copy lacks and you were logged in, you compared a logged in page with an anonymous one. Start again in a private window.
  • If Google's rendered HTML matches your rendered copy, check the raw copy as well, because a crawler that does not run scripts receives only that.
  • If someone suggests serving crawlers a special version of the page, be careful. Google's spam policies call it cloaking when a site shows search engines different content from what users see, to manipulate rankings and mislead users.

Platform notes

On Shopify, test each app block on its own. Review widgets and size guides often come from apps, and when an app loads its content with JavaScript, that content is missing from the raw HTML.

On WordPress, page builders add tabs and accordions. Check whether each one keeps its text in the raw HTML or loads it on a click.

On a single page application, most facts arrive only after rendering. Google describes dynamic rendering as a workaround and not a recommended solution, so plan for server side rendering or static rendering instead.

What Visibility Mesh checks and what it does not

The Visibility Mesh scan scores each page after rendering it in a browser engine. Put that rendered view next to your saved raw file to see the gap. On Shopify and WordPress, AI Visibility Setup: Foundation fixes crawler access and rendering at the template level. We then check that a crawler receives the same content a visitor sees (how the setup works).

Questions people ask

Is showing crawlers different content cloaking?

Google's spam policies define cloaking as showing different content to users and to search engines, with the intent to manipulate rankings and mislead users. Google's dynamic rendering page adds that a pre rendered version for crawlers is generally not cloaking when the content is similar. Serving completely different content can be. The safe course is to give every crawler the same content a visitor sees.

Should I use dynamic rendering?

Google calls dynamic rendering a workaround and not a recommended solution, since it adds complexity and needs resources. Google recommends server side rendering, static rendering or hydration instead. Each of those gives visitors and crawlers the same HTML.

Does lazy loading hide content from crawlers?

It can. Google's guide says content should load when it becomes visible in the viewport, without relying on a scroll or a click. Content that waits for an interaction may never load for a crawler. Check each lazy loaded section in the URL Inspection live test.

Does Google's rendering prove that AI crawlers see the same thing?

It does not. Google documents that it renders JavaScript. OpenAI, Anthropic and Perplexity do not say on their crawler pages whether their crawlers do. A fact that appears only after rendering may reach Google and still be missing for them.

Is text inside a tab or an accordion read by crawlers?

If the text is in the raw HTML and only hidden with styling, a crawler that reads HTML receives it. If the text loads from the server when someone clicks, a crawler does not receive it, since crawlers do not click. Search the raw HTML for a sentence from each tab to find out which kind you have.

Sources

Before you test the content, check what the crawler may fetch. For the reasons behind the test, read why rendering decides what AI sees.

Run the free scan to see what a machine reads on five key pages of your site.
See whether this applies to your site

This article is about whether AI can find you; the first category of the scan checks exactly that on your site.

The free Visibility Mesh scan checks whether AI crawlers can reach your website and reads the structured data and text of 5 key pages. The scorecard has 3 complete findings, and each one names the page and the fix. No install, no call, no card.

Run the free scan