Cloudflare added controls that block AI crawlers, and since September 15, 2026 new domains block training and agent bots by default on pages that show ads, while search crawlers stay allowed. If your Cloudflare settings also block the search crawlers, assistants cannot fetch your pages for a reason that has nothing to do with your content.
- Bot management can block assistant crawlers without you ever choosing to.
- This is invisible from inside your site. Pages look fine to you and are unreachable to them.
- Blocking is a legitimate choice. Blocking by accident is not.
Why would Cloudflare block AI crawlers?
Because a lot of site owners want it. Training crawlers consume bandwidth and take content, and a control that stops them is a reasonable product. The complication is that the same control can also stop the retrieval crawlers that would have recommended you.
Those are different things. One reads your site to train a model. The other reads it to answer a buyer question right now, and being invisible to the second is a direct cost.
How do I tell whether mine is blocking them?
Three checks, in order of effort.
- In the Cloudflare dashboard, look at your bot and crawler controls and see whether AI crawler blocking is enabled.
- Check your firewall events for blocked requests from assistant user agents.
- Run an external scan that fetches your pages the way an assistant would. If it cannot reach pages a browser can, something in front of the site is the cause.
Which crawlers should I allow?
The ones that send buyers. Retrieval crawlers that fetch a page in order to answer a live question are the ones that produce recommendations, and blocking them is blocking demand.
Training crawlers are a business decision rather than a technical one, and reasonable people choose differently. The user agents lists who is who so the choice is deliberate.
What else sits in front of a site and does this?
Any WAF or CDN with bot management, aggressive rate limiting, or a country block that happens to cover where a crawler runs from. Password protected staging left switched on catches people too.
The symptom is always the same and always confusing: the site works perfectly for you, and does not exist for them.
Common questions
Should I block AI crawlers?
Only deliberately. If your business depends on being recommended, blocking retrieval crawlers removes you from consideration. Blocking training crawlers is a separate and more defensible choice.
Will blocking hurt my Google ranking?
Blocking Google’s crawler will. Blocking assistant crawlers does not affect classic ranking, and the effect on AI answers depends on the company: OpenAI and Anthropic say blocking their search crawlers reduces or removes your visibility in their search results.
How would I know this happened?
You would not, from inside. Nothing changes on your site. It has to be checked from outside, which is what an external scan does.
Related reading
- The Front Door Test: Can AI Crawlers Even Get Into Your Store?
- robots.txt for the AI Era: Who You Are Blocking Without Knowing It
Run a free scan and get a 0 to 100 score with a fix list in priority order. Run the free scan.
Your buyers are already asking AI. This is how you make your website readable to the assistants they ask.
Everything in this article is measurable on a live storefront. The Full Assessment and Roadmap reads your website the way AI crawlers receive it and hands you every fix in plain English, in the order we would make them.