Cloudflare can block AI crawlers from your site, and the setting that decides it is called AI Crawler and Agent access policies. On July 1, 2026, Cloudflare's changelog announced a new default. From September 15, 2026, new domains would block Training and Agent bots on pages that display ads. Cloudflare's documentation was last updated on November 2, 2026, and it says new domains start with Search, Agents and Training all set to Allow. The two pages describe different defaults, so read the setting on your own domain before you trust either. If your settings block the crawlers that answer a buyer's question, assistants cannot read your site. The reason has nothing to do with your content.
You will not notice it from inside. The site loads normally in your browser, your analytics look the same, and the crawler simply gets turned away at the door.
Does this apply to a Shopify store?
Usually not in the same way. Shopify says it handles bot management for its stores at the network layer. A Shopify owner has no Cloudflare dashboard to change, and the one crawler setting left to the owner is robots.txt. This guide is for sites that sit behind their own Cloudflare account, such as a WordPress site or a custom storefront. A Shopify owner should read how to check what your robots.txt blocks instead.
Why would Cloudflare block AI crawlers?
Many site owners asked for it. Training crawlers use bandwidth and collect content for model training, and a switch that stops them is a reasonable product. The complication is that the same switch can also catch crawlers you want. Cloudflare's own changelog says crawlers that combine Search and Training are affected by settings that block Training.
Two kinds of crawler matter here. A training crawler reads your site to train a model. A search or retrieval crawler reads it to answer a buyer's question now. Our list of AI crawler user agents shows which is which for each company.
How do you check whether Cloudflare is blocking them?
- Sign in to the Cloudflare dashboard and choose the domain.
- Open Security > Settings, filter by Bot traffic, and select AI Crawler and Agent access policies. Cloudflare renamed this setting from Configure AI bot policies on November 4, 2026.
- Read the action set for Search, Agents and Training. Note whether any of them says Block, or Block on pages with ads.
- Open the security events log for the domain and filter for requests from AI user agents to see whether any were blocked.
- Fetch one of your pages from outside, the way a crawler would. Our guide to finding AI crawlers in your server logs shows what a successful visit looks like.
Which crawlers should you allow?
Allow the ones that send buyers. Search and retrieval crawlers fetch a page to answer a live question, and blocking them takes your pages out of those answers. OpenAI, for example, says a site that opts out of OAI-SearchBot will not be shown in ChatGPT search answers, apart from navigational links.
Whether to allow training crawlers is a business decision, and reasonable owners choose differently. Whatever you choose, set it on purpose in the policy screen above, so the setting is one you chose.
What does this look like on a real kind of site?
Take a recipe blog on WordPress that earns money from display ads and moved its domain to Cloudflare in October 2026. The owner wants search answers to link to the recipes but does not want them used for training anywhere on the site. In the AI Crawler and Agent access policies, Training is set to Block on pages with ads, which protects only part of the site. The owner changes Training to Block for the whole site. Search stays on Allow, and Agents get a separate decision. The whole review takes about ten minutes.
What else in front of a site can block AI crawlers?
Any firewall or CDN with bot management can do it, as can strict rate limiting or a country block that happens to cover where a crawler runs from. On WordPress, a security plugin is the usual suspect, covered in is my WordPress security plugin blocking AI crawlers. A password protected staging site that was never switched off catches people too. The Front Door test covers every one of these checks in order.
If a firewall, a plugin or robots.txt is turning crawlers away, AI Visibility Setup: Foundation opens the site to the crawlers you choose. It works on Shopify and WordPress, and you approve each change.
Questions owners ask
Does Cloudflare block AI crawlers by default?
Cloudflare's own pages describe two different defaults. On July 1, 2026, its changelog said that from September 15, 2026, new domains would block Training and Agent bots on pages with ads. Its documentation, updated on November 2, 2026, says new domains start with all three set to Allow. Open Security > Settings > AI Crawler and Agent access policies on your own domain and read what it shows.
Should I block AI crawlers?
Block them only on purpose. If your business depends on being recommended, blocking search and retrieval crawlers removes you from those answers, while blocking training crawlers is a separate choice.
Will blocking AI crawlers hurt my Google ranking?
Blocking Googlebot will hurt your Google ranking. Blocking OAI-SearchBot or PerplexityBot does not change your Google ranking. OpenAI does say that a site blocking OAI-SearchBot will not be shown in ChatGPT search answers, apart from navigational links.
How would I know this happened?
You would not know from inside your site, since nothing changes for you. It has to be checked in the Cloudflare policy screen and the security events log, or by fetching your pages from outside.
Sources
- Bots changelog, Cloudflare, entry of July 1, 2026 on the September 15, 2026 defaults and entry of November 4, 2026 on the renamed setting, read November 9, 2026.
- AI Crawler and Agent access policies, Cloudflare documentation, last updated November 2, 2026, read November 9, 2026.
- Overview of OpenAI crawlers, OpenAI.