Is my WordPress security plugin blocking AI crawlers?

It can, and your robots.txt will not tell you. A security plugin, a host firewall or a CDN can turn away OAI-SearchBot or PerplexityBot while robots.txt invites them in. The reliable check is your server log or the plugin's own record of blocked requests. Allow the search crawlers you want by the IP addresses their companies publish, not by the name in the request. Whether to let training crawlers in is a separate decision.

By Margareta Petrovic, CISA, CISSP, founder of Visibility Mesh. As of October 4, 2026, our engine has scored 957 websites.

In a security review, a written rule is not accepted until someone has tested it. A WordPress site has two sets of rules for crawlers. Robots.txt is only a request. The security plugin, the host firewall and the CDN are what let a crawler in or keep it out.

Which AI crawlers should I allow?

The companies behind the assistants run separate crawlers for search and for training. The search crawlers put your pages into answers. The training crawlers collect content for future models. OpenAI, Anthropic and Google let you treat the two separately, so you can allow search and refuse training if that is your policy.

Google's own search crawling runs on Googlebot, and Google says Google-Extended does not affect inclusion or ranking in Google Search. So a Google-Extended rule is a training decision, and AI Overviews still depend on Googlebot.

How does a security plugin end up blocking them?

Security plugins protect a site in different ways, and only some of them touch crawlers. WP Armour, a honeypot plugin, blocks bots when they try to submit a form. Its author wrote on the WordPress.org support forum that the plugin does not block any bots, AI included, from crawling the site.

Bot blocker and firewall plugins are a different matter. A user of BotBlocker Security asked on its support forum for GPT, Perplexity and Claude bots to be allowed, since the SEO bots already were. The developer later wrote that the plugin had added detection and whitelist support for GPTBot, ClaudeBot and other AI crawlers. If your bot blocker has rules like these, open them and see which AI crawlers are on the allowed side.

Rate limits are the hardest blocks to spot. Wordfence has rate limiting settings under Firewall, then Rate Limiting, including one called "How should we treat Google's crawlers". A crawler that reads many pages quickly can go over those limits, and a Wordfence support reply on WordPress.org says a throttled crawler receives a 503 response.

A crawler that reads hundreds of pages in a few minutes and never buys anything looks like trouble to a security plugin, and to be fair, most bots that behave that way are.

The block can also sit outside WordPress. Cloudflare has a setting called Block AI bots, which its documentation says blocks verified bots that crawl for AI training and leaves search crawlers alone. The same page, last updated July 1, 2026, says Cloudflare planned to set updated defaults for new domains on September 15, 2026. Under those defaults, bots classified as training or agent crawlers are blocked on pages that display ads, and search crawlers stay allowed. We cannot see from outside which settings your zone received, so read the current AI crawler settings in your own Cloudflare dashboard instead of relying on an older guide. Your host may run a firewall of its own as well.

How do I check mine?

Start with robots.txt at yoursite.com/robots.txt. WordPress serves a virtual file by default that blocks the admin folder and allows admin-ajax.php, and SEO plugins can change it. Look for the crawlers in the table by name, and read the group marked with an asterisk, which covers every crawler that has no group of its own.

One WordPress setting trips people up. If "Discourage search engines from indexing this site" is checked under Settings, then Reading, WordPress 5.3 and later do not write a block into robots.txt. WordPress adds a robots meta tag to the pages instead, so the file most people check looks clean.

Your security plugin is the next place to look. Open it and find its list of blocked or throttled requests. Search it for OAI-SearchBot, PerplexityBot and Claude-SearchBot. If your host gives you access logs, search those too and look at the status code next to each request. A 200 means the crawler received the page. A 403, 429 or 503 means something turned it away.

A name in the log does not prove the visit was real, since anyone can put OAI-SearchBot in a request. OpenAI publishes the addresses its search crawler uses at openai.com/searchbot.json, and Perplexity publishes its list at perplexity.com/perplexitybot.json. Compare the address in your log with those lists before you trust it.

How do I allow the right ones safely?

Allow crawlers by their published IP addresses rather than by the name in the request. Perplexity's documentation recommends matching the user agent together with the published IP addresses, and refreshing the firewall rules automatically as the lists change. A rule that trusts the name alone also lets in every scraper that borrows it.

Keep the security plugin switched on. Add an exception for the search crawlers you want and leave the rest of its protection as it is. If you switch the plugin off to let one crawler in, you also let in everything else it was stopping.

Decide on training separately, in robots.txt. You can allow OAI-SearchBot and disallow GPTBot, or allow both. Either choice is legitimate, and OpenAI says the two settings are independent.

Write down what you changed and when. OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change. Check your log again the next day for requests that received a 200.

What it costs, and when to hire help

If you leave the block in place, OpenAI says your site will not be shown in ChatGPT search answers. A careless fix has its own cost, since it lets in anything that calls itself a crawler.

An owner with one security plugin and access to its settings can do this alone. Hire help when the site sits behind several layers at once, such as a plugin, a host firewall and Cloudflare. Hire help too when you cannot get at the logs, or when a wrong rule on a WooCommerce store would cost you orders.

Questions owners ask

Does Wordfence block ChatGPT?

We found no Wordfence rule that targets ChatGPT by name in the sources we read for this article. Its rate limiting can throttle any crawler that goes over the limits you set, and a Wordfence support reply says a throttled crawler receives a 503 response. Check the settings under Firewall, then Rate Limiting, and look in your log for OAI-SearchBot requests that did not receive a normal page.

Why is my site still blocked when my robots.txt allows AI crawlers?

Robots.txt is a request, and RFC 9309, the standard behind it, says its rules are not a form of access authorization. A security plugin, a host firewall or a CDN can still turn the crawler away. Check the plugin's blocked requests and your access log for the status code each crawler received.

Should I block GPTBot on my WordPress site?

That is a business decision about training, not about search. OpenAI says GPTBot collects content that may be used to train its models, and that the GPTBot setting is independent of OAI-SearchBot, which decides ChatGPT search results. You can block GPTBot and still allow OAI-SearchBot.

Does blocking Google-Extended remove my site from AI Overviews?

Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. It controls whether content may be used to train Gemini models. A page needs to be indexed and eligible for a snippet to appear as a link in AI Overviews, and that depends on Googlebot.

Is it safe to allow AI bots through my security plugin?

It is safe when you allow them by the IP addresses their companies publish and keep the rest of the plugin's protection on. A rule that trusts the user agent alone lets in anyone who copies the name. Perplexity's documentation recommends checking the user agent and the published addresses together.

How soon will ChatGPT see my site after I unblock it?

OpenAI says its search systems can take about 24 hours to adjust after a robots.txt change. It does not publish how often its crawler returns to a page. Check your log the next day for OAI-SearchBot requests from OpenAI's published addresses that received a 200.

Sources

We opened every source below on October 5, 2026.

• Overview of OpenAI crawlers, OpenAI developer documentation.
• Perplexity crawlers, Perplexity documentation.
• Does Anthropic crawl data from the web, and how can site owners block the crawler?, Anthropic Help Center.
• Google's common crawlers, Google Search Central, last updated July 14, 2026.
• AI features and your website, Google Search Central, last updated December 10, 2025.
• Block AI bots, Cloudflare documentation, last updated July 1, 2026.
• do_robots(), WordPress developer reference.
• Question about AI user agent blocking, WP Armour support forum, WordPress.org.
• Unblock good AI bots, BotBlocker Security support forum, WordPress.org.
• Rate limiting Googlebot?, Wordfence support forum, WordPress.org, reply of September 17, 2025.
• RFC 9309, Robots Exclusion Protocol, IETF, September 2022.
• Press and benchmark, Visibility Mesh, figures measured October 4, 2026.