WordPress AI bot audit: find what blocks AI search

What is a WordPress AI bot audit?

A WordPress AI bot audit is a check of the rules and technical conditions that let AI services retrieve your website. It answers a practical question: if the right crawler arrives at a public page, can it get useful content?

That is different from asking whether your business appears in an AI answer. Access is one prerequisite. Relevance, useful information and the engine's choice of sources still matter. Separating those questions stops you rewriting a good article when the actual problem is a firewall challenge.

A useful audit follows the request from the outside in: domain and HTTPS, CDN or host, web application firewall, robots.txt, then the page itself. On WordPress, a security plugin, a physical robots.txt file and an SEO plugin can all affect different parts of that journey.

If several people manage the site, record who controls each layer. The person who edits a post may have no access to the hosting rule refusing the crawler.

Which AI bots should I allow on my WordPress site?

Allow the search crawlers you want to discover and cite your public content, then make a separate decision about training crawlers and user-requested fetchers. Blocking everything with an AI-related name can remove access you intended to keep.

Scroll the table to see every column. You can also focus it and use the arrow keys.

These are provider-documented roles, not guesses based on a bot's name. OpenAI documents independent search and training controls. Anthropic describes its training, search and user agents, and Perplexity distinguishes search crawling from user fetches.

Google's AI Overviews and AI Mode use the existing Search foundations. Blocking a training-related token is not the same as blocking Googlebot. Google's own guidance for AI features says a supporting page must be indexed and eligible to appear with a snippet.

How do I audit AI crawler access on WordPress?

Start with a public page, inspect each access layer, make one targeted change, and verify the response again. Work through the seven steps below before adding more plugins or changing your content strategy.

1. Choose the pages and the access policy

Pick your homepage, a service or product page, and an informative article. Use their final canonical URLs, including the correct HTTPS host. Keep account pages, checkout details, customer records and private documents outside this exercise.

Write down the intended policy in one sentence. For example: "Allow AI search to read our public guides, refuse training crawlers, and keep private areas authenticated." That makes later decisions consistent instead of a collection of copied block lists.

Keep a copy of the current configuration before changing it. If you also need to change a WordPress plugin or theme, take a WordPress backup first so you have a way back.

2. Read the live robots.txt, not just an editor preview

Open https://example.com/robots.txt, replacing the domain with your own. Look for the named crawler you care about, broad disallow rules, and a sitemap location. Save what the public URL actually returns.

A rule in an SEO plugin's editor may not be the file your host serves. WordPress can generate robots.txt dynamically, but a physical file or an upstream rule can change the result. Check the final public response after every edit, including after a cache purge.

A specific user-agent group can be selected instead of the general group. Do not assume that adding Allow: / in one place cancels every restriction somewhere else. Review the whole file with the intended crawler in mind.

3. Check the host, CDN and security firewall

If robots.txt permits access but the page returns a block, look upstream. Your hosting dashboard, CDN bot policy or security plugin may refuse automated requests, require a JavaScript challenge, or apply a rate limit.

Ask the host a specific question: "What response did this crawler receive for this URL at this time, and which rule produced it?" Include the timestamp, URL and any request identifier you have. That is more useful than asking whether the server allows AI.

Cloudflare's AI crawler and agent access policies separate Search, Agent and Training behavior. Other platforms organize their controls differently. Read the current policy rather than treating all bot-protection switches as equivalent.

Keep security protection in place. If a legitimate crawler is blocked, create a narrow exception using the provider's published verification method, such as its current IP ranges where available. A user-agent name can be copied by anybody.

4. Inspect the page response and the content behind it

A quick diagnostic can compare an ordinary request with one using a crawler-style user agent. For example, these commands request headers and a response body from a public page:

curl -I -L https://example.com/your-public-guide/
curl -L -A "OAI-SearchBot" -D response-headers.txt \
  -o response-body.html https://example.com/your-public-guide/

This is a simulation, not proof of a real OpenAI crawl. It comes from your IP address and does not reproduce a verified crawler's network, cookies or behavior. A bot-aware firewall may treat the two differently.

Read the saved body as well as the status. A page returning 200 may still contain a challenge, a login screen or an empty application shell. Your article's main answer should be present in the response, with a useful heading and links that resolve.

For recurring or disputed failures, use real access logs and firewall events to confirm the request source and response. If you are investigating availability more broadly, our WordPress site-down troubleshooting guide helps separate an outage from a visitor-specific block.

5. Check WordPress indexing and page-level signals

In WordPress, review Settings > Reading > Search engine visibility. A production site intended for discovery should not accidentally discourage search engines. Then inspect the page's robots meta tag, response headers, canonical URL and internal links.

For Google AI features, confirm the actual URL's indexing status in Google Search Console. Do not assume a successful browser visit means Google has indexed that page. A canonical pointing elsewhere or an unintended noindex needs a different fix from a firewall block.

WPCentrify's WordPress SEO audit is useful here: it checks search accessibility and page-level problems across the sites you manage. Use the finding to identify the affected page rather than applying a broad change to the whole website.

6. Change one cause, then verify the public result

Remove the specific unintended restriction. Purge the relevant cache if necessary, fetch the final URL again, and inspect the body and logs. Record what changed and who made the change.

Do not turn off the entire firewall to fix one crawler, and do not open private content to make an audit look successful. The goal is the access policy you chose in step one.

Allow time for a provider to revisit or process a robots.txt change. OpenAI says its search systems can take about 24 hours to adjust. That is a documented processing note, not a deadline for earning citations or receiving traffic.

7. Measure AI citations and visits as separate outcomes

Once access is working, check whether your useful pages are being cited for relevant questions. Record the question, market, engine, date and linked source. Keep the question set consistent when comparing later checks.

WPCentrify's AI search visibility tracker measures linked citations in Google AI Overviews and ChatGPT. Its ChatGPT coverage is for the United States in English. It does not count every assistant or unlinked brand mentions.

Then use traffic analytics to see whether AI referrals bring visitors. A citation, a visit and a customer enquiry are different results. Tracking them separately gives you a clearer reason to improve a page.

Why can an AI bot not read my WordPress site?

The response usually points to the next check, but a status code alone does not identify the cause. Match it with the response body and the host or firewall log.

Scroll the table to see every column. You can also focus it and use the arrow keys.

Can I block AI training bots without blocking AI search?

Yes. Providers such as OpenAI and Anthropic document different agents for training and search. The following small example shows the distinction. It is an example to adapt, not a complete replacement for your site's robots.txt.

# Example additions. Preserve the site's existing rules.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Review existing restrictions before adding permissive crawler groups. Carry any appropriate exclusions into the groups that need them. Robots.txt is public guidance for cooperating crawlers, not authentication: private content must remain protected by real access controls.

User-requested fetchers need their own review. OpenAI says robots.txt may not apply to ChatGPT-User; Perplexity says its user fetcher generally ignores those rules. Do not promise that a robots.txt rule can prevent every assistant or scraper from opening a public page.

How does WPCentrify help with a WordPress AI bot audit?

WPCentrify brings crawler policy, SEO checks and citation measurement into one WordPress management workflow. Its practical advantage grows when you need the same clear policy across several websites.

With AI crawler control, one policy asks 23 training crawlers to stay away while leaving AI search crawlers and user fetchers alone. The rules are added to the robots.txt WordPress serves; a physical robots.txt file is not rewritten. The daily outside check reads the live robots.txt so you can see which crawlers it refuses.

That is a useful improvement over keeping a separate, forgotten block list on every site. Pair it with the SEO audit for search-access problems and AI citation tracking to see which questions link to your pages.

Keep the boundary clear: WPCentrify's robots.txt policy does not override your host or CDN firewall, and reading robots.txt does not prove that a verified crawler reached every article. Use the manual response and log checks above for those layers.

After access is fixed, improve the page people actually need. Give a direct answer, explain the next steps, cite reliable sources and link related pages naturally. WPCentrify's internal linking tools can help connect those resources without turning the post into a list of repeated keywords.

What should I record in an AI crawler audit?

Keep a small evidence record for each site so the next person can repeat the check. This worksheet is a template, not a report of tests performed on your website.

• Site, page URL, check date and responsible person.
• Intended search, training and user-fetch access policy.
• Live robots.txt result for each relevant crawler.
• Host, CDN or firewall rule affecting the request.
• Status code, response body and verified log evidence where available.
• Indexing, canonical and snippet eligibility checks for Google.
• Exact change, recheck result and next review date.
• Relevant AI citations and referral visits recorded separately.

Repeat the access checks after a hosting move, CDN change, security-plugin replacement or robots.txt edit. A routine WordPress maintenance checklist is a good place to assign that review.

Also review the tools enforcing your policy. An unsupported security or SEO plugin can introduce its own maintenance problem. Our guide to finding and replacing abandoned WordPress plugins explains how to assess that risk.

Sources and further reading

Provider behavior and product capabilities were checked on 5 November 2026. Crawler names and policies can change, so use these primary references when reviewing your configuration.

• OpenAI: crawler purposes and independent access controls.
• Anthropic: Claude crawlers and robots.txt controls.
• Perplexity: crawlers, user fetchers and verification.
• Google Search Central: AI features and your website.
• Google Search Central: what robots.txt does.
• Cloudflare: AI crawler and agent access policies.