FAQ Schema & AI FAQ Generator is a free WordPress.org plugin with a table of 14 AI crawlers and a Block checkbox for each one. Ticking a box adds User-agent: [bot] and Disallow: / to the robots.txt WordPress serves. To keep OpenAI’s and Anthropic’s training crawlers out but stay in AI search, block GPTBot and ClaudeBot and leave OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed.
Checked on Nov 10, 2026, with plugin version 1.12.1. What we tested: the robots.txt below, run through Python’s
urllib.robotparserfor eight crawler names. What we did not test: whether each company’s crawler actually obeys the file on a live site.
Which AI crawlers can the plugin block?
The table lists 14 names. Everything is allowed by default, and each row has its own checkbox. Here is what the companies themselves say about nine of them:
| Name (company) | What the company says it does |
|---|---|
| GPTBot (OpenAI) | Crawls content for training its models |
| OAI-SearchBot (OpenAI) | Surfaces websites in ChatGPT search |
| ChatGPT-User (OpenAI) | Visits pages for actions a ChatGPT user asks for |
| ClaudeBot (Anthropic) | Collects content for training |
| Claude-SearchBot (Anthropic) | Improves search results for Claude users |
| Claude-User (Anthropic) | Fetches pages when a Claude user asks a question |
| PerplexityBot (Perplexity) | Surfaces sites in Perplexity search; not used to train AI models |
| Perplexity-User (Perplexity) | Fetches pages for user questions; generally ignores robots.txt |
| Google-Extended (Google) | Controls use for Gemini training and grounding; does not affect inclusion or ranking in Google Search |
The other five rows are Applebot-Extended, CCBot, Bytespider, meta-externalagent and Amazonbot. Our List of AI Crawlers covers each one with its source.
How do you block AI training but keep AI search?
Block the training crawlers by name and leave the search crawlers alone. OpenAI says each of its settings “is independent of the others”, so you can disallow GPTBot and still allow OAI-SearchBot. This is the robots.txt we tested with GPTBot, ClaudeBot and Common Crawl’s CCBot ticked (the Sitemap line from our live robots.txt is left out here). The first group is WordPress’s own default; the plugin adds its lines at the end:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
# AI crawler rules - managed by AI FAQ Schema
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /What Python’s robots.txt parser decided for a blog URL under those rules:
| Crawler | Result |
|---|---|
| GPTBot, ClaudeBot, CCBot | Blocked |
| OAI-SearchBot, Claude-SearchBot, PerplexityBot | Allowed |
| Googlebot, Bingbot | Allowed |
OpenAI says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers”. That is why the plugin leaves search crawlers allowed unless you tick them. The full walkthrough is in Block AI Training, Allow AI Search.
Should you block Google-Extended?
Only if you also want out of Gemini grounding. Google says Google-Extended covers training future Gemini models and grounding in Gemini Apps, and that it does not affect inclusion or ranking in Google Search. Blocking it is a trade, not a free win, so we left it out of the rules above. Details are in Google-Extended explained.
Does robots.txt really stop AI bots?
Only the bots that choose to follow it. The robots.txt standard, RFC 9309, says the rules “are not a form of access authorization.” Perplexity says its Perplexity-User fetcher generally ignores robots.txt. The plugin’s AI crawler visits screen shows which AI crawlers fetched your pages, robots.txt and llms.txt over the last 30 days, matched by user agent, so you can see who came.
How do you set it up in WordPress?
- Go to Plugins → Add New Plugin, search for FAQ Schema & AI FAQ Generator, click Install Now, then Activate.
- Open Settings → AI FAQ Schema and scroll to AI Crawler Access.
- Tick Block next to each crawler you want out, then save.
- Open yoursite.com/robots.txt and check the new lines, or paste your URL into our AI Crawler robots.txt Checker.
Two cases where nothing changes: a real robots.txt file in your web root (the settings page warns you, because the server sends that file instead), and sites set to discourage search engines, where the plugin leaves WordPress’s own robots.txt alone. You can also add your own lines in Custom robots.txt lines.
Get it on WordPress.org. The same plugin is also a free AI FAQ generator and an llms.txt generator.
FAQs
Is this AI crawler blocker plugin free?
Yes. FAQ Schema & AI FAQ Generator is free on WordPress.org under the GPLv2-or-later license. The AI crawler table and robots.txt rules need no API key or account.
Will blocking GPTBot remove my site from ChatGPT search?
No. OpenAI says GPTBot and OAI-SearchBot are set separately, so you can disallow GPTBot for training and still allow OAI-SearchBot, which surfaces sites in ChatGPT search.
Does blocking AI crawlers hurt my Google rankings?
Google says Google-Extended does not affect a site's inclusion in Google Search and is not a ranking signal. GPTBot, ClaudeBot and CCBot are not Google crawlers at all.
Can AI bots ignore robots.txt?
Yes. RFC 9309 says robots.txt rules are not access authorization, so only bots that choose to follow them stop. Perplexity says its Perplexity-User fetcher generally ignores robots.txt.