AI Crawler robots.txt Checker
Paste your robots.txt and see, for 14 AI crawlers plus Googlebot and Bingbot, whether each one may read a page, and which rule decided it. Then build a new robots.txt from a preset.
Read this first: robots.txt is a request, not a lock. The robots.txt standard (RFC 9309) says its rules “are not a form of access authorization.” Reputable crawlers follow them; some user-triggered fetchers, such as Perplexity-User, say they generally do not. Your robots.txt lives at yoursite.com/robots.txt; open it in a new tab and copy everything.
1. Check your robots.txt
Use a path, like / for the home page or /blog/my-post/. Matching is case-sensitive, as the standard requires.
2. Result
| Crawler | Type | Result | What it does |
|---|
“Decided by” shows the group and the rule that matched. No matching rule means the page is allowed.
3. Build a robots.txt
Each blocked crawler gets its own group. If you add a group for a crawler, it stops reading the * group, so repeat any * rules it still needs.
How crawlers pick their rules
A crawler looks for groups whose User-agent line names its own token. If it finds one or more, it combines them and ignores the * group. Only when no group names it does it fall back to *. Inside the chosen rules, the longest matching path wins, and Google documents that Allow wins when an Allow and a Disallow rule are the same length.
Google-Extended and Applebot-Extended are control tokens, not separate crawlers. Google says Google-Extended “doesn’t have a separate HTTP request user agent string” and “does not impact a site’s inclusion in Google Search.” Apple says Applebot-Extended “does not crawl webpages.” Blocking them sets a preference for AI training and, for Google, Gemini grounding.
On WordPress, robots.txt is usually generated by WordPress itself. A physical robots.txt file in the site root replaces the generated one, so plugin settings stop applying. More detail: Block AI training, allow AI search, OAI-SearchBot and Google-Extended.
Set this with checkboxes in WordPress
Free plugin: a table of 14 AI crawlers with one Block checkbox each, plus custom robots.txt lines, llms.txt and FAQ schema.
AI crawler robots.txt: common questions
How do I check if GPTBot is blocked on my site?
Open yoursite.com/robots.txt, copy everything, and paste it into the checker above. The GPTBot row shows Allowed or Blocked for the path you test, plus the exact rule that decided it. GPTBot follows a group that names it; when no group names it, it follows the User-agent: * group.
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI runs separate crawlers: GPTBot collects content that may be used for training, and OAI-SearchBot surfaces sites in ChatGPT search. OpenAI says each setting is independent, so you can block GPTBot and still allow OAI-SearchBot. OpenAI also says a robots.txt change can take about 24 hours to reach its search systems.
Why does a crawler ignore my * rules after I add its own group?
Under RFC 9309, a crawler that finds a group naming its token follows that group and skips the * group. If your * group blocks /wp-admin/ and you add a group with User-agent: OAI-SearchBot and Allow: /, OAI-SearchBot may now read /wp-admin/. Repeat the * rules you still need inside the new group.
Does blocking Google-Extended remove me from Google Search or AI Overviews?
No. Google says Google-Extended does not impact a site's inclusion in Google Search and is not a ranking signal. AI Overviews are part of Google Search, and Google's documentation ties them to indexing and snippet controls such as nosnippet, max-snippet and noindex, not to Google-Extended.
Is robots.txt enough to stop AI bots?
No. RFC 9309, the robots.txt standard from 2022, says its rules are not a form of access authorization, so they only work for crawlers that choose to follow them. Perplexity says its Perplexity-User fetcher generally ignores robots.txt. Stopping a bot that ignores robots.txt takes a block at the server, firewall or CDN level.