List of AI Crawlers in 2026: 28 Bots and How to Verify Them

Short answer: AI crawlers in 2026 do three jobs: training crawlers such as GPTBot and ClaudeBot, search crawlers such as OAI-SearchBot and PerplexityBot, and fetchers such as ChatGPT-User that visit a page when someone asks. The list below has 28 names from 10 companies’ own crawler pages. To tell a real visit from a fake one, check the visitor’s IP address against the owner’s published list. Our free script does that for 20 lists.

Checked on Oct 20, 2026. What we did: we read the crawler pages of OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Common Crawl, Amazon, Mistral and DuckDuckGo, downloaded the 20 IP lists they link, ran the script below on a test log built from real addresses in those lists plus forged lines, and compared 8 pages that rank on Google or Bing for this search. What we did not test: live traffic on a real server, reverse DNS, signed requests (Web Bot Auth), whether any bot actually obeys robots.txt, and bots that aren’t on these 10 pages, such as Bytespider.

What are the three kinds of AI crawlers?

Seven of the ten companies below run more than one bot, one per job, and the job decides what a robots.txt block costs you:

• Training crawlers collect pages that may be used to train models: GPTBot, ClaudeBot, MistralAI-Training, and meta-externalagent, which Meta says is also used for “improving products by indexing content directly.”
• Search crawlers build the index an assistant searches when it answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, Amzn-SearchBot, MistralAI-Index. OpenAI says sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers.”
• User-triggered fetchers visit one page because a person asked: ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User, MistralAI-User, Google-Agent, Google-GeminiNotebook.
• Control tokens go in robots.txt but never show up in a request: Google-Extended and Applebot-Extended. Google says Google-Extended “doesn’t have a separate HTTP request user agent string.”

A few names mix jobs. Apple says Applebot data powers its search features and “may also be used to help train Apple foundation models”. Amazon says Amazonbot improves its products and “may be used to train Amazon AI models”. Common Crawl’s CCBot builds an “open repository of web crawl data” that anyone can download.

Which AI crawlers are there in 2026?

Here are all 28, with each owner’s description and what the owner says about robots.txt. The last column is the number of entries in the IP list the owner links, as we downloaded it.

Company	Name	Job	robots.txt, per the owner	IP list entries
OpenAI	GPTBot	Training	Disallow = content not used in training	18
OpenAI	OAI-SearchBot	ChatGPT search	Opted-out sites not shown in search answers	39
OpenAI	ChatGPT-User	User request	“robots.txt rules may not apply”	234
OpenAI	OAI-AdsBot	Checks pages submitted as ChatGPT ads	Visits only pages submitted as ads	2
Anthropic	ClaudeBot	Training	Honors robots.txt	38 (one list for all three)
Anthropic	Claude-SearchBot	Search	Honors robots.txt	same list
Anthropic	Claude-User	User request	Honors robots.txt	same list
Perplexity	PerplexityBot	Search; not for training models	Owner asks you to allow it	8
Perplexity	Perplexity-User	User request	“generally ignores robots.txt rules”	4
Google	Google-Extended	Token: Gemini training and grounding	Robots.txt token only	none (no requests)
Google	GoogleOther	Generic, e.g. internal research	Has its own token	Google lists
Google	Google-CloudVertexBot	Crawls site owners request for Vertex AI Agents	Has its own token	Google lists
Google	Google-Agent	Agents acting on user request	“generally ignore robots.txt rules”	20
Google	Google-GeminiNotebook	URLs users add as Gemini Notebook sources	“generally ignore robots.txt rules”	Google lists
Apple	Applebot	Search; data may train models	Honors robots.txt	24
Apple	Applebot-Extended	Token: Apple model training	Robots.txt token only	none (no requests)
Meta	meta-externalagent	Training or indexing	Blockable by name	none
Meta	Meta-WebIndexer	Meta AI search	Allowing helps Meta AI cite you	none
Meta	Meta-ExternalFetcher	User request	“may bypass robots.txt rules”	none
Meta	Meta-ExternalAds	Ads and business products	Blockable by name	none
Common Crawl	CCBot	Open web dataset	Disallow example given	5
Amazon	Amazonbot	Products; may train models	Honors robots.txt	1,292
Amazon	Amzn-SearchBot	Search, e.g. Alexa	Honors robots.txt	816
Amazon	Amzn-User	User request	“may not follow all robots.txt directives”	1,023
Mistral	MistralAI-Training	Training	Can be disallowed	none
Mistral	MistralAI-Index	Search index; not training	Not stated	2
Mistral	MistralAI-User	User request	Governs which sites users can fetch	4
DuckDuckGo	DuckAssistBot	AI-assisted answers; not training	Disallow takes effect after 72 hours	486

Microsoft isn’t in the table because Bing’s crawler page names no AI-specific crawler or opt-out token. It lists Bingbot, AdIdxBot, BingPreview, MicrosoftPreview and BingVideoPreview.

Which crawler names are new or renamed?

Twelve names on the owners’ pages are missing from many lists that rank for this search: OAI-AdsBot, Claude-SearchBot, Claude-User, Meta-WebIndexer, Meta-ExternalAds, Amzn-SearchBot, Amzn-User, MistralAI-Index, MistralAI-Training, Google-Agent, Google-GeminiNotebook and Google-CloudVertexBot. We counted them on 8 pages: 7 that rank on Google for “list of ai crawlers” or “ai crawlers”, and 1 that ranks on Bing for “ai crawlers”. Two pages had all 12, one had none, and the other five had between 2 and 7.

One rename to know: Google’s page lists Google-NotebookLM as the former name of the Gemini Notebook fetcher, “supported until August 2026”. If your log filters or firewall rules match on the old name, add Google-GeminiNotebook.

Old names linger too. Anthropic’s page describes three bots, ClaudeBot, Claude-User and Claude-SearchBot, and doesn’t mention Claude-Web, yet two of the eight pages describe Claude-Web as a working Anthropic agent. At least two of the pages label PerplexityBot as collecting training data, while Perplexity’s page says it “is not used to crawl content for AI foundation models.” Lists go stale, so check the owner’s page before you copy names into robots.txt.

Which AI crawlers ignore robots.txt?

The ones whose owners say so are user-triggered fetchers, the bots that fetch a page because a person asked. In the owners’ own words:

• ChatGPT-User: “robots.txt rules may not apply.”
• Perplexity-User: “generally ignores robots.txt rules.”
• Google’s user-triggered fetchers, including Google-Agent and Google-GeminiNotebook: “generally ignore robots.txt rules.”
• Meta-ExternalFetcher: “may bypass robots.txt rules.”
• Amzn-User: “may not follow all robots.txt directives.”

One crawler outside that group gets a similar note: Meta says FacebookExternalHit, its link-preview crawler, might bypass robots.txt during security or integrity checks.

Anthropic and Mistral take the other position. Anthropic’s page says its bots honor robots.txt, and that disabling Claude-User “prevents our system from retrieving your content in response to a user query.” Mistral says MistralAI-User “governs which sites these user requests can be made to.” Either way, the robots.txt standard, RFC 9309, says its rules “are not a form of access authorization.” It is a request.

How do I check that a crawler is really who it says it is?

Match the visitor’s IP address against the list its owner publishes. A user-agent string is text that any client can send: Bing’s page says user agent strings “are easy to spoof”, and Common Crawl says it is “aware of crawlers falsely identifying themselves as CCBot.” The script below downloads the 20 IP lists that 9 of the 10 owners link from their crawler pages, then checks one IP or a whole access log. Python 3, standard library only (it uses certifi for certificates if it is installed):

#!/usr/bin/env python3
"""ai-crawler-verify.py - check whether a request that claims to be an AI crawler
comes from an IP address its owner publishes.

  python3 ai-crawler-verify.py lists                    download every published list, print counts
  python3 ai-crawler-verify.py ip 20.171.207.5 GPTBot   is this IP in the owner's published lists?
  python3 ai-crawler-verify.py log access.log           check every AI-crawler line in a combined-format log

The table below comes from each owner's crawler documentation page (read 2026-10-20).
A user-agent string can be copied by anyone; the IP check is what tells you who sent it.
"""
import ipaddress, json, re, ssl, sys, urllib.request

try:
    import certifi
    CTX = ssl.create_default_context(cafile=certifi.where())
except ImportError:
    CTX = ssl.create_default_context()

G = 'https://developers.google.com/static/crawling/ipranges/'
GOOGLE = [G + 'common-crawlers.json', G + 'special-crawlers.json', G + 'user-triggered-fetchers.json',
          G + 'user-triggered-fetchers-google.json', G + 'user-triggered-agents.json']
ANTHROPIC = ['https://claude.com/crawling/bots.json']
AMAZON = ['https://developer.amazon.com/amazonbot/ip-addresses/', 'https://developer.amazon.com/amazonbot/searchbot-ip-addresses/',
          'https://developer.amazon.com/amazonbot/live-ip-addresses/']

# user-agent token: (owner, job, lists the owner publishes for verification)
CRAWLERS = {
    'GPTBot':             ('OpenAI', 'training', ['https://openai.com/gptbot.json']),
    'OAI-SearchBot':      ('OpenAI', 'search', ['https://openai.com/searchbot.json']),
    'ChatGPT-User':       ('OpenAI', 'user fetch', ['https://openai.com/chatgpt-user.json']),
    'OAI-AdsBot':         ('OpenAI', 'ads check', ['https://openai.com/adsbot.json']),
    'ClaudeBot':          ('Anthropic', 'training', ANTHROPIC),
    'Claude-SearchBot':   ('Anthropic', 'search', ANTHROPIC),
    'Claude-User':        ('Anthropic', 'user fetch', ANTHROPIC),
    'PerplexityBot':      ('Perplexity', 'search', ['https://www.perplexity.com/perplexitybot.json']),
    'Perplexity-User':    ('Perplexity', 'user fetch', ['https://www.perplexity.com/perplexity-user.json']),
    'GoogleOther':        ('Google', 'generic', GOOGLE),
    'Google-CloudVertexBot': ('Google', 'site-owner requested', GOOGLE),
    'Google-Agent':       ('Google', 'user fetch', GOOGLE),
    'Google-GeminiNotebook': ('Google', 'user fetch', GOOGLE),
    'Applebot':           ('Apple', 'search (may train)', ['https://search.developer.apple.com/applebot.json']),
    'meta-externalagent': ('Meta', 'training', []),
    'meta-webindexer':    ('Meta', 'search', []),
    'meta-externalfetcher': ('Meta', 'user fetch', []),
    'meta-externalads':   ('Meta', 'ads', []),
    'CCBot':              ('Common Crawl', 'open dataset', ['https://index.commoncrawl.org/ccbot.json']),
    'Amazonbot':          ('Amazon', 'training + products', AMAZON[:1]),
    'Amzn-SearchBot':     ('Amazon', 'search', AMAZON[1:2]),
    'Amzn-User':          ('Amazon', 'user fetch', AMAZON[2:]),
    'MistralAI-User':     ('Mistral', 'user fetch', ['https://mistral.ai/mistralai-user-ips.json']),
    'MistralAI-Index':    ('Mistral', 'search', ['https://mistral.ai/mistralai-index-ips.json']),
    'MistralAI-Training': ('Mistral', 'training', []),
    'DuckAssistBot':      ('DuckDuckGo', 'AI answers', ['https://duckduckgo.com/duckassistbot.json']),
}
# Robots.txt-only tokens: they never appear in a request, so there is nothing to verify.
CONTROL_TOKENS = {'Google-Extended': 'Google', 'Applebot-Extended': 'Apple'}

_cache = {}


def fetch_list(url):
    """Return (networks, creationTime) for one published list."""
    if url in _cache:
        return _cache[url]
    req = urllib.request.Request(url, headers={'User-Agent': 'ai-crawler-verify/1.0 (+https://answerschema.com/)'})
    body = urllib.request.urlopen(req, timeout=30, context=CTX).read().decode('utf-8', 'replace')
    if url.endswith('.json'):
        data = json.loads(body)
        nets = [ipaddress.ip_network(p.get('ipv4Prefix') or p.get('ipv6Prefix'), strict=False) for p in data.get('prefixes', [])]
        created = data.get('creationTime', '')
    else:  # Amazon publishes plain HTML pages of single addresses
        text = re.sub(r'(?is)<(script|style)[^>]*>.*?</\1>', ' ', body)
        nets = [ipaddress.ip_network(a, strict=False) for a in sorted(set(re.findall(r'\b\d{1,3}(?:\.\d{1,3}){3}(?:/\d{1,2})?\b', text)))]
        created = ''
    _cache[url] = (nets, created)
    return _cache[url]


def find_token(ua):
    low = ua.lower()
    for token in sorted(CRAWLERS, key=len, reverse=True):  # longest first: Claude-SearchBot before ClaudeBot
        if token.lower() in low:
            return token
    return None


def verify(ip, token):
    owner, job, urls = CRAWLERS[token]
    if not urls:
        return 'NO LIST', f'{owner} publishes no IP list for {token} on its crawler page'
    addr = ipaddress.ip_address(ip)
    for url in urls:
        try:
            nets, _ = fetch_list(url)
        except Exception as e:
            return 'NOT CHECKED', f'could not download {url}: {type(e).__name__}'
        if any(addr in n for n in nets if n.version == addr.version):
            return 'VERIFIED', f'{ip} is in {url.rsplit("/", 2)[-2] if url.endswith("/") else url.rsplit("/", 1)[-1]}'
    return 'NOT IN LIST', f'{ip} is not in any list {owner} publishes for {token}'


def cmd_lists():
    seen = []
    for token, (owner, job, urls) in CRAWLERS.items():
        for url in urls:
            if url in seen:
                continue
            seen.append(url)
            try:
                nets, created = fetch_list(url)
                v4 = sum(n.version == 4 for n in nets)
                print(f'{owner:<12} {url:<78} {len(nets):>5} entries ({v4} IPv4, {len(nets) - v4} IPv6)  {created}')
            except Exception as e:
                print(f'{owner:<12} {url:<78} ERROR {type(e).__name__}: {e}')
    print('No list on the owner page:', ', '.join(t for t, c in CRAWLERS.items() if not c[2]))
    print('Robots.txt-only tokens (never sent in requests):', ', '.join(CONTROL_TOKENS))


def cmd_ip(ip, ua):
    token = find_token(ua)
    if not token:
        print(f'{ua!r} is not an AI crawler token in this table'); return
    status, why = verify(ip, token)
    print(f'{status:<12} {token:<20} {why}')


LOG = re.compile(r'^(\S+) .*?"[^"]*" \d{3} \S+ "[^"]*" "([^"]*)"')


def cmd_log(path):
    counts = {}
    with open(path, encoding='utf-8', errors='replace') as f:
        for line in f:
            m = LOG.match(line)
            if not m:
                continue
            ip, ua = m.groups()
            token = find_token(ua)
            if not token:
                continue
            status, _ = verify(ip, token)
            counts[(token, status)] = counts.get((token, status), 0) + 1
            if status == 'NOT IN LIST':
                print(f'SPOOF?  {ip:<40} claims {token}')
    print('\ncrawler                status         hits')
    for (token, status), n in sorted(counts.items()):
        print(f'{token:<22} {status:<14} {n}')


if __name__ == '__main__':
    a = sys.argv[1:]
    if a[:1] == ['lists']:
        cmd_lists()
    elif a[:1] == ['ip'] and len(a) == 3:
        cmd_ip(a[1], a[2])
    elif a[:1] == ['log'] and len(a) == 2:
        cmd_log(a[1])
    else:
        print(__doc__)

Save it as ai-crawler-verify.py. Running python3 ai-crawler-verify.py lists printed this:

OpenAI       https://openai.com/gptbot.json                                                    18 entries (18 IPv4, 0 IPv6)  2026-09-22T02:00:07.000000
OpenAI       https://openai.com/searchbot.json                                                 39 entries (39 IPv4, 0 IPv6)  2026-01-02T11:00:00.000000
OpenAI       https://openai.com/chatgpt-user.json                                             234 entries (234 IPv4, 0 IPv6)  2026-10-07T16:05:32.418450
OpenAI       https://openai.com/adsbot.json                                                     2 entries (2 IPv4, 0 IPv6)  2026-05-12T03:00:00.000000
Anthropic    https://claude.com/crawling/bots.json                                             38 entries (38 IPv4, 0 IPv6)  2026-10-07T00:00:00Z
Perplexity   https://www.perplexity.com/perplexitybot.json                                      8 entries (8 IPv4, 0 IPv6)  2025-02-07T16:56:00.000000
Perplexity   https://www.perplexity.com/perplexity-user.json                                    4 entries (4 IPv4, 0 IPv6)  2025-10-17T10:17:00.000000
Google       https://developers.google.com/static/crawling/ipranges/common-crawlers.json      317 entries (170 IPv4, 147 IPv6)  2026-10-20T14:46:18.000000
Google       https://developers.google.com/static/crawling/ipranges/special-crawlers.json     272 entries (136 IPv4, 136 IPv6)  2026-10-20T14:46:18.000000
Google       https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json  1068 entries (534 IPv4, 534 IPv6)  2026-10-20T14:46:17.000000
Google       https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers-google.json   504 entries (252 IPv4, 252 IPv6)  2026-10-20T14:46:18.000000
Google       https://developers.google.com/static/crawling/ipranges/user-triggered-agents.json    20 entries (12 IPv4, 8 IPv6)  2026-10-20T14:46:17.000000
Apple        https://search.developer.apple.com/applebot.json                                  24 entries (24 IPv4, 0 IPv6)  2026-09-15T10:00:00.000000
Common Crawl https://index.commoncrawl.org/ccbot.json                                           5 entries (4 IPv4, 1 IPv6)  2026-08-11T13:40:00Z
Amazon       https://developer.amazon.com/amazonbot/ip-addresses/                            1292 entries (1292 IPv4, 0 IPv6)  
Amazon       https://developer.amazon.com/amazonbot/searchbot-ip-addresses/                   816 entries (816 IPv4, 0 IPv6)  
Amazon       https://developer.amazon.com/amazonbot/live-ip-addresses/                       1023 entries (1023 IPv4, 0 IPv6)  
Mistral      https://mistral.ai/mistralai-user-ips.json                                         4 entries (4 IPv4, 0 IPv6)  2025-02-19T12:00:00.000000
Mistral      https://mistral.ai/mistralai-index-ips.json                                        2 entries (2 IPv4, 0 IPv6)  2026-04-19T12:00:00.000000
DuckDuckGo   https://duckduckgo.com/duckassistbot.json                                        486 entries (486 IPv4, 0 IPv6)  2026-09-01T12:44:58.000000
No list on the owner page: meta-externalagent, meta-webindexer, meta-externalfetcher, meta-externalads, MistralAI-Training
Robots.txt-only tokens (never sent in requests): Google-Extended, Applebot-Extended

Then we built a test log with one line for each of 19 bots, using an address from each bot’s own list (plus an IPv6 line for CCBot and a NO LIST line each for meta-externalagent and MistralAI-Training), and added four forged lines: two from documentation-only addresses claiming to be GPTBot and ClaudeBot, a real GPTBot address claiming to be PerplexityBot, and an address from OpenAI’s ChatGPT-User list claiming to be GPTBot. The script flagged all four. In our unit test, all 30 checks passed, including the rule that Claude-SearchBot must not be read as ClaudeBot. Output of the log mode:

$ python3 ai-crawler-verify.py log sample-access.log
SPOOF?  203.0.113.7                              claims GPTBot
SPOOF?  203.0.113.8                              claims ClaudeBot
SPOOF?  132.196.86.1                             claims PerplexityBot
SPOOF?  104.208.184.193                          claims GPTBot

crawler                status         hits
Amazonbot              VERIFIED       1
Amzn-SearchBot         VERIFIED       1
Amzn-User              VERIFIED       1
Applebot               VERIFIED       1
CCBot                  VERIFIED       2
ChatGPT-User           VERIFIED       1
Claude-SearchBot       VERIFIED       1
Claude-User            VERIFIED       1
ClaudeBot              NOT IN LIST    1
ClaudeBot              VERIFIED       1
DuckAssistBot          VERIFIED       1
GPTBot                 NOT IN LIST    2
GPTBot                 VERIFIED       1
Google-Agent           VERIFIED       1
GoogleOther            VERIFIED       1
MistralAI-Index        VERIFIED       1
MistralAI-Training     NO LIST        1
MistralAI-User         VERIFIED       1
OAI-AdsBot             VERIFIED       1
OAI-SearchBot          VERIFIED       1
Perplexity-User        VERIFIED       1
PerplexityBot          NOT IN LIST    1
PerplexityBot          VERIFIED       1
meta-externalagent     NO LIST        1

The last forged line is the one to notice. The address really belongs to OpenAI, but not to GPTBot, so a check that only asks “is this OpenAI?” would have passed it.

What can’t an IP check tell you?

Four things, from our test and the owners’ pages:

• Some bots have no list. Meta’s crawler page gives user-agent strings but no IP ranges, and Mistral links no list for MistralAI-Training. The script reports those as NO LIST.
• Some lists cover several bots. Anthropic publishes one list for all three bots, and says an address on it means the crawler “is coming from Anthropic.” OpenAI’s GPTBot and OAI-SearchBot lists share 6 entries. For those addresses, the IP proves the company, not the job.
• Lists have dates. Each JSON file carries a creation time. Google’s five files were generated the day we ran the script, while Perplexity’s PerplexityBot file dates from Feb 7, 2025. An address that isn’t listed may come from a list that is out of date, so recheck before you block anything.
• Control tokens never appear. You won’t find Google-Extended or Applebot-Extended in a log. Google crawls with its existing user agents and applies the token to how it uses the content.

Amazon publishes its addresses as web pages rather than JSON files, so the script reads the addresses out of the HTML. A change to those pages could break that part. And don’t use IP blocks to opt out of training: Anthropic warns that blocking its IP addresses “may not work correctly or persistently guarantee an opt-out” because it stops Anthropic from reading your robots.txt. Use robots.txt to opt out, and use the IP check to spot fakes.

How do I block AI training crawlers but keep AI search?

Put the training names in one robots.txt group. RFC 9309 lets one group carry several user-agent lines. We ran this group through Python’s robots.txt parser: the 7 names below came back blocked, while OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, MistralAI-Index, Amzn-SearchBot, Googlebot, Applebot and Bingbot stayed allowed.

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: meta-externalagent
User-agent: MistralAI-Training
User-agent: CCBot
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /

Amazonbot isn’t in the group because Amazon describes it as more than a training crawler; add it if you want. A group like this only stops bots that read robots.txt. For the full walkthrough see Block AI Training, Allow AI Search, and test your own file with the free AI crawler robots.txt checker. We also cover two names in depth: OAI-SearchBot and Google-Extended.

Where does AI FAQ Schema fit?

AI FAQ Schema is our free WordPress plugin. Its AI Crawler Control screen lists 14 AI crawlers with a block checkbox each, plus a box for your own robots.txt lines, which is where newer names such as OAI-AdsBot or Amzn-SearchBot can go. Its AI Crawler Visits screen counts which AI crawlers read your site over the last 30 days. It matches by user agent only, so when a number looks odd, run the script above on your server log.