Free Tool · No signup
AI Crawler Checker
See exactly which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) your robots.txt allows, blocks, or never mentions at all.
What to look for
Blocked doesn't always mean invisible to AI: Common Crawl (CCBot) feeds many AI labs' training sets even when a specific bot like GPTBot is blocked. Check both.
Google-Extended and Applebot-Extended are separate from search: Blocking these only opts out of AI training. It doesn't affect whether Googlebot or Applebot can still index you for search.
"Never mentioned" follows your general rule: A bot with no dedicated block just follows your User-agent: * rules, the same ones that apply to every other unnamed crawler.
There's no single standard yet: Each AI company defines and documents its own bot names. This list covers the major, publicly documented ones, not every crawler that exists.
Get a full crawlability audit
SEOMentor checks robots.txt, sitemap, SSL, broken links, and 50+ issues in one 30-second scan.
Scan your site free