Free Tools/AI Crawler Checker

Free Tool · No signup

AI Crawler Checker

See exactly which AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more) your robots.txt allows, blocks, or never mentions at all.

Why use our AI Crawler Checker

Instant AI crawler audit

Enter your domain and instantly see which AI bots (GPTBot, ClaudeBot, Google-Extended, PerplexityBot) can access your site, no manual robots.txt reading required.

Free, no signup required

Check AI crawler access online for free, unlike enterprise SEO platforms that bundle this check into a paid subscription.

Covers the latest AI crawlers

Checks 18 named bots used for AI training, live browsing, and link previews, including OpenAI, Anthropic, Google, Perplexity, and Reddit's crawlers, kept current as new bots launch.

Shows your raw robots.txt too

See the exact rules being applied, not just a verdict, so you can confirm why a bot is allowed, blocked, or falling back to your default rule.

Website domain *

What to look for

Blocked doesn't always mean invisible to AI: Common Crawl (CCBot) feeds many AI labs' training sets even when a specific bot like GPTBot is blocked. Check both.

Google-Extended and Applebot-Extended are separate from search: Blocking these only opts out of AI training. It doesn't affect whether Googlebot or Applebot can still index you for search.

"Never mentioned" follows your general rule: A bot with no dedicated block just follows your User-agent: * rules, the same ones that apply to every other unnamed crawler.

There's no single standard yet: Each AI company defines and documents its own bot names. This list covers the major, publicly documented ones, not every crawler that exists.

Real AI Crawler Checker example: before vs. after

A publisher's robots.txt before noticing GPTBot was crawling freely, vs. after deciding which AI bots to allow.

Before · No AI bot rules

User-agent: *
Disallow: /admin/

# No rules for GPTBot, CCBot, Google-Extended
# → all AI crawlers follow the default: allowed

Every AI training crawler (GPTBot, CCBot, Google-Extended, and more) is silently allowed by default, since none are named explicitly.

After · AI bots explicitly set

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Allow: /

A deliberate choice: blocks OpenAI's training crawler while still allowing Google's AI Overviews to use the content, instead of one default rule for all.

Frequently Asked Questions

How do I block GPTBot and other AI crawlers from my site?

Add a User-agent block for that specific bot in robots.txt, for example "User-agent: GPTBot" followed by "Disallow: /". Each AI crawler needs its own block, a rule for GPTBot doesn't affect ClaudeBot or PerplexityBot, they're separate crawlers with separate names.

Does blocking AI crawlers hurt my Google or Bing rankings?

No. AI training crawlers like GPTBot, Google-Extended, and CCBot are entirely separate from the search crawlers (Googlebot, Bingbot) that determine your search rankings. Blocking one has no effect on the other.

Should I block AI crawlers or let them access my site?

It depends on your goals. Blocking prevents your content from being used in AI model training. Allowing access can help your content get cited in AI answer engines like ChatGPT and Perplexity, which is a growing referral traffic source. There's no universally correct choice, it's a real tradeoff to decide for your own site.

What's the difference between GPTBot and ChatGPT-User?

GPTBot is OpenAI's training crawler, it collects content to train future models. ChatGPT-User is a separate crawler that fetches a page in real time when a user asks ChatGPT to browse or summarise it. Blocking GPTBot doesn't block ChatGPT-User, and vice versa, they serve different purposes.

See the text a page gives a crawler that does not run JavaScript with our HTML to Markdown Converter.

Get a full crawlability audit

SEOMentor checks robots.txt, sitemap, SSL, broken links, and 50+ issues in one 30-second scan.

Scan your site free

Explore More Tools

No AI needed

llms.txt Generator

Build the llms.txt file that tells ChatGPT, Perplexity, and Claude which of your pages to cite. Auto-fill from your sitemap or add pages manually.

Try it free

No AI needed

llms.txt Validator

Check any domain's llms.txt against the real spec: missing title, broken links, relative URLs, and other mistakes that stop an AI assistant using it properly.

Try it free

No AI needed

HTML to Markdown Converter

Convert any public URL or pasted HTML into clean Markdown. Strips navigation and clutter, keeps headings, lists, links and tables, and shows the token count.

Try it free