Free Tools/Robots.txt Tester

Technical SEO

Robots.txt Tester

Test whether a specific URL path is allowed or blocked for any crawler, using the same longest-match-wins rule Google itself uses to evaluate robots.txt.

Why use our Robots.txt Tester

Test any path against real rules instantly

Fetch a live robots.txt and check whether a specific path is blocked in seconds, no manual rule-by-rule reading.

Free robots.txt tester, no signup

Test crawl rules online with no account required.

Uses Google's actual matching algorithm

Applies the documented longest-match-wins rule, with ties favoring Allow, the same logic Googlebot itself uses.

Test different crawlers separately

Check Googlebot, Bingbot, GPTBot, and other crawlers independently, since rules can target specific user-agents differently.

Real Robots.txt Tester example: before vs. after

Debugging why a new blog category wasn't getting crawled.

Before · Unclear why pages weren't indexed

/blog/category/ blocked by an old Disallow rule

A leftover rule from an old staging setup was silently blocking an entire category from being crawled.

After · Fixed

/blog/category/ now allowed for Googlebot

Removing the stale rule let Googlebot crawl and index the category pages within days.

Frequently Asked Questions

How does robots.txt matching actually work?

For a given user-agent, the crawler finds the most specific matching Allow or Disallow rule based on path length, longest match wins. If an Allow and Disallow rule match with equal length, Allow wins. This tool applies that exact logic.

What happens if no rule matches a path?

It defaults to allowed. robots.txt is an opt-out system, crawling is allowed unless explicitly disallowed.

Does blocking a page in robots.txt remove it from Google's index?

Not necessarily. robots.txt blocks crawling, not indexing. A blocked URL can still appear in search results (often with no description) if Google discovers it through links elsewhere. Use a noindex meta tag to actually prevent indexing.

Why test different user-agents separately?

robots.txt can target specific crawlers differently, for example, blocking GPTBot from training on your content while still allowing Googlebot to crawl for search. Testing per agent reveals these differences.

What's the difference between this and Google Search Console's robots.txt tester?

Search Console requires a verified property and only shows the version of robots.txt Google has already fetched. This tool fetches the live file directly and works for any domain, no verification required.

Need to generate a new robots.txt from scratch? Try our Robots.txt Generator, or set AI crawler rules with our llms.txt Generator.

Find every crawlability issue on your site

SEOMentor checks robots.txt, sitemaps, and indexing status across your entire site in one scan.

Get started free

Explore More Tools

AI-Powered

Meta Title Generator

Generate 5 SEO-optimized title tag options from your keyword and page topic. Includes character count and click-worthiness score.

Try it free

AI-Powered

Meta Description Generator

Generate 5 SEO-optimized meta description options from your keyword. Includes 140-160 character check and click-worthiness tips.

Try it free

AI-Powered

Schema Markup Generator

Build valid JSON-LD structured data for any page type Article, Product, FAQ, LocalBusiness, and more. Copy-paste ready.

Try it free