Free Tool · No signup
robots.txt Tester
Check whether Googlebot can crawl any URL on your website. Enter your domain and a page path to see exactly what your robots.txt allows or blocks.
Enter the path from the root, e.g. /blog/ or /wp-admin
What to look for
Accidentally blocked pages: A wrong Disallow rule can block your entire site from Google. /admin/ is fine to block, / is not.
Crawl budget waste: Allowing Google to crawl cart pages, search results, and filter URLs wastes crawl budget on pages that shouldn't rank.
Allow vs Disallow priority: When both match a URL, the most specific rule wins, not the order they appear in the file.
Disallow: (empty) means allow all: An empty Disallow value tells crawlers they can access everything. It's the correct default for most public sites.
Get a full crawlability audit
SEOMentor checks robots.txt, sitemap, SSL, broken links, and 50+ issues in one 30-second scan.
Scan your site free