Free Tool · No AI
Robots.txt Generator
Build a correct robots.txt file for your site in seconds. Select your CMS, choose what to block, and copy the result.
Your CMS / Platform
Crawl permission
Block common pages
Custom disallow paths
One path per line, e.g. /private/
Sitemap URL (optional)
robots.txt
💡 How to upload this
Upload robots.txt to your site's root directory so it's accessible at yoursite.com/robots.txt. In WordPress, use a plugin like Yoast SEO. In Next.js, create src/app/robots.ts.
Why robots.txt matters for SEO
Controls crawl budget — Googlebot has limited time per site. Blocking low-value pages lets it spend more time on your important content.
Prevents duplicate content indexing — Cart pages, search results, and filter URLs can create thousands of duplicate pages if left unblocked.
Protects private areas — Admin pages, login URLs, and internal tools should never appear in search results.
Not a security tool — robots.txt is a directive, not a barrier. Never use it to hide sensitive data — use authentication instead.
Check if your robots.txt is blocking Google
SEOMentor checks robots.txt, sitemap, SSL, and 50+ other technical issues in one 30-second scan.
Scan your site free