Robots.txt Generator

Build a correct robots.txt file for your site in seconds. Select your CMS, choose what to block, and copy the result.

Why use our Robots.txt Generator

CMS presets built in

Select WordPress, Shopify, or another common CMS and get sensible default disallow rules pre-filled, no need to research the right paths yourself.

Free robots.txt builder, no signup

Generate a valid robots.txt file online with no account required, unlike CMS SEO plugins that bundle this behind a premium tier.

Avoids the most costly mistakes

Helps you avoid accidentally blocking your entire site with a stray Disallow: /, one of the most common self-inflicted SEO disasters.

Copy-paste ready output

Generates a complete, correctly formatted robots.txt file instantly, ready to upload to your domain root with no manual syntax cleanup.

Your CMS / Platform

Crawl permission

Block common pages

Search result pages

/?s=: Prevent duplicate content from search

Cart / Checkout

/cart: Transactional pages not for indexing

Account / Login pages

/account: Private user pages

Print versions

/*?print=: Avoid duplicate print pages

Tag archive pages

/tag/: Thin content archives

Author archive pages

/author/: Thin content on single-author blogs

Custom disallow paths

One path per line, e.g. /private/

Sitemap URL (optional)

robots.txt

💡 How to upload this

Upload robots.txt to your site's root directory so it's accessible at yoursite.com/robots.txt. In WordPress, use a plugin like Yoast SEO. In Next.js, create src/app/robots.ts.

Why robots.txt matters for SEO

Controls crawl budget: Googlebot has limited time per site. Blocking low-value pages lets it spend more time on your important content.

Prevents duplicate content indexing: Cart pages, search results, and filter URLs can create thousands of duplicate pages if left unblocked.

Protects private areas: Admin pages, login URLs, and internal tools should never appear in search results.

Not a security tool: robots.txt is a directive, not a barrier. Never use it to hide sensitive data, use authentication instead.

Real Robots.txt Generator example: before vs. after

A WordPress site's robots.txt: default/missing rules vs. a properly configured file.

Before · No robots.txt / default

User-agent: *
Disallow:

# No sitemap reference
# wp-admin, cart, and search pages all crawlable

Wastes crawl budget on admin pages and internal search results, and gives crawlers no sitemap link to discover pages faster.

After · Configured robots.txt

User-agent: *
Disallow: /wp-admin/
Disallow: /cart/
Disallow: /?s=
Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

Blocks low-value paths from wasting crawl budget and points crawlers straight to the sitemap for faster page discovery.

Frequently Asked Questions

What is a robots.txt file and do I need one?

Robots.txt is a plain text file at your domain's root (yourdomain.com/robots.txt) that tells search engine crawlers which parts of your site they can and can't access. You don't strictly need one, if it's missing, crawlers assume everything is allowed, but having one gives you explicit control over crawl behaviour.

What happens if I block a page in robots.txt by mistake?

A Disallow rule blocks crawling, not indexing, a blocked page can sometimes still appear in search results (without a description) if other sites link to it. Worse, if Google can't crawl a page, it can't see a noindex tag placed there either, so blocking is the wrong tool for removing a page from search entirely.

Should I block /wp-admin/ and other admin folders?

Yes, this is standard practice. Blocking admin, login, and cart/checkout paths saves crawl budget for pages that actually matter for SEO. Most CMS platforms, including WordPress, already ship with sensible defaults for this.

Do I need to include my sitemap URL in robots.txt?

It's recommended, not required. Adding a Sitemap: line pointing to your sitemap.xml gives crawlers a direct, reliable way to discover it, in addition to submitting it in Google Search Console.