Free SEO Tool

robots.txt Generator

Build a valid robots.txt in seconds. Control which bots crawl your site, protect admin areas, block bad crawlers, and add your sitemap URL.

quick_start.preset
Start from a template or build from scratch:
global_settings.conf
Sitemap URL
Tell search engines where to find your sitemap. This applies to all bots.
Sitemap:
Crawl-delay
Optional: ask bots to wait N seconds between requests. Leave 0 to omit.
Applied to: All bots (*)
Seconds
googlebot.rules
Allow all
Common Protected Areas
/wp-admin/
WordPress admin panel — should always be blocked
/wp-includes/
WP core files — not useful to index
/wp-login.php
Login page — block from indexing
/cart/ & /checkout/
WooCommerce pages — private user flows
/search/
Internal search results — thin/duplicate content
/?s= (search query URLs)
WP search query strings — duplicate content risk
/author/ pages
Author archives — low value, thin content
Custom Disallow Rules
Custom Allow Rules
block_bots.deny
Block known bad crawlers, scrapers, and AI training bots. Each checked bot gets its own Disallow: / rule.
custom_agents.conf
Add extra bot rules for any User-agent not listed above.
robots.txt valid
0 lines
Validation & Tips
💡Configure your settings to see validation feedback.

Auto-manage robots.txt in WordPress

RankAl SEO generates and updates your robots.txt directly from WordPress admin — no FTP needed.

Download Free Plugin View PRO →

// robots.txt reference

User-agent directive

Specifies which crawler the following rules apply to. Use * for all bots, or a specific name like Googlebot.

User-agent: Googlebot

Disallow directive

Blocks a bot from crawling the specified path. An empty value means "allow everything". A single / blocks the entire site.

Disallow: /wp-admin/

Allow directive

Explicitly permits crawling of a path — useful when a parent is disallowed but you want a subfolder crawled.

Disallow: /private/ Allow: /private/public-page.html

Sitemap directive

Points crawlers to your sitemap. This is not agent-specific and should be placed at the end of the file. Use the full absolute URL.

Sitemap: https://yoursite.com/sitemap.xml