Free Robots.txt Generator
Create professional robots.txt files to control how search engines crawl your website. Block unwanted bots, set crawl delays, and improve your SEO instantly.
β‘ Quick Presets
βοΈ Robots.txt Configuration
Use * for all bots, or specify like "Googlebot", "Bingbot", "DuckDuckBot"
Paths crawlers should NOT access. Use / to block entire site.
Paths crawlers CAN access (overrides Disallow for specific URLs)
β οΈ Googlebot ignores crawl-delay. Use Google Search Console for Google crawl rate.
Add your XML sitemap location to help search engines discover all your pages
Add custom user-agent blocks or advanced rules (one per line)
π How to Use Your Robots.txt File
- Configure your rules above using presets or custom settings
- Click "Generate Robots.txt" to create your file
- Click "Download" to save the robots.txt file
- Upload the file to your website's root directory (e.g.,
https://yourdomain.com/robots.txt) - Test your robots.txt usingGoogle Search Console's robots.txt Tester
- Changes typically take effect within 24-48 hours
π€ Common Bot User-Agents
GooglebotGoogle SearchBingbotBing SearchSlurpYahoo SearchDuckDuckBotDuckDuckGoBaiduspiderBaidu SearchYandexBotYandex SearchfacebookexternalhitFacebookTwitterbotTwitter/XLinkedInBotLinkedInGPTBotChatGPT / OpenAICCBotCommon CrawlPetalBotHuawei SearchPopular Robots.txt Rules
Ready-to-use rules for common scenarios β just check and generate
Block Entire Site
User-agent: * Disallow: /
Maintenance mode, staging sites
Allow Everything
User-agent: * Allow: /
Default for most websites
Block Admin Area
User-agent: * Disallow: /wp-admin/ Disallow: /admin/
WordPress, CMS admin
Block Login Pages
User-agent: * Disallow: /login/ Disallow: /signin/
Authentication pages
Block Images
User-agent: * Disallow: /images/ Disallow: /uploads/
Media directories
Googlebot Only
User-agent: Googlebot Allow: / User-agent: * Disallow: /
Allow only Google
How to Use Your Robots.txt File
Configure Rules
Select presets or add custom rules for your website
Generate File
Click generate to create your robots.txt file
Upload to Root
Save as robots.txt and upload to your website root
Test & Validate
Use Google Search Console to test your file
Important Notes & Best Practices
Robots.txt Doesn't Guarantee Blocking
Search engines may still index pages if they find external links. Use noindex meta tags or password protection for sensitive content.
Syntax Matters
Robots.txt is case-sensitive. Use correct capitalization for user-agents and paths. One space after colon is required.
Always Include Sitemap
Add your sitemap URL in robots.txt to help search engines discover all your pages efficiently.
Test Before Deploying
Always test your robots.txt file using Google Search Console's robots.txt Tester to avoid blocking important pages.
Frequently Asked Questions
What is the difference between Allow and Disallow?βΌ
Disallow tells crawlers NOT to access specific URLs or directories. Allow explicitly permits crawling of specific URLs even within a disallowed directory. For example, you could disallow /private/ but allow /private/public-page.html.
Can I block specific search engines only?βΌ
Yes! Use specific user-agent names like 'Googlebot' for Google, 'Bingbot' for Bing, 'Slurp' for Yahoo, or 'DuckDuckBot' for DuckDuckGo. Add rules under each specific user-agent.
What is Crawl-delay and should I use it?βΌ
Crawl-delay tells bots to wait a specified number of seconds between requests. This is useful for reducing server load on busy sites. However, Googlebot ignores crawl-delay - use Google Search Console for crawl rate adjustment instead.
Where should I place my robots.txt file?βΌ
The robots.txt file must be placed in the root directory of your website. For example: https://yourdomain.com/robots.txt. Search engines automatically check this location when crawling your site.
How long does it take for robots.txt changes to take effect?βΌ
Search engines typically re-check robots.txt every few days. Google may take up to 24 hours to detect changes. You can force a re-crawl using Google Search Console's URL Inspection tool.
Can robots.txt prevent indexing of my pages?βΌ
Robots.txt prevents crawling but doesn't guarantee that pages won't be indexed. If other sites link to your blocked pages, search engines might still index them without crawling. Use noindex meta tags for complete removal.