Free Technical SEO Tool
Robots.txt &
Crawler Policy Generator
Create a properly formatted robots.txt file in seconds. Guide Googlebot and Bingbot, block private directories, specify your XML sitemap, and safeguard your proprietary content from unapproved AI web scrapers.
Robots.txt Builder
Specify which search engine crawlers and AI bots can index your website.
/admin/
/api/
/checkout/
/private/
Block OpenAI (GPTBot)
Prevents ChatGPT web scrapers from ingesting site content
Block Anthropic (ClaudeBot)
Restricts Claude web crawlers from reading your pages
Block Common Crawl (CCBot)
Blocks large-scale open AI training dataset crawlers
Generated robots.txt
# robots.txt generated by Webigo SEO Tools # Generated at: 2026-10-17T20:16:42.681Z User-agent: * Disallow: /admin/ Disallow: /api/ Disallow: /checkout/ Disallow: /private/ Sitemap: https://webigo.ai/sitemap.xml
Where do I upload this file?
Place the generated robots.txt file in the public root directory of your domain so it is accessible at https://yourdomain.com/robots.txt. Search engines look for this exact URL before indexing your website.
Frequently Asked Questions about Robots.txt & Crawlers
Technical SEO guidance for crawl budget management and bot authorization.