Robots.txt Generator

robots.txt is a plain-text file placed at the root of your domain (e.g. https://example.com/robots.txt) that tells web crawlers which pages they are allowed or not allowed to access. It follows the Robots Exclusion Protocol and is the standard way to communicate crawl preferences to search engines and bots.

No. robots.txt is a voluntary standard — well-behaved bots like Googlebot, Bingbot, and most major crawlers follow it. However, malicious scrapers and some AI training crawlers may ignore it entirely. For truly private content, use server-side access controls (authentication, firewall rules) rather than relying solely on robots.txt.

Disallow blocks a crawler from accessing a path. Allow explicitly permits access to a path that would otherwise be blocked by a more general Disallow rule. Allow rules take precedence over Disallow for the same path length — for example, you can Disallow /private/ but Allow /private/public/ to expose just that subfolder.

Crawl-delay tells a bot to wait N seconds between requests to your server. This prevents aggressive crawlers from overloading your server. Note: Googlebot ignores Crawl-delay in favor of its own crawl rate settings in Google Search Console. Bingbot and some others do respect it.

Many site owners now block AI data-collection bots like GPTBot (OpenAI), CCBot (Common Crawl), and anthropic-ai to prevent their content from being used to train AI models without consent. You can add separate User-agent blocks for each. Use the 'Block AI crawlers' preset to generate these rules automatically.