robots.txt Builder

FREE

Build your robots.txt with full control over AI crawlers and standard bots. Copy or download the finished file.

AI Crawler Permissions

NEW

Control which AI systems can crawl and learn from your content. This directly affects your visibility in ChatGPT, Perplexity, Copilot, and other AI-powered search tools.

7 allowed

1 blocked

ChatGPT

GPTBot

OpenAI's training and browsing crawler

ALLOW

ChatGPT Search

OAI-SearchBot

OpenAI's real-time search index

ALLOW

Perplexity AI

PerplexityBot

Perplexity's search and citation crawler

ALLOW

Claude (Anthropic)

ClaudeBot

Anthropic's AI assistant crawler

ALLOW

Common Crawl

CCBot

Dataset used by many AI models for training

BLOCK

Apple AI

Applebot

Apple Intelligence and Siri sources

ALLOW

Bing / Copilot

Bingbot

Powers Microsoft Copilot AI search

ALLOW

DuckAssist

DuckAssistBot

DuckDuckGo's AI answer feature

ALLOW

Standard Crawlers

Custom User-agent blocks with per-path rules.

USER-AGENT

SITEMAP URL

Optional — helps crawlers discover all your pages.

Outputrobots.txt

# AI CrawlersUser-agent: GPTBotAllow: / User-agent: OAI-SearchBotAllow: / User-agent: PerplexityBotAllow: / User-agent: ClaudeBotAllow: / User-agent: CCBotDisallow: / User-agent: ApplebotAllow: / User-agent: BingbotAllow: / User-agent: DuckAssistBotAllow: / User-agent: *Allow: /Disallow: /admin User-agent: GooglebotAllow: / Sitemap: https://example.com/sitemap.xml

33 lines · 388 bytes