Free tool
AI crawler robots.txt generator
Choose which AI crawlers may read your site and get the matching robots.txt lines.
Your robots.txt
User-agent: * Allow: / User-agent: GPTBot Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: ClaudeBot Allow: / User-agent: Claude-User Allow: / User-agent: PerplexityBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: CCBot Allow: / Sitemap: https://example.com/sitemap.xml
Save it as robots.txt at the root of your site. It is a request that well-behaved crawlers follow, not a security control.
Why it matters
Different AI companies use different crawlers, and you decide which ones may read your pages. Allowing them is a business choice: a site that wants to be named by assistants usually allows them.
Want us to look at the whole site? Book a call and we will prepare a free audit first.
Questions
Will blocking a crawler remove me from AI answers?
It can reduce what that company's products know about your site, but assistants may still mention you from other sources.
Is robots.txt enforceable?
It is a request that well-behaved crawlers follow. It is not a security control.
Ready to be found by Google and named by AI assistants?
Book a 30-minute call. We prepare a free audit of your site first, so you leave with a written read whether or not you hire us.