Key takeaways
- robots.txt is a request that well-behaved crawlers follow. It is not a security control.
- Blocking a crawler can reduce what that company's products know about you, so it is a business decision.
- Separate the crawlers that train models from the ones that fetch a page when a person asks.
Your robots.txt file tells crawlers which parts of your site they may fetch. For years that meant search engines. It now also decides which AI companies' crawlers may read your pages, and different companies run different crawlers for different jobs. A business that wants to be named by assistants usually wants them to be able to read its site.
What the file can and cannot do
It is a request. Well-behaved crawlers follow it and badly behaved ones ignore it, so it is never the right tool for hiding anything private. It also controls fetching, not whether a page can appear in results: a blocked page can still be listed if other sites link to it.
The crawlers worth knowing
| Crawler | Company | What it is for |
|---|---|---|
| GPTBot | OpenAI | Collecting content that may be used to train models |
| OAI-SearchBot | OpenAI | Finding pages to show in search results inside ChatGPT |
| ChatGPT-User | OpenAI | Fetching a page when a person asks about it |
| ClaudeBot | Anthropic | Collecting content that may be used to train models |
| Claude-User | Anthropic | Fetching a page when a person asks about it |
| PerplexityBot | Perplexity | Indexing pages for its search results |
| Google-Extended | A token that controls whether content is used for Gemini; Google says it does not affect ranking in Search | |
| Applebot-Extended | Apple | A token that controls whether content is used to train Apple's models |
| CCBot | Common Crawl | Building an open dataset that many projects use |
Company names and purposes change, so check each company's own documentation before relying on a table like this one.
Training versus fetching
The useful split is between crawlers that gather content for training and those that fetch a page because a person asked, or to show it in search. If you block the fetchers, an assistant may be unable to read your page when someone asks about your business. If you block the trainers, you are choosing how your content is used in future models.
How to decide
- List what you want: to be named in answers, to protect certain content, or both.
- Allow the search and fetch crawlers on every page you want cited.
- Decide separately about training crawlers, and apply the decision to the whole site or only the sections you want to protect.
- Keep private areas behind a login, not behind robots.txt.
An example
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /members/
User-agent: OAI-SearchBot
Allow: /
Sitemap: https://example.com/sitemap.xml
Check before you publish
A single wrong line can block your whole site. Test the file with the free robots.txt tester against the pages that matter, and use the AI crawler generator to build the lines for each company. If you want us to check the whole picture, our technical SEO work starts there.