Free tool
robots.txt tester
Paste a robots.txt and check whether a crawler may fetch a page, and which line decides it.
Allowed
Googlebot may fetch /admin/help/faq. Allow: /admin/help/ on line 3 is the longest match.
The same page, every crawler
- GooglebotAllowed
- GPTBotAllowed
- OAI-SearchBotAllowed
- ChatGPT-UserAllowed
- ClaudeBotAllowed
- Claude-UserAllowed
- PerplexityBotAllowed
- Google-ExtendedAllowed
- Applebot-ExtendedAllowed
- CCBotAllowed
Sitemaps named: https://example.com/sitemap.xml
Why it matters
One wrong line in robots.txt can block a whole site or a whole section from search engines and AI assistants. Testing a file against real pages before you publish it catches the mistake while it is still free to fix.
Want us to look at the whole site? Book a call and we will prepare a free audit first.
Questions
Which rule wins when two rules match?
The longest matching pattern wins. If an Allow and a Disallow match with the same length, the Allow wins.
Does this fetch my live robots.txt?
No. You paste the text in, and nothing is sent anywhere. Open your robots.txt in a browser and copy it across.
Ready to be found by Google and named by AI assistants?
Book a 30-minute call. We prepare a free audit of your site first, so you leave with a written read whether or not you hire us.