Glossary

What is robots.txt?

A file at the root of a site telling crawlers which areas they may or may not fetch.

In more detail

It controls crawling, not indexing. It is also where sites state their policy for AI crawlers.

Disallowing the admin area while allowing everything else.

Book a 30-minute call. We prepare a free audit of your site first, so you leave with a written read whether or not you hire us.