Can you block AI crawlers, and does it work?
Not reliably. Some crawlers that are blocked on paper still reach the content, and some do not identify themselves at all.
See what eight AI engines say about your company. We track ChatGPT, Gemini, Google AI Mode, Perplexity and more, every day.
The standard tools are robots.txt rules and firewall settings aimed at known AI user agents. Our research found they work unevenly. Some crawlers that a site’s rules block still get to the content, and some do not announce who they are, so there is nothing to block by name. Blocking also has a cost.
If the major engines cannot read your own site, they answer questions about you from whatever else they can find, and you lose the one source you control. For most companies we recommend the opposite: make the site easy for AI to read and accurate enough that you want it quoted.