Can you block AI crawlers, and does it work?

Quick answer

Not reliably. Some crawlers that are blocked on paper still reach the content, and some do not identify themselves at all.

See what eight AI engines say about your company. We track ChatGPT, Gemini, Google AI Mode, Perplexity and more, every day.

Get a sample AIQ report →

The standard tools are robots.txt rules and firewall settings aimed at known AI user agents. Our research found they work unevenly. Some crawlers that a site’s rules block still get to the content, and some do not announce who they are, so there is nothing to block by name. Blocking also has a cost.

If the major engines cannot read your own site, they answer questions about you from whatever else they can find, and you lose the one source you control. For most companies we recommend the opposite: make the site easy for AI to read and accurate enough that you want it quoted.

Work with Five Blocks

Five Blocks helps companies manage exactly this.

If this is a live issue for you, our team can help. Let’s talk about your situation.

Talk to our team

Tell us a little about your situation and we will be in touch.

Held in strict confidence. We will respond promptly. This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Skip to content