How can you check whether AI crawlers can read your website?

Quick answer

Test each AI company's crawler against your site on a schedule, because any one of them can be blocked by a hosting or security setting without anyone noticing.

See what eight AI engines say about your company. We track ChatGPT, Gemini, Google AI Mode, Perplexity and more, every day.

Get a sample AIQ report →

OpenAI, Google, Microsoft, Perplexity, Anthropic and others each send their own crawlers, identified by their own user agents. A firewall rule, a bot-protection setting or a CDN default can block one of them while the site looks perfectly normal to human visitors. We learned this firsthand: after relaunching fiveblocks.com, our tests showed the hosting firewall was quietly blocking OpenAI’s crawlers.

Nothing on the site looked wrong, and we caught it only because we test. We now check 18 crawler user agents on a schedule and treat a block the way we would treat a site outage, because to that engine, it is one.

Work with Five Blocks

Five Blocks helps companies manage exactly this.

If this is a live issue for you, our team can help. Let’s talk about your situation.

Talk to our team

Tell us a little about your situation and we will be in touch.

Held in strict confidence. We will respond promptly. This site is protected by reCAPTCHA and the Google Privacy Policy and Terms of Service apply.

Skip to content