How can you check whether AI crawlers can read your website?
Test each AI company's crawler against your site on a schedule, because any one of them can be blocked by a hosting or security setting without anyone noticing.
See what eight AI engines say about your company. We track ChatGPT, Gemini, Google AI Mode, Perplexity and more, every day.
OpenAI, Google, Microsoft, Perplexity, Anthropic and others each send their own crawlers, identified by their own user agents. A firewall rule, a bot-protection setting or a CDN default can block one of them while the site looks perfectly normal to human visitors. We learned this firsthand: after relaunching fiveblocks.com, our tests showed the hosting firewall was quietly blocking OpenAI’s crawlers.
Nothing on the site looked wrong, and we caught it only because we test. We now check 18 crawler user agents on a schedule and treat a block the way we would treat a site outage, because to that engine, it is one.