Home/Tools/AI Crawler Checker
Free tool · AI Search (AEO & GEO)

AI Crawler Checker

Find out in seconds whether ChatGPT, Claude, Perplexity, Gemini and Google can read your website. Enter any page: we test your robots.txt against 19 crawlers, then load the page as each major AI bot to catch firewall blocks, page tags and JavaScript-only content.

FreeNo sign-up19 crawlers testedFirewall checkPage tagsllms.txt check

Enter a domain or any page URL. We read your robots.txt, then load the page as each major AI crawler to catch firewall blocks, page tags and JavaScript-only content.

How it works

How To Use The AI Crawler Checker

01

Enter any page

Type your domain or paste the URL of any page. You do not need the robots.txt address.

02

Read the report

Every crawler gets Allowed or Blocked with the reason: a robots.txt rule, a firewall block, or a page tag. You also get an AI search visibility score.

03

Fix what is blocked

Open the robots.txt generator with your current rules loaded, switch the right crawlers back on and upload the new file.

Being readable is step one

Want ChatGPT And Gemini To Recommend You, Not Just Crawl You?

Our AEO & GEO team checks which questions your customers ask AI, where you are missing from the answers, and what to fix first.

Get a free AI visibility auditTalk to a strategist
Questions

AI Crawler Checker FAQ

Straight answers to what people ask about this tool.

What does the AI Crawler Checker test?

Three things. It applies your robots.txt rules the way crawlers do (RFC 9309) for 19 user agents, from Googlebot and Bingbot to OAI-SearchBot, Claude-User, PerplexityBot, GPTBot and Google-Extended. It loads your page as 10 major crawlers to spot firewall or CDN blocks. And it reads the page for robots meta tags, X-Robots-Tag headers and how much text is readable without JavaScript. It also checks for an llms.txt file.

What is a good AI search visibility score?

The score counts only the crawlers that put you in search results and AI answers. 100 means all of them can read your homepage. Anything below 100 means at least one AI tool cannot cite or recommend you. Training crawlers are shown separately and do not lower the score.

My site is blocked but I never blocked AI. Why?

The usual causes are a "User-agent: *" group with "Disallow: /" left over from a staging site, an SEO or security plugin that added AI blocks, or rules copied from a template. CDNs such as Cloudflare can also block AI bots at the firewall, which robots.txt does not show. The page check catches this: it marks those crawlers as likely blocked by a firewall.

Does blocking training crawlers hurt my visibility?

No. Training crawlers such as GPTBot and ClaudeBot collect data for future models. Being cited in ChatGPT search or Claude answers depends on the search and assistant crawlers, which you can allow separately.

How accurate is the firewall check?

We load your page from our server using each crawler's name. Most firewalls decide by that name, so the result is usually right. Some, including Cloudflare's verified bot checks, also look at where a request comes from, and can treat the real crawlers differently. Read firewall results as likely, and confirm in your CDN or server logs.

How often should I run a check?

After every site migration, theme or plugin change that touches robots.txt, and once a quarter otherwise. AI companies add new crawlers regularly, and we update this tool when they do.