Homepage, plus a product page and a blog page where they exist
✓Free forever✓No login required✓Results in about 30 seconds✓7 access checks
What is Crawl Radar?
Before AI platforms like ChatGPT, Perplexity, or Gemini can recommend your brand, their crawlers need to actually reach your website. Many sites unknowingly block AI crawlers through robots.txt rules, WAF configurations, or CDN settings, making themselves invisible to AI search.
Crawl Radar tests 10 major AI crawlers against up to 3 pages on your site, detecting blocks, challenges, cloaking, throttling, and other access issues. It is the pre-flight check to run before you spend anything optimising your content. Once the crawlers can reach you, the AI Readiness Score tells you what they make of what they find, and the free report shows whether AI is naming you today. Reachability is the first of six stages a model works through before it quotes a page.
What Crawl Radar finds
Hard blocks
The bot gets a 4xx or 5xx error while our browser control loads the page fine. The bot is explicitly denied access.
Cloudflare and CAPTCHA challenges
The bot receives an interstitial challenge page instead of your content. Common with aggressive WAF and bot-management settings.
Content cloaking
The bot receives a page half the size of the browser’s or smaller, with both getting a 200. Your server is serving different content to crawlers.
Throttling
The bot response takes 10 times longer than the browser’s, which suggests crawler traffic is being rate-limited on purpose.
Redirect mismatches
The bot lands on a different final URL than the browser, or gets redirected when the browser does not. Usually a bot-specific redirect rule.
X-Robots-Tag blocks
Response headers carry an X-Robots-Tag with noindex or none, which tells the crawler not to index the page even though it loaded.
JS rendering gaps
The bot receives an empty JavaScript shell while the browser gets fully rendered content. Reported as cloaked, since the readable result is the same.
Frequently asked questions
What is Crawl Radar?
Crawl Radar is a free tool that tests whether the 10 AI crawlers that matter can reach and read your website: GPTBot (OpenAI), ChatGPT-User, OAI-SearchBot, ClaudeBot (Anthropic), Claude-User, Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended (Gemini), and Bytespider (ByteDance). It requests your pages with each bot’s real user-agent string and reports whether that bot got your content, a block, a challenge page, a stripped-down version, or nothing at all.
How does the test work?
We request each page twice: once with the AI bot’s real user-agent and once with a standard Chrome user-agent as a control. Comparing the two responses (status code, body size, response time, headers, and final URL) is what separates a bot-specific problem from a site that is simply down for everyone. Seven checks run against that comparison: hard blocks, Cloudflare and CAPTCHA challenges, redirect mismatches, X-Robots-Tag directives, content cloaking, throttling, and JavaScript rendering gaps.
Is the Crawl Radar test free?
Yes, completely free. No login, no credit card, no paid tier sitting behind it. You get the summary verdict and results for the top 4 bots (GPTBot, ClaudeBot, PerplexityBot, and Google-Extended) as soon as the test finishes. An email unlocks the full 10-bot breakdown with per-page detail and a specific fix for each problem found.
How many pages does it test?
Up to 3. Your homepage always, plus the first product or collection page and the first blog or content page we can find through your sitemap or your homepage links. If a site has neither, the test runs on whatever it does find, down to the homepage alone. Three pages is enough to catch a rule that applies to one section of a site but not another, which a homepage-only check would miss.
What’s the difference between Crawl Radar and an AI Readiness Score?
A AI Readiness Score measures whether your site is built for AI citation once a crawler is already inside: structured data, content format, authority signals. Crawl Radar answers the question that comes before that one. Can the crawler get in at all? If GPTBot receives a 403, nothing you do to your content will get you cited by ChatGPT. Run Crawl Radar first, fix anything it flags, then work on the AI Readiness Score.
What does 'challenged' mean?
Challenged means the bot received a Cloudflare or CAPTCHA interstitial instead of your page, while our Chrome control request loaded the real content. That is almost always a CDN or WAF rule set to challenge non-browser traffic. The bot reaches your server and never gets through the door, which for an AI crawler is the same outcome as a block.
What does 'cloaked' mean?
Cloaked means the bot’s response body came back at half the size of the browser’s response or smaller, with both requests returning HTTP 200. Something between your server and the bot is serving a stripped-down page to crawlers, usually server-side bot detection. We report the same status when the bot receives an empty JavaScript shell and the browser receives fully rendered content, because what the AI ends up able to read is the same either way.
Why might I see 'inconclusive' results?
Inconclusive means we could not tell whether that bot specifically is being blocked. It happens when our Chrome control request fails too (your site was down, or our test infrastructure was blocked at the network layer), when both the bot and the control receive challenge pages, or when the bot request times out. It is not a finding against your site, and re-running the test usually resolves it.
How can I fix a blocked AI crawler?
Start with robots.txt, where most blocks live. Look for Disallow rules under User-agent: GPTBot, ClaudeBot, PerplexityBot, or Google-Extended, and for a blanket User-agent: * Disallow: / that catches all of them at once. If robots.txt is clean, the block is at your edge, and Cloudflare bot-management or WAF rules are the usual cause: a verified-bots-only setting will challenge every AI crawler by default. Check your response headers for an X-Robots-Tag carrying noindex as well. The test names the specific fix for each bot it flags.
Can I get ongoing monitoring instead of a one-time test?
This test is a one-time snapshot. Cited’s Site Health module runs the same crawler-access checks weekly and on demand as part of the platform, alongside the other five stages of the readiness funnel, and every scan reports what changed since the last one. Try free for 14 days.
