Free tool
Is my site blocked from Claude?
Last updated
Claude reads the web through two Anthropic crawlers. Claude-SearchBot indexes content to improve Claude's search results, and Claude-User fetches a page when a user asks Claude a question; Anthropic says blocking either may reduce the site's visibility in user search. ClaudeBot is the third: it collects content that may be used for training, and blocking it signals that the site's future content should be excluded from training data. Anthropic documents that all three honour robots.txt.
Source: Anthropic, Does Anthropic crawl data from the web, read .
Eight requests: robots.txt, the homepage as a browser, and the homepage as six AI crawlers. Nothing is stored.
The crawlers that decide it for Claude
- Claude-SearchBotAnthropic
- Analyses content to improve the relevance and accuracy of Claude's search results. Blocking it prevents the site being indexed for search. (Anthropic, Does Anthropic crawl data from the web, read )In this check: robots.txt verdict, and the homepage requested with its user agent.
- Claude-UserAnthropic
- Fetches a page when a user asks Claude a question. Blocking it prevents Claude retrieving the site in response to a user's query. (Anthropic, Does Anthropic crawl data from the web, read )In this check: robots.txt verdict, and the homepage requested with its user agent.
- ClaudeBotAnthropic
- Collects web content that could contribute to training Anthropic's models. Blocking it excludes the site's future content from training data. (Anthropic, Does Anthropic crawl data from the web, read )In this check: robots.txt verdict, and the homepage requested with its user agent.
Other ways a site is invisible to Claude
Crawler access is the first condition of AI visibility, not the only one. This check does not measure the following.
- JavaScript-only content. Anthropic's crawler fetched JavaScript files but did not run them, in an analysis of hundreds of millions of crawler fetches. Text that only appears after JavaScript runs is not in what it reads. (Vercel and MERJ, The rise of the AI crawler (December 2024))
- Not in the index Claude searches. Anthropic does not document which search index Claude uses. Claude's citations overlapped 86.7% with Brave Search's top results, and Brave has been on Anthropic's subprocessor list since March 2025. (Profound, Claude citations against Brave results)
The free audit crawls up to 25 pages the way an answer engine reads them and reports pages whose content only appears with JavaScript, pages marked noindex and blocked crawlers, with the evidence. Whether a page is in a search index is reported by the index's owner, in Google Search Console or Bing Webmaster Tools; neither this check nor the audit sees it.