AI crawler directory
What is Claude-SearchBot?
Last verified
Anthropic's search crawler. It navigates the web to improve the relevance and accuracy of search responses in Claude. Anthropic says disabling it prevents indexing the site for search, which may reduce its visibility in user search results.
Claude-SearchBot at a glance
- Operator
- Anthropic
- Kind
- Crawler: fetches on its own schedule
- Used for
- Search index
- Feeds
- Claude search results
- robots.txt token
Claude-SearchBot- User agent string
- Anthropic names the robots.txt token but does not publish a full user agent string.
- Follows robots.txt
- Yes. Anthropic says its bots honour robots.txt directives and the non-standard Crawl-delay extension.
- Runs JavaScript
- Not documented by the operator
- Published IP ranges
- One list for all of Anthropic's bots: https://claude.com/crawling/bots.jsonCheck an address against these ranges
- Official source
- Does Anthropic crawl data from the web, and how can site owners block the crawler?https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Verified
Every AI crawler in one table
Claude-SearchBot is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.