AI crawler directory
What is ClaudeBot?
Last verified
Anthropic's training crawler. It collects web content that could contribute to training Anthropic's generative AI models. Anthropic says restricting ClaudeBot signals that the site's future materials should be excluded from its training datasets.
ClaudeBot at a glance
- Operator
- Anthropic
- Kind
- Crawler: fetches on its own schedule
- Used for
- Model training
- Feeds
- Anthropic model training
- robots.txt token
ClaudeBot- User agent string
- Anthropic names the robots.txt token but does not publish a full user agent string.
- Follows robots.txt
- Yes. Anthropic says its bots honour robots.txt directives and the non-standard Crawl-delay extension.
- Runs JavaScript
- Not documented by the operator
- Published IP ranges
- One list for all of Anthropic's bots: https://claude.com/crawling/bots.jsonCheck an address against these ranges
- Official source
- Does Anthropic crawl data from the web, and how can site owners block the crawler?https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler
- Verified
Every AI crawler in one table
ClaudeBot is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.