AI crawler directory
What is meta-externalagent?
Last verified
Meta's crawler for use cases such as training foundation AI models or improving products by indexing content directly. Meta says it may cache robots.txt for up to 24 hours.
meta-externalagent at a glance
- Operator
- Meta
- Kind
- Crawler: fetches on its own schedule
- Used for
- Model training, Other
- Feeds
- Meta foundation model training; Meta products (indexing)
- robots.txt token
meta-externalagent- User agent strings
meta-externalagent/1.1 (+/documentation/sharing/webmasters/web-crawlers)meta-externalagent/1.1
- Follows robots.txt
- Yes. Meta: add a disallow for the relevant crawler to block it.
- Runs JavaScript
- Not documented by the operator
- Published IP ranges
- Not documented by the operator
- Official source
- Meta Web Crawlershttps://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
- Verified
Every AI crawler in one table
meta-externalagent is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.