AI crawler directory
What is meta-externalfetcher?
Last verified
Fetches individual links at a user's request and supports product functions such as evaluating and improving agentic AI capabilities, including helping AI navigate websites to complete tasks for users.
meta-externalfetcher at a glance
- Operator
- Meta
- Kind
- Fetcher: requests a page when a user asks
- Used for
- User-triggered fetch
- Feeds
- Meta AI agentic features
- robots.txt token
meta-externalfetcher- User agent strings
meta-externalfetcher/1.1 (+/documentation/sharing/webmasters/web-crawlers)meta-externalfetcher/1.1
- Follows robots.txt
- Not always. Meta: may bypass robots.txt because it performs fetches that were requested by the user.
- Runs JavaScript
- Not documented by the operator
- Published IP ranges
- Not documented by the operator
- Official source
- Meta Web Crawlershttps://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
- Verified
Every AI crawler in one table
meta-externalfetcher is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.