AI crawler directory
What is meta-webindexer?
Last verified
Navigates the web to improve the quality of Meta AI search results. Meta says allowing it helps Meta cite and link to the site's content in Meta AI's responses.
meta-webindexer at a glance
- Operator
- Meta
- Kind
- Crawler: fetches on its own schedule
- Used for
- Search index
- Feeds
- Meta AI
- robots.txt token
meta-webindexer- User agent strings
meta-webindexer/1.1 (+/documentation/sharing/webmasters/web-crawlers)meta-webindexer/1.1
- Follows robots.txt
- Yes. Meta: add a disallow for the relevant crawler to block it.
- Runs JavaScript
- Not documented by the operator
- Published IP ranges
- Not documented by the operator
- Official source
- Meta Web Crawlershttps://developers.facebook.com/documentation/sharing/webmasters/web-crawlers
- Verified
Every AI crawler in one table
meta-webindexer is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.