AI crawler directory
What is Google-Extended?
Last verified
A robots.txt control token, not a crawler. It governs whether content Google crawls may train future Gemini models and ground answers in Gemini Apps and Grounding with Google Search on Vertex AI. It has no effect on Google Search, AI Overviews or AI Mode, which use Googlebot's index.
Google-Extended at a glance
- Operator
- Kind
- Control token: read in robots.txt, sends no requests
- Used for
- Model training, Grounding
- Feeds
- Gemini Apps (training and grounding); Vertex AI API for Gemini (training); Grounding with Google Search on Vertex AI
- robots.txt token
Google-Extended- User agent string
- No user agent of its own. Google crawls with its existing user agents and reads this token only as a control in robots.txt.
- Follows robots.txt
- Yes. Google documents the token for use in robots.txt only.
- Runs JavaScript
- Not documented by the operator
- Published IP ranges
- None of its own: the crawling is done by Google's common crawlers: https://developers.google.com/static/crawling/ipranges/common-crawlers.json
- Official sources
- List of Google's common crawlershttps://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- AI features and your websitehttps://developers.google.com/search/docs/appearance/ai-features
- Verified
Every AI crawler in one table
Google-Extended is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.