AI crawler directory
What is Google-Agent?
Last verified
Used by agents hosted on Google infrastructure to navigate the web and perform actions when a user asks. Google lists it among its user-triggered fetchers and is also experimenting with Web Bot Auth for it, under the identity https://agent.bot.goog.
Google-Agent at a glance
- Operator
- Kind
- Fetcher: requests a page when a user asks
- Used for
- User-triggered fetch
- Feeds
- Agents hosted on Google infrastructure
- robots.txt token
- Not documented by the operator. It is identified by its user agent only.
- User agent strings
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent)Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko; compatible; Google-Agent; +https://developers.google.com/crawling/docs/crawlers-fetchers/google-agent) Chrome/W.X.Y.Z Safari/537.36
- Follows robots.txt
- No. Google: because the fetch was requested by a user, user-triggered fetchers generally ignore robots.txt rules.
- Runs JavaScript
- Not documented by the operator
- Published IP ranges
- user-triggered-agents.json: https://developers.google.com/static/crawling/ipranges/user-triggered-agents.jsonCheck an address against these ranges
- Official source
- List of Google user-triggered fetchershttps://developers.google.com/crawling/docs/crawlers-fetchers/google-user-triggered-fetchers
- Verified
Every AI crawler in one table
Google-Agent is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.