AI crawler directory
What is Googlebot?
Last verified
Google Search's crawler, and the crawler behind Google's AI features in Search. Google says robots.txt rules for Googlebot are the control for how a site is crawled for Search, AI Overviews and AI Mode included, and a page must be indexed and eligible for a snippet to be a supporting link there.
Googlebot at a glance
- Operator
- Kind
- Crawler: fetches on its own schedule
- Used for
- Search index
- Feeds
- Google Search; AI Overviews; AI Mode; Discover; Google Images; Google News
- robots.txt token
Googlebot- User agent strings
Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36
- Follows robots.txt
- Yes. Google: common crawlers always obey robots.txt when crawling automatically.
- Runs JavaScript
- Yes. Google: a headless Chromium renders the page and executes the JavaScript, after a queue.
- Published IP ranges
- common-crawlers.json: https://developers.google.com/static/crawling/ipranges/common-crawlers.jsonCheck an address against these ranges
- Official sources
- List of Google's common crawlershttps://developers.google.com/crawling/docs/crawlers-fetchers/google-common-crawlers
- AI features and your websitehttps://developers.google.com/search/docs/appearance/ai-features
- Understand JavaScript SEO basicshttps://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics
- Verified
Every AI crawler in one table
Googlebot is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.