AI crawler directory
What is GPTBot?
Last verified
OpenAI's training crawler. It crawls content that may be used to train OpenAI's generative AI foundation models. Disallowing GPTBot indicates a site's content should not be used for that training. OpenAI treats it as independent of OAI-SearchBot.
GPTBot at a glance
- Operator
- OpenAI
- Kind
- Crawler: fetches on its own schedule
- Used for
- Model training
- Feeds
- OpenAI foundation model training
- robots.txt token
GPTBot- User agent string
Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot
- Follows robots.txt
- Yes. OpenAI documents the GPTBot robots.txt tag as the training control.
- Runs JavaScript
- Not documented by the operator
- Published IP ranges
- gptbot.json: https://openai.com/gptbot.jsonCheck an address against these ranges
- Official source
- Overview of OpenAI Crawlershttps://developers.openai.com/api/docs/bots
- Verified
Every AI crawler in one table
GPTBot is one of 29 crawlers and fetchers in the AI crawler directory, each checked against its operator's own documentation.