Kuldeep Paul

AI search index · OpenAI

OAI-SearchBot

Builds the index behind ChatGPT search. OpenAI says sites that block it will not be shown in ChatGPT search answers, though they can still appear as navigational links. Robots.txt changes take about 24 hours to apply.

Compiled by · Updated Oct 29, 2026

7.6%of top sites fully block OAI-SearchBot
433sites blocking, of 5,701 with robots.txt
61of the top 1,000 block it
Operator
OpenAI
Purpose
AI search index
Used for
ChatGPT search
Obeys robots.txt
Yes
robots.txt token
OAI-SearchBot
User-agent string
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36; compatible; OAI-SearchBot/1.4; +https://openai.com/searchbot
IP list
openai.com/searchbot.json
Official docs
developers.openai.com
Feeds
ChatGPT, ChatGPT Atlas

Block OAI-SearchBot in robots.txt

User-agent: OAI-SearchBot
Disallow: /

Blocking a search or user-triggered bot removes your pages from that assistant's answers. To opt out of model training only, block the operator's training crawler instead.

Well-known sites that block OAI-SearchBot

amazon.com #30, msn.com #92, nytimes.com #163, cnn.com #195, bbc.com #216, bbc.co.uk #236, launchpad.net #281, weibo.com #318, amazon.co.uk #326, amazon.de #387, claude.ai #411, fandom.com #473, cnbc.com #474, amazon.co.jp #490, quora.com #491, usatoday.com #617, dailymail.com #619, telegraph.co.uk #629, amazon.fr #632, amazon.ca #633, amazon.in #676, sharethrough.com #681, yelp.com #702, amazon.es #714

Ranks from the Tranco list. Full per-site data in the robots.txt study.

Other OpenAI bots

GPTBot (model training) · ChatGPT-User (user-triggered fetch) · OAI-AdsBot (ad verification) · ChatGPT Agent (ai agent)

Questions

What is OAI-SearchBot?

OAI-SearchBot is a ai search index bot operated by OpenAI, used for ChatGPT search. Builds the index behind ChatGPT search. OpenAI says sites that block it will not be shown in ChatGPT search answers, though they can still appear as navigational links. Robots.txt changes take about 24 hours to apply.

How do I block OAI-SearchBot?

Add "User-agent: OAI-SearchBot" followed by "Disallow: /" to the robots.txt file at the root of each host you want to protect.

Does OAI-SearchBot respect robots.txt?

Yes. OpenAI documents that OAI-SearchBot honours robots.txt directives.

How many websites block OAI-SearchBot?

In a Oct 29, 2026 analysis of the top 10,000 websites by Kuldeep Paul, 7.6% of sites with a robots.txt file fully disallowed OAI-SearchBot.

Analysis using this data

  1. Who blocks AI crawlers? What 10,000 robots.txt files revealPublishers block training, not answers; half of all blocklists leak; Cloudflare writes the most common AI policy.
  2. The opt-out asymmetry: you can block the small AI engines, not the big oneThe AI surfaces with the most users are the ones publishers can't refuse and can't measure.

← All AI crawlers

How to cite this data

This dataset is free to use, including commercially, under Creative Commons Attribution 4.0. The one rule is attribution:

  1. Credit the source as “Kuldeep Paul, kuldeep.cc” wherever you use a figure, chart or table from it.
  2. Link to this page (https://kuldeep.cc/data/ai-search/crawlers/oai-searchbot), not to the download file, so readers can check the source of every number.
  3. Include the date. These figures change; say “as of Oct 29, 2026” or link to the version you used.
  4. Mark changes. If you filter, recalculate or chart the data yourself, say so (“analysis based on data from…”).
  5. Cite the original source for single figures when you quote one reported number; it is linked in every row.

In running text: according to Kuldeep Paul's OAI-SearchBot — AI crawler directory (kuldeep.cc). More in the methodology.

APA

Paul, K. (2026). OAI-SearchBot — AI crawler directory [Data set]. kuldeep.cc. https://kuldeep.cc/data/ai-search/crawlers/oai-searchbot

MLA

Paul, Kuldeep. "OAI-SearchBot — AI crawler directory." kuldeep.cc, 29 October 2026, https://kuldeep.cc/data/ai-search/crawlers/oai-searchbot.

Chicago

Paul, Kuldeep. 2026. "OAI-SearchBot — AI crawler directory." Dataset. kuldeep.cc. Updated Oct 29, 2026. https://kuldeep.cc/data/ai-search/crawlers/oai-searchbot.

BibTeX

@misc{paul2026oaisearchbot, author = {Paul, Kuldeep}, title = {OAI-SearchBot — AI crawler directory}, year = {2026}, howpublished = {\url{https://kuldeep.cc/data/ai-search/crawlers/oai-searchbot}}, note = {Dataset, CC BY 4.0, updated 2026-10-29} }

HTML link

<a href="https://kuldeep.cc/data/ai-search/crawlers/oai-searchbot">OAI-SearchBot — AI crawler directory</a> (data: Kuldeep Paul, kuldeep.cc)