Kuldeep Paul

AI search engine reference

How each AI engine finds pages, which bots it sends, how its clicks show up in your analytics, and whether it sells ads. One table, sourced from official documentation and verified reporting.

Compiled by , data aggregatorUpdated 18 enginesDownload CSV · JSONMethodology

Reach of the largest AI answer surfaces (millions of users)

AI Overviews · monthly: 2,500M; ChatGPT · weekly: 1,200M; AI Mode · monthly: 1,000M; Gemini app · monthly: 950MAI Overviews · monthly2,500MChatGPT · weekly1,200MAI Mode · monthly1,000MGemini app · monthly950M
As stated by Google and OpenAI in 2026. Weekly and monthly counts aren't directly comparable. Google surfaces in black. Source: AI assistant user numbers
AI engines compared. Select an engine for its full profile and sources.
EngineWhere answers come fromCrawlersReferrerAds
ChatGPTOpenAIOpenAI's own search index, built by OAI-SearchBot, combined with third-party search providers.OAI-SearchBotChatGPT-UserGPTBotOAI-AdsBotChatGPT Agentchatgpt.comchat.openai.comYes
ChatGPT AtlasOpenAIUses ChatGPT's search stack (OAI-SearchBot index plus partners); Atlas has no separate index.OAI-SearchBotChatGPT-Userchatgpt.comInherits ChatGPT plan rules
Google AI OverviewsGoogleGoogle Search's own index, crawled by Googlebot.Googlebotgoogle.comYes
Google AI ModeGoogleGoogle Search's index, queried with query fan-out (many sub-searches per prompt).Googlebotgoogle.comYes
GeminiGoogleGrounding with Google Search.GooglebotGoogle-ExtendedGoogle-Agentgemini.google.combard.google.comNo ads in the Gemini app as of October 2026
PerplexityPerplexity AIIts own index, served on Vespa, covering hundreds of billions of pages.PerplexityBotPerplexity-Userperplexity.aiwww.perplexity.aiSponsored follow-up questions were tested from late 2024
CometPerplexity AIPerplexity's search index; Comet adds an in-browser assistant.Perplexity-Userperplexity.aiNo
ClaudeAnthropicWeb search launched in March 2025 and was found to use Brave Search results.ClaudeBotClaude-SearchBotClaude-Userclaude.aiNo
Microsoft CopilotMicrosoftBing's index, built by Bingbot.Bingbotcopilot.microsoft.combing.comYes
Meta AIMetaMeta's own web index built by meta-webindexer, plus partner search results.meta-webindexerMeta-ExternalAgentMeta-ExternalFetchermeta.aiMeta has said it plans to monetise Meta AI; no ads in answers as of October 2026
GrokxAITwo tools: live web search and X (Twitter) search.GrokBotxAI-Grokgrok.comx.comAds have been discussed for Grok answers on X; none confirmed in Grok responses as of October 2026
DeepSeekDeepSeekDeepSeek opened its in-house search engine in October 2026 and offers web search through its API.DeepSeekBotchat.deepseek.comdeepseek.comNo
Le ChatMistral AIPartner search (reported as Brave Search) plus Mistral's own MistralAI-Index crawler.MistralAI-IndexMistralAI-UserMistralAI-Trainingchat.mistral.aiNo
You.comYou.comYou.com's own web and news indexes, built by YouBot; also sold through its search APIs.YouBotyou.comNo
Brave Search (Ask Brave)BraveBrave's independent index (35+ billion pages).Bravebotsearch.brave.comBrave Search shows private search ads; AI answers are unsponsored
Kagi AssistantKagiKagi Search: its own Teclis and TinyGem indexes plus licensed results from other engines.Kagibotkagi-fetcherkagi.comNo
DuckDuckGo (Duck.ai & Search Assist)DuckDuckGoDuckAssistBot crawls pages in real time for Search Assist answers.DuckAssistBotduckduckgo.comduck.aiPrivate search ads on result pages; none inside AI answers
Amazon Rufus (retired)AmazonAmazon catalogue, reviews and Q&A, plus web content fetched by Amazonbot and Amzn-SearchBot.AmazonbotAmzn-SearchBotAmzn-UserSponsored product suggestions appeared inside Rufus answers

Can you opt out? Share of top sites blocking each engine's crawlers

ChatGPT: Answer / search crawler 7.6%, Training crawler 17.9%; Claude: Answer / search crawler 7.4%, Training crawler 16.9%; Perplexity: Answer / search crawler 11.1%, Training crawler n/a; Gemini: Answer / search crawler 15.5%, Training crawler n/a; AI Overviews: Answer / search crawler 2.4%, Training crawler n/aAnswer / search crawlerTraining crawlerChatGPT7.6%17.9%Claude7.4%16.9%Perplexity11.1%noneGemini15.5%noneAI Overviews2.4%none
AI Overviews and AI Mode can only be refused by blocking Googlebot, which also removes a site from Google Search. Source: robots.txt study of the top 10,000 sites

More in this reference

  1. Who blocks AI crawlers? A study of the top 10,000 sites22.2% block at least one AI bot. Original data.
  2. AI crawler directory197 bots and user agents, with user-agent strings and IP lists.
  3. AI referrer domains and GA4 regexTrack ChatGPT, Perplexity, Gemini and Copilot traffic.
  4. AI assistant user numbersHow many people use ChatGPT, Gemini, AI Overviews and AI Mode.

About this dataset

What it covers

Eighteen AI answer engines, assistants, browsers and search features, with where each gets its web data, which crawlers feed it, how its traffic appears in analytics, whether it sells ads and how it shows sources.

How it was built

Each profile is written by hand from the operator's own documentation where it exists (crawler pages, help centres, launch posts), supplemented by technical reporting. Every claim about a search index links to its source on the engine's page.

Limitations

AI companies rarely disclose their search providers. Index relationships described as “reported” come from third-party testing or reporting, not official confirmation, and can change without notice.

Columns in ai-engines.csv

slug / name / operator / kind
Engine identity and type
launched
Product launch date
searchIndex
Where the engine's web answers come from
crawlers
Crawler and fetcher tokens it uses
referrers
Referrer hostnames its clicks carry
utm
UTM parameters it appends, if any
ads
Whether and how it shows ads
docs / sources
Official documentation and supporting sources

Questions

Where does ChatGPT get its information?

From two places: its training data, and live web search. For search, ChatGPT uses OpenAI's own index, built by the OAI-SearchBot crawler, plus results from third-party search providers. Bing historically supplied results, and in 2025 reporting showed Google results reaching ChatGPT via SerpApi. When a user shares a link or asks it to read a page, the ChatGPT-User agent fetches it directly.

Does ChatGPT use Bing?

Partly. Bing powered ChatGPT's browsing from 2023, and Bing results still feed its search. OpenAI now also runs its own crawler, OAI-SearchBot, and its own index. To appear in ChatGPT search, a site must allow OAI-SearchBot in robots.txt; being in Bing is not enough.

What search engine does Claude use?

When Claude added web search in March 2025, its results matched Brave Search, and Brave was listed as a subprocessor. Anthropic also operates Claude-SearchBot to index pages for search quality, and Claude-User to fetch pages on request.

What search engine does Perplexity use?

Its own. PerplexityBot builds an index covering hundreds of billions of pages, served on Vespa, which Perplexity also sells as a Search API. Perplexity-User fetches pages live for user queries and, per Perplexity's docs, generally ignores robots.txt.

Can I block AI Overviews without leaving Google Search?

Not with robots.txt. AI Overviews and AI Mode are built from Google's Search index, so the only crawler-level opt-out is blocking Googlebot, which also removes you from Search. Google-Extended only controls Gemini training and grounding. Page-level nosnippet and max-snippet directives limit what can be quoted.

How do I see AI traffic in Google Analytics 4?

Create a custom channel group or exploration filter on session source matching the AI referrer domains (chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com, claude.ai and others). ChatGPT also tags citation links with utm_source=chatgpt.com. A ready-made regex is on the AI referrers page.

Analysis using this data

  1. The opt-out asymmetry: you can block the small AI engines, not the big oneThe AI surfaces with the most users are the ones publishers can't refuse and can't measure.

How to cite this data

This dataset is free to use, including commercially, under Creative Commons Attribution 4.0. The one rule is attribution:

  1. Credit the source as “Kuldeep Paul, kuldeep.cc” wherever you use a figure, chart or table from it.
  2. Link to this page (https://kuldeep.cc/data/ai-search), not to the download file, so readers can check the source of every number.
  3. Include the date. These figures change; say “as of Oct 29, 2026” or link to the version you used.
  4. Mark changes. If you filter, recalculate or chart the data yourself, say so (“analysis based on data from…”).
  5. Cite the original source for single figures when you quote one reported number; it is linked in every row.

In running text: according to Kuldeep Paul's AI search engine reference (kuldeep.cc). More in the methodology.

APA

Paul, K. (2026). AI search engine reference [Data set]. kuldeep.cc. https://kuldeep.cc/data/ai-search

MLA

Paul, Kuldeep. "AI search engine reference." kuldeep.cc, 29 October 2026, https://kuldeep.cc/data/ai-search.

Chicago

Paul, Kuldeep. 2026. "AI search engine reference." Dataset. kuldeep.cc. Updated Oct 29, 2026. https://kuldeep.cc/data/ai-search.

BibTeX

@misc{paul2026aisearch, author = {Paul, Kuldeep}, title = {AI search engine reference}, year = {2026}, howpublished = {\url{https://kuldeep.cc/data/ai-search}}, note = {Dataset, CC BY 4.0, updated 2026-10-29} }

HTML link

<a href="https://kuldeep.cc/data/ai-search">AI search engine reference</a> (data: Kuldeep Paul, kuldeep.cc)