AI crawler directory
The bots AI companies send to your site: what each one is for, who runs it, whether it obeys robots.txt, and how many of the top 10,000 websites block it.
Most-blocked AI crawlers
How AI bots are used: documented crawlers by purpose
Documented AI crawlers (45)
| User agent | Operator | Purpose | Used for | Obeys robots.txt | Top sites blocking |
|---|---|---|---|---|---|
| CCBot | Common Crawl | Model training | Common Crawl open dataset | Yes | 19% |
| Bytespider | ByteDance | Model training | ByteDance LLMs (Doubao) | No | 18.7% |
| GPTBot | OpenAI | Model training | OpenAI foundation models | Yes | 17.9% |
| ClaudeBot | Anthropic | Model training | Anthropic model development | Yes | 16.9% |
| Google-Extended | Model training | Gemini training and grounding | Yes | 15.5% | |
| Meta-ExternalAgent | Meta | Model training | Meta AI models and products | Yes | 15.3% |
| Diffbot | Diffbot | Mixed / general | Knowledge graph and AI data | Partial | 15.1% |
| Amazonbot | Amazon | Mixed / general | Alexa answers and Amazon AI services | Yes | 14.9% |
| anthropic-ai | Anthropic | Model training | Legacy token | Unclear | 14.8% |
| cohere-ai | Cohere | User-triggered fetch | Cohere products | Unclear | 14.7% |
| Applebot-Extended | Apple | Model training | Apple Intelligence and foundation models | Yes | 14.7% |
| PetalBot | Huawei | AI search index | Petal Search and Huawei assistant | Yes | 13.6% |
| Claude-Web | Anthropic | User-triggered fetch | Legacy token | Unclear | 13% |
| PerplexityBot | Perplexity AI | AI search index | Perplexity search index | Yes | 11.1% |
| Google-CloudVertexBot | AI agent | Vertex AI Agents | Yes | 10.3% | |
| YouBot | You.com | AI search index | You.com search and APIs | Yes | 10.3% |
| FacebookBot | Meta | Model training | Meta speech and language models | Yes | 10% |
| ChatGPT-User | OpenAI | User-triggered fetch | ChatGPT, Custom GPTs, GPT Actions | Partial | 9.6% |
| Timpibot | Timpi | Model training | Timpi search and LLM data | Unclear | 9.6% |
| GoogleOther | Mixed / general | Google internal research and development | Yes | 8.9% | |
| Meta-ExternalFetcher | Meta | User-triggered fetch | Meta AI | No | 8.4% |
| AI2Bot | Ai2 | Model training | Open language models (OLMo) | Yes | 8.3% |
| Perplexity-User | Perplexity AI | User-triggered fetch | Perplexity, Comet | No | 8.1% |
| cohere-training-data-crawler | Cohere | Model training | Cohere LLMs | Unclear | 8.1% |
| DuckAssistBot | DuckDuckGo | AI search index | DuckDuckGo AI-assisted answers | Yes | 8% |
| Claude-User | Anthropic | User-triggered fetch | Claude | Yes | 7.8% |
| KimiBot | Moonshot AI | Model training | Kimi models | Yes | 7.6% |
| OAI-SearchBot | OpenAI | AI search index | ChatGPT search | Yes | 7.6% |
| MistralAI-Training | Mistral AI | Model training | Mistral models | Yes | 7.6% |
| MistralAI-User | Mistral AI | User-triggered fetch | Le Chat | Yes | 7.5% |
| Claude-SearchBot | Anthropic | AI search index | Claude web search | Yes | 7.4% |
| DeepSeekBot | DeepSeek | Mixed / general | DeepSeek models and search | No | 6.8% |
| meta-webindexer | Meta | AI search index | Meta AI search | Yes | 6.6% |
| Amzn-User | Amazon | User-triggered fetch | Alexa and Amazon AI assistants | Partial | 5.7% |
| Amzn-SearchBot | Amazon | AI search index | Alexa and Amazon search experiences | Yes | 5.6% |
| Bravebot | Brave | AI search index | Brave Search index | Yes | 5.3% |
| GrokBot | xAI | AI search index | Grok | Unclear | 5.1% |
| MistralAI-Index | Mistral AI | AI search index | Le Chat search | Yes | 4.8% |
| Applebot | Apple | AI search index | Siri, Spotlight, Safari suggestions | Yes | 4.7% |
| Google-Agent | AI agent | Google AI agents (e.g. Project Mariner) | Partial | 4.7% | |
| ChatGPT Agent | OpenAI | AI agent | ChatGPT agent mode | Unclear | 4.6% |
| Kimi-SearchBot | Moonshot AI | AI search index | Kimi search | Yes | 4.5% |
| OAI-AdsBot | OpenAI | Ad verification | ChatGPT Ads | Not applicable | 4.2% |
| Bingbot | Microsoft | AI search index | Bing, Microsoft Copilot | Yes | 2.8% |
| Googlebot | AI search index | Google Search, AI Overviews, AI Mode | Yes | 2.4% |
Full registry (152 more)
Less common AI crawlers, scrapers, data providers and coding agents. Descriptions come from the community-maintained ai.robots.txt project (MIT licence), cross-referenced against the robots.txt study.
| User agent | Operator | Function | Obeys robots.txt | Top sites blocking |
|---|---|---|---|---|
| AddSearchBot | Unknown | AI Search Crawlers | Unclear | 5.2% |
| AgentDataBot | Unknown | AI Data Providers | Unclear | 4.3% |
| AgentTimes | The Agent Times | Data Scraper from RSS Feeds. | Unclear | 4.2% |
| AGI Agent | AGI, Inc | AI Agents | Unclear | 4.1% |
| AI2Bot-DeepResearchEval | Ai2, a non-profit AI research institute | AI Assistants | Unclear | 5.2% |
| Ai2Bot-Dolma | Ai2 | Content is used to train open language models. | Yes | 6.9% |
| aiHitBot | aiHit | A massive, artificial intelligence/machine learning, automated system. | Yes | 5.7% |
| AIWebIndex | Lyrenth | AI Search Crawlers | Yes | 4.4% |
| amazon-kendra | Amazon | Collects data for AI natural language search | Yes | 5.1% |
| amazon-QBusiness | Unknown | AI Assistants | Unclear | 4.7% |
| AmazonBuyForMe | Amazon | AI Agents | Unclear | 5% |
| Andibot | Andi | Search engine using generative AI, AI Search Assistant | Unclear | 5% |
| Anomura | Direqt | Collects data for AI search | Yes | 4.9% |
| ApifyBot | Unknown | AI Data Providers | Unclear | 4.9% |
| ApifyWebsiteContentCrawler | Unknown | AI Data Providers | Unclear | 4.9% |
| Aranet-SearchBot | Unknown | Undocumented AI Agents | Unclear | 4.3% |
| atlassian-bot | Atlassian | AI search, assistants and agents | Yes | 8.1% |
| Awario | Awario | AI Data Scrapers | Unclear | 4.5% |
| AzureAI-SearchBot | Unknown | AI Search Crawlers | Unclear | 4.5% |
| bedrockbot | Amazon | Data scraping for custom AI applications. | Yes | 5% |
| bigsur.ai | Big Sur AI that fetches website content to enable AI-powered web agents, sales assistants, and content marketing solutions for busi… | AI Assistants | Unclear | 4.8% |
| BixelBot | Unknown | AI Data Providers | Unclear | 4.3% |
| Brightbot | Unknown | AI Data Providers | Unclear | 5.5% |
| Brightbot 1.0 | https://brightdata.com/brightbot | LLM/AI training. | Unclear | 5.6% |
| BuddyBot | BuddyBotLearning | AI Learning Companion | Unclear | 4.7% |
| CartAI Agentic Commerce | CartAI that accesses online stores to support user-directed shopping and checkout workflows | AI Agents | Unclear | 4.1% |
| Channel3Bot | Unknown | AI Search Crawlers | Unclear | 5% |
| ChatGLM-Spider | Unknown | AI Data Scrapers | Unclear | 5.9% |
| Claude-Code | Unknown | AI Coding Agents | Unclear | 4.4% |
| CledaraBot | Cledara that visits SaaS providers to complete customer-approved administrative tasks, including invoice collection and user managem… | AI Agents | Unclear | 4.1% |
| Cloudflare-AutoRAG | Cloudflare | Collects data for AI search | Yes | 4.9% |
| CloudflareBrowserRenderingCrawler | Cloudflare that returns rendered website content for research, monitoring, and AI data workflows | AI Data Providers | Unclear | 5% |
| CloudVertexBot | Unknown | AI Data Scrapers | Unclear | 5% |
| Code | Unknown | AI Coding Agents | Unclear | 4.4% |
| Codex | Unknown | AI Coding Agents | Unclear | 4.1% |
| Cotoyogi | ROIS | AI LLM Scraper. | Yes | 9.3% |
| CragCrawler | CragSoftware, a Brazil-based software company specializing in data engineering and AI web scraping services | AI Data Providers | Unclear | 4.4% |
| Crawl4AI | Unknown | Undocumented AI Agents | Unclear | 5.2% |
| Crawlspace | Crawlspace | AI Data Providers | Yes | 5.1% |
| Cursor | Unknown | AI Coding Agents | Unclear | 4.4% |
| Datenbank Crawler | Datenbank | AI Data Scrapers | Unclear | 5.5% |
| Devin | Devin AI | AI Coding Agents | Yes | 5.3% |
| Diffbot-User | Diffbot | AI Assistants | Yes | 4.7% |
| Dot | Unknown | AI Agents | Unclear | 4.1% |
| DoubaoBot | ByteDance | AI crawler for ByteDance's Doubao AI assistant. | Unclear | 4.4% |
| Echobot Bot | Echobox | AI Data Scrapers | Unclear | 5.2% |
| EchoboxBot | Echobox | Data collection to support AI-powered products. | Unclear | 5.3% |
| ElevenlabsBot | Unknown | AI Assistants | Unclear | 4.1% |
| ERNIEBot | Baidu | Collects public web content for Baidu's ERNIE large language models. | Unclear | 4.7% |
| ExaBot | Unknown | AI Data Providers | Unclear | 5.9% |
| ExaSearchBot | Exa | AI Search Crawlers | Unclear | 4.9% |
| ExaUser | Unknown | AI Assistants | Unclear | 4.1% |
| facebookexternalhit | Meta/Facebook | Ostensibly only for sharing, but likely used as an AI crawler as well | No | 3.3% |
| Factset_spyderbot | Factset | AI model training. | Unclear | 5.7% |
| FirecrawlAgent | Firecrawl that extracts web content and converts it into structured data for use in LLM and AI applications | AI Data Providers | Yes | 6.5% |
| FriendlyCrawler | Unknown | We are using the data from the crawler to build datasets for machine learning experiments. | Yes | 7.4% |
| GeistHaus-PageFetcher | GeistHaus, a company developing AI systems for therapy and psychological assessment | AI Assistants | Unclear | 4.4% |
| Gemini-Deep-Research | Unknown | AI Assistants | Unclear | 5.3% |
| Google-Firebase | Used as part of AI apps developed by users of Google's Firebase AI products. | Unclear | 4.7% | |
| Google-Gemini-CLI | Unknown | AI Coding Agents | Unclear | 4.2% |
| Google-NotebookLM | Unknown | AI Assistants | Unclear | 5.3% |
| GoogleAgent-Mariner | AI Agents | Unclear | 5.1% | |
| GoogleAgent-URLContext | Google that retrieves web content on behalf of Gemini API users | AI Assistants | Unclear | 4.2% |
| GoogleOther-Image | Scrapes data. | Yes | 4.8% | |
| GoogleOther-Video | Scrapes data. | Yes | 4.8% | |
| Grok Bot | Unknown | AI Agents | Unclear | 4.1% |
| GrokAppAndroid | xAI | AI Search Crawlers | Unclear | 4.1% |
| HarkBot | Unknown | AI Agents | Unclear | 4.1% |
| HenkBot | Unknown | AI Data Providers | Unclear | 4.4% |
| Hunyuan | Tencent | Hunyuan is a training crawler operated by Tencent. | Unclear | 4.3% |
| iAskBot | Unknown | Undocumented AI Agents | Unclear | 5.5% |
| iaskspider | Unknown | Undocumented AI Agents | Unclear | 6.4% |
| ICC-Crawler | NICT | Scrapes data to train and support AI technologies. | Yes | 10.1% |
| ImagesiftBot | ImageSift | ImageSiftBot is a web crawler that scrapes the internet for publicly available images to support their suite of web intelligence products | Yes | 9.3% |
| imageSpider | Unknown | AI Data Scrapers | Unclear | 5.6% |
| img2dataset | img2dataset | Scrapes images for use in LLMs. | Unclear | 7.1% |
| Instinct | Unknown | AI Agents | Unclear | 4.1% |
| ISSCyberRiskCrawler | ISS-Corporate | Scrapes data to train machine learning models. | No | 5.9% |
| kagi-fetcher | Kagi that fetches web content to answer user queries through Kagi AI, their suite of AI-powered tools including Assistant, Res… | AI Assistants | Unclear | 4.5% |
| Kangaroo Bot | Unknown | AI Data Scrapers | Unclear | 6.9% |
| KeenableBot | Keenable | AI Search Crawlers | Unclear | 4.4% |
| Kimi-Agent | Moonshot AI that browses websites while carrying out user-directed research and other tasks | AI Agents | Unclear | 4.3% |
| Kimi-User | Moonshot AI that fetches web content on behalf of users interacting with Kimi | AI Assistants | Unclear | 4.8% |
| KlaviyoAIBot | Klaviyo | AI Assistants | Yes | 5% |
| KunatoCrawler | Unknown | Undocumented AI Agents | Unclear | 5.2% |
| laion-huggingface-processor | Unknown | AI Data Scrapers | Unclear | 5.3% |
| LAIONDownloader | Large-scale Artificial Intelligence Open Network | AI tools and models for machine learning research. | No | 4.6% |
| LCC | Unknown | AI Data Scrapers | Unclear | 4.8% |
| Lightpanda | Anyone who downloads the Lightpanda client. Possibly being used by a Grok-adjacent organization's botnet. | AI Data Scrapers | At the discretion of Lightpanda users. | 4.2% |
| LinerBot | Unknown | AI Assistants | Unclear | 5.4% |
| Linguee Bot | Linguee | AI powered translation service | No | 4.7% |
| LinkupBot | Unknown | AI Search Crawlers | Unclear | 5.4% |
| Manus-User | Butterfly Effect, a company based in China | AI Agents | Unclear | 5% |
| MoonshotBot | Kimi | MoonshotBot is a training crawler. | Unclear | 4.3% |
| Mozilla-Tabstack | Mozilla that performs programmatic, AI-driven interactions with web content through Tabstack | AI Data Providers | Yes | 4.3% |
| Muse | Unknown | AI Agents | Unclear | 4.1% |
| MyCentralAIScraperBot | Unknown | AI data scraper | Unclear | 4.9% |
| NagetBot | Naget Inc (founded by Chris Samarinas, headquarter in Amherst, Massachusetts) | AI data scraper | Unclear | 4.3% |
| nbot | NBot | AI Search Crawlers | Unclear | 4.1% |
| netEstate Imprint Crawler | netEstate | AI Data Scrapers | Unclear | 5.5% |
| newsai | Unknown | AI data scraper | Unclear | 4.2% |
| NotebookLM | Unknown | AI Assistants | Unclear | 4.2% |
| NovaAct | Unknown | AI Agents | Unclear | 5% |
| omgili | Webz.io | Data is sold. | Yes | 14.5% |
| omgilibot | Webz.io | Data is sold. | Yes | 10.2% |
| OpenAI | OpenAI | — | Yes | 4.5% |
| opencode | Unknown | AI Coding Agents | Unclear | 4.4% |
| Operator | Unknown | AI Agents | Unclear | 4.8% |
| PanguBot | the Chinese company Huawei | AI Data Scrapers | Unclear | 7.4% |
| Panscient | Panscient | Data collection and analysis using machine learning and AI. | Yes | 5.3% |
| panscient.com | Panscient | Data collection and analysis using machine learning and AI. | Yes | 5.2% |
| Payhawk Invoice Fetching Agent | Payhawk that retrieves invoices from supplier websites for users and attaches them to expense records | AI Agents | Unclear | 4.1% |
| PhindBot | phind | AI Assistants | Unclear | 5.4% |
| Poggio-Citations | Poggio, a company that provides AI sales enablement tools for creating tailored narratives, business cases, and account plan… | AI Assistants | Unclear | 5% |
| Poseidon Research Crawler | Poseidon Research | AI research crawler | Unclear | 4.9% |
| qodercli | Qoder that helps developers build, debug, and modify software from the terminal | AI Coding Agents | Unclear | 4.3% |
| QualifiedBot | Qualified | AI Assistants | Unclear | 8.6% |
| Querit-SearchBot | Querit that indexes web content for their search API service, which is designed to provide real-time search results for larg… | AI Data Providers | Unclear | 4.4% |
| QueritBot | Querit, a company providing a search API for large language model integration | AI Data Providers | Unclear | 4.4% |
| QuillBot | Quillbot | Company offers AI detection, writing tools and other services. | Unclear | 4.8% |
| quillbot.com | Quillbot | Company offers AI detection, writing tools and other services. | Unclear | 5% |
| QwenBot | Alibaba | Collects public web content for Alibaba's Qwen (Tongyi Qianwen) large language models. | Unclear | 4.4% |
| Reflectionbot | Reflection | Undocumented AI Agents | Unclear | 4.5% |
| SBIntuitionsBot | SB Intuitions | Uses data gathered in AI development and information analysis. | Yes | 5.8% |
| Scrapy | Zyte | Scrapes data for a variety of uses including training AI. | Unclear | 8.8% |
| SemrushBot-OCOB | Semrush | Crawls your site for ContentShake AI tool. | Yes | 5.2% |
| SemrushBot-SWA | Semrush | Checks URLs on your site for SEO Writing Assistant. | Yes | 8.7% |
| Shap-User | Unknown | AI Assistants | Unclear | 5.1% |
| ShapBot | Parallel | AI Data Providers | Yes | 6.2% |
| Sidetrade indexer bot | Sidetrade | Extracts data for a variety of uses including training AI. | Unclear | 5.5% |
| Spider | Unknown | AI Data Scrapers | Unclear | 4.9% |
| TavilyBot | Unknown | AI Data Providers | Unclear | 5.2% |
| Terra Cotta | Unknown | AI Data Providers | Unclear | 4.3% |
| TerraCotta | Ceramic AI | AI Data Providers | Yes | 5.2% |
| Thinkbot | Thinkbot | Insights on AI integration and automation. | No | 5.2% |
| TikTokSpider | ByteDance | LLM training. | Unclear | 5.4% |
| TongyiBot | Alibaba that fetches web content for the Tongyi Qianwen assistant and related Qwen-generated answers | AI Assistants | Unclear | 4.4% |
| Trae | Unknown | AI Coding Agents | Unclear | 4.4% |
| TwinAgent | Twin, a platform that creates automated workers to perform tasks by integrating with APIs and controlling web applications through browser automa… | AI Agents | Unclear | 5% |
| UseAI | Unknown | AI Assistants | Unclear | 4.4% |
| VelenPublicWebCrawler | Velen Crawler | Scrapes data for business data sets and machine learning models. | Yes | 7.3% |
| WARDBot | WEBSPARK | AI Data Scrapers | Unclear | 7.9% |
| Webzio-Extended | Unknown | AI Data Scrapers | Unclear | 8.6% |
| wpbot | QuantumCloud | Live chat support and lead generation. | Unclear | 5.2% |
| WRTNBot | Unknown | Undocumented AI Agents | Unclear | 5.2% |
| xAI-Grok | xAI | xAI-Grok is an xAI crawler token associated with Grok content collection. | Unclear | 4.8% |
| YaK | Meltwater | According to the Meltwater Consumer Intelligence page 'By applying AI, data science, and market research expertise to a live feed of global data sources, we transform unstructured data into actionable insights allowing better decision-making'. | Unclear | 5.2% |
| YandexAdditional | Yandex | Scrapes/analyzes data for the YandexGPT LLM. | Yes | 5% |
| YandexAdditionalBot | Yandex | Scrapes/analyzes data for the YandexGPT LLM. | Yes | 5.2% |
| YiBot | 01.ai, an AI company founded by Kai-Fu Lee. | This bot crawls the web to gather data and content for indexation and AI model training. | Unclear | 4.3% |
| YiyanBot | Baidu that fetches web content for the yiyan | AI Assistants | Unclear | 4.4% |
| ZanistaBot | Unknown | AI Search Crawlers | Unclear | 5% |
About this dataset
What it covers
Every AI crawler, scraper, fetcher and agent user agent I could identify: 45 documented bots with full profiles (purpose, user-agent string, IP list, documentation) and 152 more from the community registry, each with the share of top sites that block it.
How it was built
Documented bots are written from each operator's own crawler documentation. The wider registry comes from the MIT-licensed ai.robots.txt project. Block rates come from my robots.txt study of the top 10,000 domains.
Limitations
User-agent strings can be spoofed; verify with published IP lists or reverse DNS. Undocumented bots are described as their operators or the community describe them, and operators rename bots without notice.
Columns in ai-crawlers.csv
- token
- The name used in robots.txt
- operator
- Company or project running it
- purpose
- training, search, user, agent, ads or mixed
- product
- What the collected content is used for
- respectsRobots
- Whether it obeys robots.txt (Yes, No, Partial, Unclear)
- userAgent
- Full user-agent string, where published
- ipList
- URL of the operator's published IP ranges
- docs
- Official documentation
- source
- Documentation-based profile or ai.robots.txt registry
Questions
What is the difference between GPTBot, OAI-SearchBot and ChatGPT-User?
GPTBot collects content that may be used to train OpenAI's models. OAI-SearchBot builds the index that ChatGPT search cites. ChatGPT-User fetches a page only when a user asks for it. Each is controlled separately in robots.txt, so you can block training while staying visible in ChatGPT search.
Should I block AI crawlers?
Block training crawlers (GPTBot, ClaudeBot, Google-Extended, CCBot, Applebot-Extended, Meta-ExternalAgent) if you don't want your content used for model training. Allow search and user-triggered agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) if you want to be cited in AI answers. Blocking those removes you from those engines.
Do AI crawlers respect robots.txt?
The documented crawlers from OpenAI, Anthropic, Google, Apple, Amazon, Perplexity (PerplexityBot) and Common Crawl say they do. User-triggered fetchers such as Perplexity-User, Meta-ExternalFetcher and Google-Agent may not, because a person initiated the request. Bytespider and DeepSeekBot are widely reported to ignore it.
How do I verify a crawler is real?
Match the request IP against the operator's published IP list (OpenAI, Perplexity, Google, Bing, Common Crawl and DuckDuckGo publish JSON files) or use reverse DNS. User-agent strings alone are easy to spoof.
Analysis using this data
- Who blocks AI crawlers? What 10,000 robots.txt files revealPublishers block training, not answers; half of all blocklists leak; Cloudflare writes the most common AI policy.
- The opt-out asymmetry: you can block the small AI engines, not the big oneThe AI surfaces with the most users are the ones publishers can't refuse and can't measure.
How to cite this data
This dataset is free to use, including commercially, under Creative Commons Attribution 4.0. The one rule is attribution:
- Credit the source as “Kuldeep Paul, kuldeep.cc” wherever you use a figure, chart or table from it.
- Link to this page (https://kuldeep.cc/data/ai-search/crawlers), not to the download file, so readers can check the source of every number.
- Include the date. These figures change; say “as of Oct 29, 2026” or link to the version you used.
- Mark changes. If you filter, recalculate or chart the data yourself, say so (“analysis based on data from…”).
- Cite the original source for single figures when you quote one reported number; it is linked in every row.
In running text: according to Kuldeep Paul's AI crawler directory (kuldeep.cc). More in the methodology.
Paul, K. (2026). AI crawler directory [Data set]. kuldeep.cc. https://kuldeep.cc/data/ai-search/crawlers
Paul, Kuldeep. "AI crawler directory." kuldeep.cc, 29 October 2026, https://kuldeep.cc/data/ai-search/crawlers.
Paul, Kuldeep. 2026. "AI crawler directory." Dataset. kuldeep.cc. Updated Oct 29, 2026. https://kuldeep.cc/data/ai-search/crawlers.
@misc{paul2026crawlers, author = {Paul, Kuldeep}, title = {AI crawler directory}, year = {2026}, howpublished = {\url{https://kuldeep.cc/data/ai-search/crawlers}}, note = {Dataset, CC BY 4.0, updated 2026-10-29} }
<a href="https://kuldeep.cc/data/ai-search/crawlers">AI crawler directory</a> (data: Kuldeep Paul, kuldeep.cc)