Kuldeep Paul

Who blocks AI crawlers?

I fetched the robots.txt and llms.txt files of the 10,000 most popular websites and checked how each one treats 197 AI crawlers and user agents. Original data, collected Oct 29, 2026.

Compiled by , data aggregatorUpdated 10,000 domains analysedDownload CSV · JSONMethodology

22.2%block at least one AI crawler
17.9%block GPTBot
57.6%of GPTBot blockers still allow ChatGPT search
9.8%publish an llms.txt

Shares of the 5,701 domains that served a valid robots.txt, except llms.txt, which is a share of all 10,000.

Key findings

  1. About one in five big sites blocks AI. 22.2% of top sites with a robots.txt explicitly disallow at least one AI crawler, and 29.6% name one. The largest sites block most: 27.4% of the top 1,000 versus 21.8% of sites ranked 5,001–10,000.
  2. Training data is the target, not visibility. The most-blocked bots are dataset and training crawlers: CCBot (19%), Bytespider (18.7%), GPTBot (17.9%) and ClaudeBot (16.9%).
  3. Most GPTBot blockers want to stay in ChatGPT. 587 sites (57.6% of GPTBot blockers) block training but leave OAI-SearchBot open, the split OpenAI's documentation recommends.
  4. llms.txt is real but early. 983 domains (9.8%) serve one, mostly SaaS, developer and e-commerce brands.

Most-blocked AI crawlers

User agentShare of sites blocking%SitesTop 1kNamed
CCBot19%1,0851211,057
Bytespider18.7%1,067125954
GPTBot17.9%1,0191091,210
ClaudeBot16.9%9661111,065
Google-Extended15.5%882931,007
Meta-ExternalAgent15.3%87198799
Diffbot15.1%862102682
Amazonbot14.9%85085792
anthropic-ai14.8%84182756
cohere-ai14.7%83994710
Applebot-Extended14.7%83790775
omgili14.5%829103633
PetalBot13.6%77691608
Claude-Web13%74388623
PerplexityBot11.1%63388790
Google-CloudVertexBot10.3%58672400
YouBot10.3%58585443
omgilibot10.2%58388386
ICC-Crawler10.1%57874353
FacebookBot10%57078438
ChatGPT-User9.6%54973729
Timpibot9.6%54885340
ImagesiftBot9.3%53391329
Cotoyogi9.3%53268303
GoogleOther8.9%50967365
Scrapy8.8%50287290
SemrushBot-SWA8.7%49861276
Webzio-Extended8.6%49180277
QualifiedBot8.6%48863267
Meta-ExternalFetcher8.4%48170314
AI2Bot8.3%47579279
Perplexity-User8.1%46478397
cohere-training-data-crawler8.1%46075251
atlassian-bot8.1%45958238
DuckAssistBot8%45468322
WARDBot7.9%45149220
Claude-User7.8%44272400
KimiBot7.6%43643205
OAI-SearchBot7.6%43361621
MistralAI-Training7.6%43243202

“Blocking” means the group that applies to the bot (its own, or * if it isn't named) disallows /. “Named” counts sites that mention the bot at all, including to allow it.

By popularity

RankDomainsWith robots.txtBlock any AIllms.txt
Top 1,0001,00050727.4%11.6%
1,001–5,0004,0002,21521.6%10.2%
5,001–10,0005,0002,97921.8%9.2%

Deeper findings

Training crawlers are blocked about twice as often as answer crawlers

OpenAI: Training 17.9%, Search / index 7.6%, User-triggered 9.6%; Anthropic: Training 16.9%, Search / index 7.4%, User-triggered 7.8%; Perplexity: Training n/a, Search / index 11.1%, User-triggered 8.1%; Meta: Training 15.3%, Search / index 6.6%, User-triggered 8.4%; Mistral: Training 7.6%, Search / index 4.8%, User-triggered 7.5%TrainingSearch / indexUser-triggeredOpenAI17.9%7.6%9.6%Anthropic16.9%7.4%7.8%Perplexitynone11.1%8.1%Meta15.3%6.6%8.4%Mistral7.6%4.8%7.5%
Share of sites blocking each operator's bots. Publishers refuse to be training data far more often than they refuse to be cited.

Of 810 sites blocking GPTBot, share leaving each other training crawler open

cohere-training-data-crawler: 79.5%; MistralAI-Training: 75.3%; Amazonbot: 37%; Applebot-Extended: 35.1%; Meta-ExternalAgent: 34.1%; Google-Extended: 27.3%; Bytespider: 23.3%; ClaudeBot: 18.8%; CCBot: 12.7%cohere-training-data-crawler79.5%MistralAI-Training75.3%Amazonbot37%Applebot-Extended35.1%Meta-ExternalAgent34.1%Google-Extended27.3%Bytespider23.3%ClaudeBot18.8%CCBot12.7%
Excludes sites that block every bot with a wildcard rule.

Share of sites blocking at least one AI crawler, by top-level domain

.fr (n=74): 45.9%; .de (n=134): 29.9%; .com (n=2919): 23.6%; .jp (n=104): 23.1%; .net (n=253): 22.1%; .org (n=306): 21.6%; .br (n=65): 16.9%; .ru (n=156): 9.6%; .gov (n=78): 7.7%; .io (n=96): 7.3%; .edu (n=109): 5.5%.fr (n=74)45.9%.de (n=134)29.9%.com (n=2919)23.6%.jp (n=104)23.1%.net (n=253)22.1%.org (n=306)21.6%.br (n=65)16.9%.ru (n=156)9.6%.gov (n=78)7.7%.io (n=96)7.3%.edu (n=109)5.5%
Top-level domains with at least 60 sites in the sample.

How 5,701 robots.txt files treat AI

no AI rules: 3,864 · 67.8%; blocks training and AI search/agents: 594 · 10.4%; blocks training only: 534 · 9.4%; names AI bots, blocks none: 466 · 8.2%; blocks every bot (*): 232 · 4.1%; blocks search/agents only: 11 · 0.2%no AI rules3,864 · 67.8%blocks training and AI search/agents594 · 10.4%blocks training only534 · 9.4%names AI bots, blocks none466 · 8.2%blocks every bot (*)232 · 4.1%blocks search/agents only11 · 0.2%

Most AI blocklists are copied

46.2% of AI blocklists with five or more entries are shared, character for character, by at least three sites. The most common one, a 23-bot list on 152 sites, is Cloudflare's managed robots.txt: the web's most widely deployed AI policy is a CDN default. 336 sites (5.9%) publish Cloudflare-style Content-Signal lines, most often search=yes and ai-train=no.

Many rules target bots that no longer exist

66.2% of AI blocklists with three or more entries include a legacy Anthropic token (anthropic-ai or Claude-Web). 51 sites block a legacy token but leave ClaudeBot, Anthropic's current crawler, open, including espn.com, marca.com, ovh.com, e621.net, nikkansports.com. 7 sites, including chatgpt.com and academia.edu, write Perplexity‑User with a non-breaking hyphen (U+2011), so the rule never matches.

llms.txt sites rarely block AI

Only 4.2% of sites that publish an llms.txt block GPTBot, against 17.9% of all sites. The web is splitting into publishers who build walls and product companies who lay out welcome mats.

Google-Extended doesn't do what most sites think

667 sites block Google-Extended; 1 of them also blocks Googlebot. Google-Extended controls Gemini training and grounding, not AI Overviews or AI Mode, so almost all of those sites still appear in Google's AI answers.

AI policies of the top 1,000 sites

240 of 240
Top-1,000 domains that block at least one AI crawler or publish an llms.txt.
RankDomainBlocksExplicitly allowsllms.txt
2cloudflare.com—GPTBot, ChatGPT-User, anthropic-ai, Claude-Web, PerplexityBot, Google-Extended, CCBot, cohere-aiYes
4facebook.comGPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Brightbot, Scrapy, YaK——
7fbcdn.netGPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Brightbot, Scrapy, YaK——
14instagram.comGPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Brightbot, Scrapy, YaK——
30amazon.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
33github.comBytespiderGPTBot, OAI-SearchBot, ClaudeBot, anthropic-ai, PerplexityBotYes
39fastly.net——Yes
42whatsapp.netGPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Brightbot, Scrapy, YaK——
52pinterest.comYandexAdditional, YandexAdditionalBotFacebookBot, DuckAssistBot, Bravebot—
59wordpress.org—GPTBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, BytespiderYes
61adobe.com——Yes
66whatsapp.comGPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Brightbot, Scrapy, YaK——
70digicert.com—GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-ExtendedYes
72yahoo.comGPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Google-Extended, FacebookBot, CCBot, Bytespider, YouBot, cohere-ai +8 moreOAI-SearchBot, Claude-SearchBot—
76opera.com——Yes
85amazonvideo.comChatGPT-User, Claude-User, PerplexityBot, Perplexity-User, GoogleOther, Google-CloudVertexBot, Meta-ExternalAgent, Meta-ExternalFetcher, meta-webindexer, CCBot, Bytespider, MistralAI-User +72 more——
87vimeo.comFacebookBot, CCBot, Bytespider, Diffbot, ImagesiftBot, omgili, omgilibotGPTBot, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended—
92msn.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther +41 more——
103sentry.io——Yes
112chatgpt.comanthropic-ai, Claude-Web, PerplexityBot, Google-Extended, FacebookBot, CCBot, Bytespider, img2dataset, omgili, omgilibot——
118intuit.com——Yes
120samsung.com—GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Google-ExtendedYes
129t-online.deGPTBot, ClaudeBot, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, CCBot, BytespiderGoogle-Extended—
136dropbox.com——Yes
149webex.com——Yes
157snapchat.comGPTBot, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, GoogleOther, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot, Amazonbot, CCBot +28 moreOAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, meta-webindexer, DuckAssistBot—
160tumblr.comClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Amazonbot, CCBot, YouBot, ImagesiftBot, omgili, omgilibot——
163nytimes.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot +27 moreAmazonbot—
165gravatar.comGPTBot, Google-Extended, FacebookBot, Amazonbot, CCBot, Bytespider, omgili, omgilibot—Yes
166kaspersky.com—GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, CCBot, MistralAI-User, DuckAssistBot, YouBot, DeepSeekBot, DiffbotYes
168ubuntu.com—GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider, cohere-aiYes
170shopify.com——Yes
174paypal.com——Yes
186unity3d.comBytespider—Yes
191ebay.comGPTBot, OAI-AdsBot, ClaudeBot, anthropic-ai, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot, Bytespider, ChatGLM-SpiderOAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User—
193macromedia.com——Yes
195cnn.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, GoogleOther, Applebot-Extended +52 more——
196theguardian.comClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Amazonbot, Amzn-SearchBot, CCBot +6 more——
198medium.comGPTBot, ClaudeBot, GoogleOther, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Amazonbot, Bytespider——
203stripe.com——Yes
216bbc.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent +22 more——
220forter.comMeta-ExternalAgent, FacebookBot, CCBot, Bytespider, Diffbot, ImagesiftBot, omgili, omgilibotGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Amazonbot, MistralAI-User, cohere-ai, DeepSeekBotYes
224sciencedirect.comGPTBot, ChatGPT-User, ClaudeBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Google-Agent, Applebot-Extended +26 moreOAI-SearchBot, Claude-SearchBot, DuckAssistBot—
236bbc.co.ukGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent +22 more——
240hcaptcha.com——Yes
242cisco.com——Yes
246roku.com——Yes
247amplitude.comBytespiderGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot, cohere-aiYes
250weather.comGoogleOther, Google-CloudVertexBot, FacebookBot, Amazonbot, CCBot, Bytespider, YouBot, Diffbot, FriendlyCrawler, ImagesiftBot, omgili, omgilibot +1 moreChatGPT-UserYes
251sourceforge.net——Yes
261hubspot.com——Yes
263linktr.ee—GPTBot, OAI-SearchBot, ChatGPT-User, ChatGPT Agent, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalFetcher, Amazonbot, CCBot, MistralAI-User, DuckAssistBot, BravebotYes
271who.intCCBot, aiHitBot, ExaBot, Panscient, Scrapy——
276canva.comGPTBot, ClaudeBot, anthropic-ai, Claude-Web, Applebot-Extended, Meta-ExternalAgent, FacebookBot, CCBot, Bytespider, AI2Bot, Diffbot, Timpibot +9 morePerplexityBot, Perplexity-User—
281launchpad.netGPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot +27 more——
283reuters.comApplebot-ExtendedOAI-SearchBot, ChatGPT-User, GoogleOther, meta-webindexer, Amzn-User—
286datadoghq.com——Yes
297nature.comGPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Google-Extended, GoogleOther, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Amazonbot +10 more——
310alibaba.comGPTBot, Google-Extended——
318weibo.comOAI-SearchBot, ChatGPT-User, Perplexity-User, Meta-ExternalAgent, Bytespider, DeepSeekBot——
319pubmatic.com——Yes
323slack.com——Yes
325rubiconproject.com——Yes
326amazon.co.ukGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
328etsy.com——Yes
340bilibili.com——Yes
344trustpilot.comanthropic-ai, Applebot-Extended, Bytespider, YouBot, cohere-ai, Diffbot, Timpibot, ImagesiftBot, Kangaroo Bot, omgili, omgilibot, Webzio-ExtendedGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, meta-webindexer, FacebookBot, CCBot—
345ea.comMeta-ExternalAgent, Bytespider, cohere-ai, cohere-training-data-crawler, AI2Bot, FriendlyCrawler, ICC-Crawler—Yes
346zendesk.com——Yes
347gnu.orgpanscient.com, SemrushBot-OCOB, SemrushBot-SWA——
351branch.io—GPTBot, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBotYes
353dell.com—BytespiderYes
357checkpoint.com—GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, CCBotYes
362nvidia.com——Yes
363espn.comGPTBot, ChatGPT-User, anthropic-ai, Google-Extended, FacebookBot, CCBot, Bytespider, omgili, omgilibot——
366cm.com——Yes
377issuu.comAddSearchBot, SemrushBot-OCOB, YaK——
383washingtonpost.comClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot, Amazonbot, CCBot, Bytespider, cohere-training-data-crawler +4 more——
385name.comMeta-ExternalAgent—Yes
386bloomberg.comGPTBot, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, Amazonbot, CCBot +6 moreOAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User—
387amazon.deGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
389plesk.com——Yes
394paloaltonetworks.com—OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBotYes
403go.comGPTBot, Google-Extended——
406calendly.comCCBot—Yes
411claude.aiGPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended——
412okta.com——Yes
418unsplash.comClaudeBot, Claude-SearchBot, Bytespider, ImagesiftBot, SemrushBot-OCOB, SemrushBot-SWA——
420indeed.comGoogleOther, MyCentralAIScraperBot, omgili, omgilibot, Scrapy, Webzio-ExtendedGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Google-Agent, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, meta-webindexer, FacebookBot, Amazonbot, CCBot, Bytespider, DuckAssistBot, YouBot, cohere-training-data-crawler, DeepSeekBot, GrokBot, AI2Bot, Diffbot, TimpibotYes
421aol.comGPTBot, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, FacebookBot, CCBot, Bytespider, YouBot, cohere-ai, Diffbot, FriendlyCrawler +6 moreChatGPT-User, PerplexityBot—
428taboola.com——Yes
431temu.comBytespiderGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Google-Extended, FacebookBot—
435dailymotion.com——Yes
442dynatrace.com——Yes
443wps.com——Yes
446intel.comICC-CrawlerGPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-ExtendedYes
455bluehost.com——Yes
461yahoo.co.jpGPTBot, Google-Extended——
464pixabay.comGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider——
468selectel.ru——Yes
471businessinsider.comClaudeBot, anthropic-ai, CCBot, Bytespider, Diffbot, Timpibot, Scrapy—Yes
473fandom.comGPTBot, OAI-SearchBot, ClaudeBot, Meta-ExternalAgent, Meta-ExternalFetcher, CCBot, GrokBot, xAI-Grok——
474cnbc.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-Agent +12 more——
482goodreads.comGPTBot, CCBot——
483shein.com——Yes
490amazon.co.jpGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
491quora.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Applebot-Extended, Bytespider, Bravebot, Diffbot, ImagesiftBot +1 moreGoogle-Extended, Amazonbot—
493statista.comBrightbot 1.0GPTBot, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot, Amazonbot, CCBot, Bytespider, DuckAssistBot, YouBot, cohere-ai, cohere-training-data-crawler, AI2Bot, Diffbot, Timpibot—
497aliexpress.comGPTBot, ClaudeBot, Google-Extended, Amazonbot, CCBot, Bytespider——
498scribd.comClaudeBot, Claude-Web, CCBot, Bytespider, AI2Bot, Diffbot, ImagesiftBot, omgili——
509foxnews.com——Yes
512trendmicro.com—OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Meta-ExternalAgent, FacebookBot, Amazonbot, CCBot, Bytespider, DuckAssistBot, cohere-aiYes
515indiatimes.comMeta-ExternalAgent——
525behance.netGPTBot, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, CCBot, Bytespider, cohere-training-data-crawler, AI2Bot +8 more——
534braze.com——Yes
535uol.com.brClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot +28 more——
537cloudns.net——Yes
544newrelic.com——Yes
546capcut.com—GPTBot, OAI-SearchBot, ChatGPT-User, OAI-AdsBot, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Google-Agent, Applebot-Extended, Meta-ExternalFetcher, meta-webindexer, Amzn-SearchBot, Amzn-User, CCBot, DuckAssistBotYes
550slideshare.netClaudeBot, Claude-Web, CCBot, Bytespider, AI2Bot, Diffbot, ImagesiftBot, omgili——
552ft.comClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot, CCBot +5 more——
564tandfonline.comGPTBot——
570wired.comClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, meta-webindexer +19 more——
572tripadvisor.comGPTBot, ClaudeBot, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot, Bytespider, YouBot, cohere-ai, img2dataset +1 moreOAI-SearchBot, ChatGPT-User, PerplexityBot—
573playstation.com——Yes
577shop.app——Yes
578klaviyo.com—GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Agent, Meta-ExternalAgent, DuckAssistBot, YouBotYes
579comcast.com——Yes
582dreamhost.com——Yes
583time.com——Yes
587squarespace.com—GPTBot, ClaudeBot, anthropic-ai, Google-Extended, GoogleOther, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Amazonbot, CCBot, Bytespider, DuckAssistBot, YouBot, cohere-ai, cohere-training-data-crawler, AI2BotYes
591threads.comGPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Brightbot, Scrapy, YaK——
605atlassian.com——Yes
606target.com——Yes
616wix.com——Yes
617usatoday.comGPTBot, OAI-SearchBot, ChatGPT-User, ChatGPT Agent, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended +91 more——
619dailymail.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot +45 moreGoogleOther—
624sagepub.comMeta-ExternalAgent, Amazonbot, CCBot, Bytespider, cohere-ai, Diffbot, ImagesiftBot, omgili, omgilibotGPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended—
626endpoints.newsGPTBot, ClaudeBot, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, CCBot, Bytespider, cohere-ai, Diffbot, omgili, omgilibotGoogle-Extended, GoogleOther—
629telegraph.co.ukGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, GoogleOther, Google-CloudVertexBot +39 moreGoogle-Extended—
632amazon.frGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
633amazon.caGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
634techcrunch.comGPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, Applebot-Extended, FacebookBot, CCBot, Bytespider, cohere-ai, Diffbot +1 more——
635nintendo.comGPTBot, ClaudeBot, Claude-Web, Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, CCBot, Bytespider, cohere-ai, cohere-training-data-crawler, AI2Bot +10 more——
636ikea.comAmazonbot——
637bitrix24.ru—GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-ExtendedYes
643sophos.comAmazonbot, DeepSeekBotGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Amzn-SearchBot, Amzn-User, YouBotYes
644onetrust.com——Yes
655intercom.io——Yes
657meta.comScrapy, YaK——
6612gis.comGoogleOther——
664stackadapt.com——Yes
668iso.orgGPTBot, ChatGPT-User, Google-Extended, FacebookBot, CCBot, omgili, omgilibot——
670optimizely.com——Yes
674elpais.comClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Amazonbot, Amzn-SearchBot, Amzn-User, CCBot, Bytespider, MistralAI-User, DuckAssistBot +4 more——
676amazon.inGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
678primevideo.comChatGPT-User, Claude-User, PerplexityBot, Perplexity-User, GoogleOther, Google-CloudVertexBot, Meta-ExternalAgent, Meta-ExternalFetcher, meta-webindexer, CCBot, Bytespider, MistralAI-User +72 more——
680redhat.com——Yes
681sharethrough.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, GoogleOther, Meta-ExternalAgent, CCBot, Bytespider, YouBot, cohere-ai +4 more——
686surveymonkey.com——Yes
691cambridge.orgChatGPT-User——
698ring.com—GPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, cohere-aiYes
700lemonde.frClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, Google-Extended, GoogleOther, Google-CloudVertexBot, Google-Agent, Applebot-Extended, meta-webindexer, FacebookBot +100 more——
701cbsnews.comGPTBot, panscient.com——
702yelp.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Meta-ExternalAgent +2 more——
705fb.comGPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, Brightbot, Scrapy, YaK——
711bandcamp.comGPTBot, ClaudeBot, Google-Extended, Meta-ExternalAgent, FacebookBot, Amazonbot, CCBot, Bytespider, Diffbot, omgili——
714amazon.esGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
717dailymail.co.ukGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot +45 moreGoogleOther—
730pinimg.comYandexAdditional, YandexAdditionalBotFacebookBot, DuckAssistBot, Bravebot—
735amazon.itGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
737cnet.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Applebot-Extended, Meta-ExternalAgent +99 more——
738quantserve.comGoogleOther——
743asus.com——Yes
746gumgum.comBrightbotGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, GoogleOtherYes
759bild.dePerplexityBot, Perplexity-User, Applebot-Extended, Amazonbot, CCBot, Bytespider, MistralAI-User, DuckAssistBot, YouBot, cohere-ai, cohere-training-data-crawler, AI2Bot +52 more——
761globalsign.comGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBot, BytespiderOAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot—
769airbnb.comGPTBot, ClaudeBot, Applebot-Extended, MistralAI-Training, AI2Bot, ImagesiftBot, Webzio-ExtendedOAI-SearchBot, ChatGPT-User, anthropic-ai, PerplexityBot, Meta-ExternalAgent, cohere-ai—
770mailchimp.com——Yes
777fontawesome.com——Yes
783welt.dePerplexityBot, Perplexity-User, Applebot-Extended, Amazonbot, CCBot, Bytespider, MistralAI-User, DuckAssistBot, YouBot, cohere-ai, cohere-training-data-crawler, AI2Bot +52 more——
786amazon.co.zaGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
789allegro.pl—CCBotYes
793wattpad.comGPTBot——
795zillow.comApplebot-Extended, Meta-ExternalAgent, Amazonbot, YouBot, AI2Bot, AI2Bot-DeepResearchEval, Ai2Bot-Dolma, amazon-kendra, YandexAdditional, YandexAdditionalBot——
796apnews.comGPTBot, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Applebot-Extended, Amazonbot, CCBot, cohere-ai, Timpibot——
800moloco.comBytespiderCCBotYes
805zoho.com——Yes
814wp.plGPTBot, CCBotOAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User—
818prnewswire.com——Yes
821fiverr.com——Yes
823ted.comGPTBot, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, Applebot-Extended, Meta-ExternalAgent, FacebookBot, Amazonbot, CCBot, Bytespider, cohere-ai +14 moreOAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, MistralAI-User, DuckAssistBot, YouBot—
824conviva.comBrightbotGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, GoogleOther—
827people.comClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher +37 moreGPTBot, OAI-SearchBot, ChatGPT-User—
838liftoff.io——Yes
843netlify.app——Yes
850life360.com—GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-ExtendedYes
851academia.eduGPTBot, Google-Extended, Applebot-ExtendedOAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, MistralAI-User, DuckAssistBot, YouBot—
852supabase.co——Yes
862theverge.comChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent +36 moreGPTBot, Amazonbot—
865unpkg.com—GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, CCBotYes
867frontiersin.org——Yes
868amazon.com.brGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
869one.one——Yes
872agora.io——Yes
873cursor.sh——Yes
875pexels.comGPTBot, ClaudeBot, Applebot-Extended, CCBot, Bytespider——
877ebay.co.ukGPTBot, ClaudeBot, anthropic-ai, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot, Bytespider, ChatGLM-SpiderOAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User—
879porkbun.com——Yes
880cloudinary.com——Yes
882ebay.deGPTBot, ClaudeBot, anthropic-ai, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot, Bytespider, ChatGLM-Spider——
888no-ip.com——Yes
890pendo.io——Yes
898nbcnews.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended +21 more——
902auth0.com——Yes
904daum.netGoogleOther——
906hbr.orgGPTBot, Google-Extended, CCBot——
907figma.comGPTBot, ClaudeBot, Google-Extended, FacebookBot, CCBot, Bytespider, cohere-ai, amazon-kendra, omgili, omgilibotOAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot—
909wyzecam.com—GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended, Meta-ExternalAgent, CCBotYes
913fwmrm.net——Yes
915singular.net—GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, Amazonbot, CCBot, BytespiderYes
918lg.com——Yes
919latimes.comGPTBot, ClaudeBot, Bytespider, Diffbot, ImagesiftBot, omgili, omgilibot——
926mercadolivre.com.brGPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, AmazonbotFacebookBot—
929seedtag.com—GPTBot, ChatGPT-User, ClaudeBot, PerplexityBot, BytespiderYes
935classlink.com——Yes
936patreon.comPerplexityBot, Google-CloudVertexBot, FacebookBot, Timpibot——
940genius.comGPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Meta-ExternalFetcher, FacebookBot, CCBot, Bytespider, YouBot +7 more——
941tailscale.com——Yes
964mercadolibre.com.arGPTBot, ChatGPT-User, ClaudeBot, Claude-User, PerplexityBot, Perplexity-User, AmazonbotFacebookBot—
967amazon.com.auGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, GoogleOther, Google-CloudVertexBot +78 more——
968otto.de—GPTBot, OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-UserYes
972corriere.itGPTBot, ChatGPT-User, ClaudeBot, anthropic-ai, Claude-Web, Google-Extended, Google-CloudVertexBot, Applebot-Extended, Meta-ExternalAgent, Amazonbot, CCBot, Bytespider +4 moreOAI-SearchBot, PerplexityBot, Amzn-SearchBot—
976ancestry.comGoogle-Extended——
980onet.plMeta-ExternalAgent, meta-webindexer, Amazonbot, Amzn-SearchBot, Amzn-User, Bytespider, YouBot, cohere-ai, Diffbot, Timpibot, Ai2Bot-Dolma, Diffbot-User +1 more——
985trendyol.com——Yes
986spiegel.deGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, Applebot-Extended, Meta-ExternalAgent, FacebookBot, CCBot +23 more——
989att.com——Yes
993instructure.com——Yes
998huffpost.comGPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, anthropic-ai, Claude-Web, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended +12 more——

Method

The sample is the top 10,000 domains of the Tranco research ranking, downloaded Oct 29, 2026. For each domain I requested /robots.txt and /llms.txt over HTTPS (falling back to www.), with a clearly identified research user agent. 5,701 domains returned a valid robots.txt; the rest are mostly API, CDN and tracking hostnames that serve no website.

Files were parsed per RFC 9309: a bot is governed by the group naming it (version suffixes ignored), otherwise by the * group, and counts as blocked when that group disallows / without a root-level Allow. An llms.txt counts only when it returns 200 at the exact path, is not HTML, and contains Markdown structure. The bot list combines my documented crawler set with the ai.robots.txt registry. More in the methodology.

About this dataset

What it covers

The robots.txt and llms.txt files of the 10,000 most popular domains in the Tranco research ranking, fetched on Oct 29, 2026, and how each file treats 197 AI crawlers and user agents. The download lists the AI policy of every top-1,000 site that blocks an AI bot or publishes an llms.txt; the per-bot and per-tier aggregates are on this page.

How it was built

Each domain was requested over HTTPS (with a www. fallback) using an identified research user agent. Files were parsed per RFC 9309: a bot follows the group naming it (version suffixes ignored), otherwise the * group, and is blocked when that group disallows / without a root-level Allow.

Limitations

robots.txt states intent; it doesn't enforce anything, and many sites also block bots at the firewall. 4,299 domains returned no valid robots.txt, mostly API, CDN and tracking hostnames. Partial blocks (specific paths only) are not counted as blocks.

Columns in robots-txt-study.csv

rank
Position in the Tranco list
domain
Registered domain
blocks
AI crawler tokens the site explicitly and fully disallows
allows
Documented AI crawlers the site names without blocking
llmsTxt
Whether the site serves a valid llms.txt

Questions

Does blocking GPTBot keep my content out of AI training?

Not on its own. Half (50%) of the sites that block GPTBot leave at least one other major training crawler, such as ClaudeBot, Google-Extended, CCBot, Meta-ExternalAgent or Applebot-Extended, unblocked. Only 344 sites (6%) block all eight major training crawlers.

Which websites block AI crawlers the most?

By country domain, French (.fr, 45.9%) and German (.de, 29.9%) sites block AI most, against 23.6% of .com sites. Universities (.edu) and governments (.gov) block least. The largest sites block more: 27.4% of the top 1,000.

What percentage of websites block AI crawlers?

Among the top 10,000 websites (Tranco list, Oct 29, 2026), 22.2% of sites with a robots.txt file explicitly block at least one AI crawler, and 29.6% name at least one AI bot. Among the top 1,000 the rate rises to 27.4%.

Which AI crawler is blocked the most?

CCBot is the most-blocked AI crawler, disallowed by 19% of sites with a robots.txt, followed by Bytespider (18.7%) and GPTBot (17.9%).

How many websites block GPTBot?

17.9% of top-10,000 sites with a robots.txt block GPTBot. Of those, 57.6% still allow OAI-SearchBot, so they opt out of OpenAI training while staying visible in ChatGPT search.

How many websites have an llms.txt file?

983 of the top 10,000 domains (9.8%) served a valid llms.txt file at the site root, rising to 11.6% among the top 1,000.

Analysis using this data

  1. Who blocks AI crawlers? What 10,000 robots.txt files revealPublishers block training, not answers; half of all blocklists leak; Cloudflare writes the most common AI policy.
  2. The opt-out asymmetry: you can block the small AI engines, not the big oneThe AI surfaces with the most users are the ones publishers can't refuse and can't measure.

How to cite this data

This dataset is free to use, including commercially, under Creative Commons Attribution 4.0. The one rule is attribution:

  1. Credit the source as “Kuldeep Paul, kuldeep.cc” wherever you use a figure, chart or table from it.
  2. Link to this page (https://kuldeep.cc/data/ai-search/robots-txt-study), not to the download file, so readers can check the source of every number.
  3. Include the date. These figures change; say “as of Oct 29, 2026” or link to the version you used.
  4. Mark changes. If you filter, recalculate or chart the data yourself, say so (“analysis based on data from…”).
  5. Cite the original source for single figures when you quote one reported number; it is linked in every row.

In running text: according to Kuldeep Paul's Who blocks AI crawlers? robots.txt study of the top 10,000 sites (kuldeep.cc). More in the methodology.

APA

Paul, K. (2026). Who blocks AI crawlers? robots.txt study of the top 10,000 sites [Data set]. kuldeep.cc. https://kuldeep.cc/data/ai-search/robots-txt-study

MLA

Paul, Kuldeep. "Who blocks AI crawlers? robots.txt study of the top 10,000 sites." kuldeep.cc, 29 October 2026, https://kuldeep.cc/data/ai-search/robots-txt-study.

Chicago

Paul, Kuldeep. 2026. "Who blocks AI crawlers? robots.txt study of the top 10,000 sites." Dataset. kuldeep.cc. Updated Oct 29, 2026. https://kuldeep.cc/data/ai-search/robots-txt-study.

BibTeX

@misc{paul2026robotstxtstudy, author = {Paul, Kuldeep}, title = {Who blocks AI crawlers? robots.txt study of the top 10,000 sites}, year = {2026}, howpublished = {\url{https://kuldeep.cc/data/ai-search/robots-txt-study}}, note = {Dataset, CC BY 4.0, updated 2026-10-29} }

HTML link

<a href="https://kuldeep.cc/data/ai-search/robots-txt-study">Who blocks AI crawlers? robots.txt study of the top 10,000 sites</a> (data: Kuldeep Paul, kuldeep.cc)