Kuldeep Paul

Model training · ByteDance

Bytespider

ByteDance's crawler, widely reported as aggressive and as not respecting robots.txt. Many sites block it at the network level instead.

Compiled by · Updated Oct 29, 2026

18.7%of top sites fully block Bytespider
1,067sites blocking, of 5,701 with robots.txt
125of the top 1,000 block it
Operator
ByteDance
Purpose
Model training
Used for
ByteDance LLMs (Doubao)
Obeys robots.txt
No
robots.txt token
Bytespider
User-agent string
Not published by the operator
IP list
Not published
Official docs
None published

Block Bytespider in robots.txt

User-agent: Bytespider
Disallow: /

Well-known sites that block Bytespider

amazon.com #30, github.com #33, yahoo.com #72, amazonvideo.com #85, vimeo.com #87, msn.com #92, chatgpt.com #112, t-online.de #129, snapchat.com #157, nytimes.com #163, gravatar.com #165, unity3d.com #186, ebay.com #191, cnn.com #195, theguardian.com #196, medium.com #198, bbc.com #216, forter.com #220, sciencedirect.com #224, bbc.co.uk #236, amplitude.com #247, weather.com #250, canva.com #276, launchpad.net #281

Ranks from the Tranco list. Full per-site data in the robots.txt study.

Questions

What is Bytespider?

Bytespider is a model training bot operated by ByteDance, used for ByteDance LLMs (Doubao). ByteDance's crawler, widely reported as aggressive and as not respecting robots.txt. Many sites block it at the network level instead.

How do I block Bytespider?

Add "User-agent: Bytespider" followed by "Disallow: /" to the robots.txt file at the root of each host you want to protect. Bytespider is reported not to honour robots.txt, so a firewall or bot-management rule is more reliable.

Does Bytespider respect robots.txt?

Not reliably. Bytespider is documented or widely reported as ignoring robots.txt.

How many websites block Bytespider?

In a Oct 29, 2026 analysis of the top 10,000 websites by Kuldeep Paul, 18.7% of sites with a robots.txt file fully disallowed Bytespider.

Analysis using this data

  1. Who blocks AI crawlers? What 10,000 robots.txt files revealPublishers block training, not answers; half of all blocklists leak; Cloudflare writes the most common AI policy.
  2. The opt-out asymmetry: you can block the small AI engines, not the big oneThe AI surfaces with the most users are the ones publishers can't refuse and can't measure.

← All AI crawlers

How to cite this data

This dataset is free to use, including commercially, under Creative Commons Attribution 4.0. The one rule is attribution:

  1. Credit the source as “Kuldeep Paul, kuldeep.cc” wherever you use a figure, chart or table from it.
  2. Link to this page (https://kuldeep.cc/data/ai-search/crawlers/bytespider), not to the download file, so readers can check the source of every number.
  3. Include the date. These figures change; say “as of Oct 29, 2026” or link to the version you used.
  4. Mark changes. If you filter, recalculate or chart the data yourself, say so (“analysis based on data from…”).
  5. Cite the original source for single figures when you quote one reported number; it is linked in every row.

In running text: according to Kuldeep Paul's Bytespider — AI crawler directory (kuldeep.cc). More in the methodology.

APA

Paul, K. (2026). Bytespider — AI crawler directory [Data set]. kuldeep.cc. https://kuldeep.cc/data/ai-search/crawlers/bytespider

MLA

Paul, Kuldeep. "Bytespider — AI crawler directory." kuldeep.cc, 29 October 2026, https://kuldeep.cc/data/ai-search/crawlers/bytespider.

Chicago

Paul, Kuldeep. 2026. "Bytespider — AI crawler directory." Dataset. kuldeep.cc. Updated Oct 29, 2026. https://kuldeep.cc/data/ai-search/crawlers/bytespider.

BibTeX

@misc{paul2026bytespider, author = {Paul, Kuldeep}, title = {Bytespider — AI crawler directory}, year = {2026}, howpublished = {\url{https://kuldeep.cc/data/ai-search/crawlers/bytespider}}, note = {Dataset, CC BY 4.0, updated 2026-10-29} }

HTML link

<a href="https://kuldeep.cc/data/ai-search/crawlers/bytespider">Bytespider — AI crawler directory</a> (data: Kuldeep Paul, kuldeep.cc)