User-triggered fetch · Meta
Meta-ExternalFetcher
Fetches individual links on behalf of users of Meta AI and other products. Meta notes it may bypass robots.txt because a user initiated the fetch.
- Operator
- Meta
- Purpose
- User-triggered fetch
- Used for
- Meta AI
- Obeys robots.txt
- No
- robots.txt token
Meta-ExternalFetcher- User-agent string
- Not published by the operator
- IP list
- Not published
- Official docs
- developers.facebook.com
- Feeds
- Meta AI
Block Meta-ExternalFetcher in robots.txt
User-agent: Meta-ExternalFetcher
Disallow: /Blocking a search or user-triggered bot removes your pages from that assistant's answers. To opt out of model training only, block the operator's training crawler instead.
Well-known sites that block Meta-ExternalFetcher
amazon.com #30, amazonvideo.com #85, msn.com #92, snapchat.com #157, nytimes.com #163, sciencedirect.com #224, launchpad.net #281, amazon.co.uk #326, washingtonpost.com #383, bloomberg.com #386, amazon.de #387, fandom.com #473, amazon.co.jp #490, uol.com.br #535, ft.com #552, wired.com #570, usatoday.com #617, dailymail.com #619, telegraph.co.uk #629, amazon.fr #632, amazon.ca #633, amazon.in #676, primevideo.com #678, yelp.com #702
Other Meta bots
Meta-ExternalAgent (model training) · meta-webindexer (ai search index) · FacebookBot (model training)
Questions
What is Meta-ExternalFetcher?
Meta-ExternalFetcher is a user-triggered fetch bot operated by Meta, used for Meta AI. Fetches individual links on behalf of users of Meta AI and other products. Meta notes it may bypass robots.txt because a user initiated the fetch.
How do I block Meta-ExternalFetcher?
Add "User-agent: Meta-ExternalFetcher" followed by "Disallow: /" to the robots.txt file at the root of each host you want to protect. Meta-ExternalFetcher is reported not to honour robots.txt, so a firewall or bot-management rule is more reliable.
Does Meta-ExternalFetcher respect robots.txt?
Not reliably. Meta-ExternalFetcher is documented or widely reported as ignoring robots.txt.
How many websites block Meta-ExternalFetcher?
In a Oct 29, 2026 analysis of the top 10,000 websites by Kuldeep Paul, 8.4% of sites with a robots.txt file fully disallowed Meta-ExternalFetcher.
Analysis using this data
- Who blocks AI crawlers? What 10,000 robots.txt files revealPublishers block training, not answers; half of all blocklists leak; Cloudflare writes the most common AI policy.
- The opt-out asymmetry: you can block the small AI engines, not the big oneThe AI surfaces with the most users are the ones publishers can't refuse and can't measure.
How to cite this data
This dataset is free to use, including commercially, under Creative Commons Attribution 4.0. The one rule is attribution:
- Credit the source as “Kuldeep Paul, kuldeep.cc” wherever you use a figure, chart or table from it.
- Link to this page (https://kuldeep.cc/data/ai-search/crawlers/meta-externalfetcher), not to the download file, so readers can check the source of every number.
- Include the date. These figures change; say “as of Oct 29, 2026” or link to the version you used.
- Mark changes. If you filter, recalculate or chart the data yourself, say so (“analysis based on data from…”).
- Cite the original source for single figures when you quote one reported number; it is linked in every row.
In running text: according to Kuldeep Paul's Meta-ExternalFetcher — AI crawler directory (kuldeep.cc). More in the methodology.
Paul, K. (2026). Meta-ExternalFetcher — AI crawler directory [Data set]. kuldeep.cc. https://kuldeep.cc/data/ai-search/crawlers/meta-externalfetcher
Paul, Kuldeep. "Meta-ExternalFetcher — AI crawler directory." kuldeep.cc, 29 October 2026, https://kuldeep.cc/data/ai-search/crawlers/meta-externalfetcher.
Paul, Kuldeep. 2026. "Meta-ExternalFetcher — AI crawler directory." Dataset. kuldeep.cc. Updated Oct 29, 2026. https://kuldeep.cc/data/ai-search/crawlers/meta-externalfetcher.
@misc{paul2026metaexternalfetcher, author = {Paul, Kuldeep}, title = {Meta-ExternalFetcher — AI crawler directory}, year = {2026}, howpublished = {\url{https://kuldeep.cc/data/ai-search/crawlers/meta-externalfetcher}}, note = {Dataset, CC BY 4.0, updated 2026-10-29} }
<a href="https://kuldeep.cc/data/ai-search/crawlers/meta-externalfetcher">Meta-ExternalFetcher — AI crawler directory</a> (data: Kuldeep Paul, kuldeep.cc)