Kuldeep Paul

AI agent · OpenAI

ChatGPT Agent

Identifies ChatGPT's agent when it browses and acts on websites for a user. Requests are signed with HTTP Message Signatures so they can be verified.

Compiled by · Updated Oct 29, 2026

4.6%of top sites fully block ChatGPT Agent
263sites blocking, of 5,701 with robots.txt
42of the top 1,000 block it
Operator
OpenAI
Purpose
AI agent
Used for
ChatGPT agent mode
Obeys robots.txt
Unclear
robots.txt token
ChatGPT Agent
User-agent string
Not published by the operator
IP list
Not published
Official docs
developers.openai.com
Feeds
ChatGPT

Block ChatGPT Agent in robots.txt

User-agent: ChatGPT Agent
Disallow: /

Well-known sites that block ChatGPT Agent

usatoday.com #617

Ranks from the Tranco list. Full per-site data in the robots.txt study.

Other OpenAI bots

GPTBot (model training) · OAI-SearchBot (ai search index) · ChatGPT-User (user-triggered fetch) · OAI-AdsBot (ad verification)

Questions

What is ChatGPT Agent?

ChatGPT Agent is a ai agent bot operated by OpenAI, used for ChatGPT agent mode. Identifies ChatGPT's agent when it browses and acts on websites for a user. Requests are signed with HTTP Message Signatures so they can be verified.

How do I block ChatGPT Agent?

Add "User-agent: ChatGPT Agent" followed by "Disallow: /" to the robots.txt file at the root of each host you want to protect.

Does ChatGPT Agent respect robots.txt?

OpenAI has not clearly documented whether ChatGPT Agent honours robots.txt.

How many websites block ChatGPT Agent?

In a Oct 29, 2026 analysis of the top 10,000 websites by Kuldeep Paul, 4.6% of sites with a robots.txt file fully disallowed ChatGPT Agent.

Analysis using this data

  1. Who blocks AI crawlers? What 10,000 robots.txt files revealPublishers block training, not answers; half of all blocklists leak; Cloudflare writes the most common AI policy.
  2. The opt-out asymmetry: you can block the small AI engines, not the big oneThe AI surfaces with the most users are the ones publishers can't refuse and can't measure.

← All AI crawlers

How to cite this data

This dataset is free to use, including commercially, under Creative Commons Attribution 4.0. The one rule is attribution:

  1. Credit the source as “Kuldeep Paul, kuldeep.cc” wherever you use a figure, chart or table from it.
  2. Link to this page (https://kuldeep.cc/data/ai-search/crawlers/chatgpt-agent), not to the download file, so readers can check the source of every number.
  3. Include the date. These figures change; say “as of Oct 29, 2026” or link to the version you used.
  4. Mark changes. If you filter, recalculate or chart the data yourself, say so (“analysis based on data from…”).
  5. Cite the original source for single figures when you quote one reported number; it is linked in every row.

In running text: according to Kuldeep Paul's ChatGPT Agent — AI crawler directory (kuldeep.cc). More in the methodology.

APA

Paul, K. (2026). ChatGPT Agent — AI crawler directory [Data set]. kuldeep.cc. https://kuldeep.cc/data/ai-search/crawlers/chatgpt-agent

MLA

Paul, Kuldeep. "ChatGPT Agent — AI crawler directory." kuldeep.cc, 29 October 2026, https://kuldeep.cc/data/ai-search/crawlers/chatgpt-agent.

Chicago

Paul, Kuldeep. 2026. "ChatGPT Agent — AI crawler directory." Dataset. kuldeep.cc. Updated Oct 29, 2026. https://kuldeep.cc/data/ai-search/crawlers/chatgpt-agent.

BibTeX

@misc{paul2026chatgptagent, author = {Paul, Kuldeep}, title = {ChatGPT Agent — AI crawler directory}, year = {2026}, howpublished = {\url{https://kuldeep.cc/data/ai-search/crawlers/chatgpt-agent}}, note = {Dataset, CC BY 4.0, updated 2026-10-29} }

HTML link

<a href="https://kuldeep.cc/data/ai-search/crawlers/chatgpt-agent">ChatGPT Agent — AI crawler directory</a> (data: Kuldeep Paul, kuldeep.cc)