Does ChatGPT Respect robots.txt? What Each OpenAI Bot Does

Mostly yes, but it depends on which OpenAI bot you mean. OpenAI says robots.txt is how you opt out of its search crawler (OAI-SearchBot) and its training crawler (GPTBot). For ChatGPT-User, which acts when a person asks ChatGPT to open a page, OpenAI says robots.txt rules may not apply.

That one sentence is the whole answer. The rest of this guide shows you what each bot is for, what to put in your file, and how to check that your file does what you think it does.

The 3 OpenAI bots and what each one does

OpenAI lists several crawlers. Three matter for most site owners.

Two details are easy to miss.

ChatGPT-User is not a crawler. OpenAI says it is not used to crawl the web in an automatic fashion. It fetches a page because someone asked.

OpenAI does not promise that a robots.txt rule is the same as a lock. Its page describes the opt-outs. It does not say every request from every tool will behave the same way, so do not treat robots.txt as security.

What blocking one bot does not do

Many site owners block GPTBot and think they are finished. Here is what that does and does not cover.

• Blocking GPTBot is the training opt-out. It tells OpenAI your content should not be used to train its models.
• It does not change OAI-SearchBot. The search crawler has its own entry. If you want to stay out of ChatGPT search, you need to block OAI-SearchBot too. OpenAI says sites that opt out of search will not be shown in ChatGPT search answers, though they can still appear as navigational links.
• It does not undo the past. A new rule applies to future crawling. It does not remove anything that was collected earlier.
• It does not control user-started fetches. That is the ChatGPT-User row in the table.

How to set robots.txt for what you want

Your robots.txt file lives at the root of your domain, for example https://example.com/robots.txt. It is a plain text file. Each group starts with a User-agent: line naming the bot, followed by Allow: or Disallow: lines.

I tested each example below with the AI Crawler Checker's own rule parser. That confirms what the file says, not how a bot behaves in real life.

Stay in ChatGPT search, but opt out of training

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

Result: GPTBot is blocked and OAI-SearchBot is allowed. ChatGPT-User has no rule, so it counts as allowed here.

Block all three OpenAI bots

User-agent: GPTBot
Disallow: /

User-agent: OAI-SearchBot
Disallow: /

User-agent: ChatGPT-User
Disallow: /

Result: all three are blocked in the file. Remember that OpenAI says the last one may not follow it.

Allow all three

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

Result: all three are allowed. A site with no rules for them behaves the same way.

Where the file goes and how long it takes

Upload the file to the root of your domain and make sure it loads in a browser. OpenAI says it can take about 24 hours from a robots.txt update for its systems to adjust for search results. So a change you make today may not show up in ChatGPT tomorrow morning.

Common mistakes

• A blanket block. User-agent: * with Disallow: / blocks every bot that has no group of its own. If you add a rule for OAI-SearchBot below it, that bot is allowed and the rest stay blocked. Make sure that is what you meant.
• A typo in the bot name. A rule for a name nobody uses does nothing.
• Rules under the wrong group. Each Allow or Disallow line belongs to the User-agent line above it.
• A firewall or CDN rule. A bot can be turned away before it ever reads your file. The file can look fine while the bot is blocked somewhere else.

How to check what your robots.txt really allows

Do not guess. Test it.

1. Open the AI Crawler Checker.
2. Enter your domain. The tool fetches only your public robots.txt and llms.txt files.
3. Read the list. Each bot shows as allowed or blocked, with the exact rule behind the result.
4. To test one section, enter a path such as /blog/my-post instead of /.
5. Fix your file and run it again.

What the checker cannot do: it reads robots.txt only. It does not see your CDN or firewall settings, and it cannot prove that a bot actually visited your site. Your server logs can show that.

What if you need more than a polite request?

robots.txt is a request. Honest crawlers follow it, and a crawler that does not care can ignore it. If a page must stay private, put it behind a login. Site owners who want a stronger block use firewall or CDN rules.

Some vendors publish the IP addresses their bots use. OpenAI lists them for each of its bots, and Anthropic publishes a list too. One warning from Anthropic is worth remembering: blocking its crawler's IP addresses may not reliably work as an opt-out, because it stops the bot from reading your robots.txt. Anthropic says to opt out by editing robots.txt.

What about Claude, Perplexity and Google?

Each company runs its own bots, and the rules are not identical.

Note that Google-Extended is a token, not a separate crawler. Google says it has no separate user agent string. It only controls how already-crawled content is used.

Should you block or allow?

This is your decision, and the trade-offs are real.

• If you want your pages to appear in ChatGPT search, allow OAI-SearchBot.
• If you want to keep your content out of OpenAI's training, disallow GPTBot.
• If you want neither, block both, and keep in mind that ChatGPT-User may still fetch a page when a person asks.
• If a page is private, do not depend on robots.txt. Use a login.

I cannot tell you that either choice will raise or lower your traffic. The sources say what each bot is for. They do not measure the effect on any particular site.

Troubleshooting

"I blocked GPTBot, but ChatGPT still mentions my site." GPTBot is only the training opt-out. Search, user-started fetches and content collected earlier are separate.

"The checker shows a bot as allowed, but I meant to block it." Look at the rule it shows. A wildcard group, a rule placed in the wrong group, or a missing Disallow line is the usual cause.

"My change is not showing." Open your robots.txt in a browser to make sure the new version is live. Then allow about a day, as OpenAI notes for search.

"Everything looks right but a bot still gets through." robots.txt is a request. Check your server logs. If you need a hard block, ask your host or CDN about firewall rules.

Limits of this guide

I read OpenAI's, Anthropic's, Perplexity's and Google's own pages on November 4, 2026. I did not test how any bot behaves on a live site. The robots.txt examples were checked with the AI Crawler Checker's rule parser, not with the bots themselves. Companies change their crawlers and policies, so confirm anything important on the source pages before you rely on it.

The short version

OpenAI uses robots.txt for GPTBot (training) and OAI-SearchBot (search), and says it may not apply to ChatGPT-User, which acts for a person. Set each bot on purpose, then test your own robots.txt to see what your file really says.