Internet & Apps
Does ChatGPT Respect robots.txt? What Each OpenAI Bot Does
Does ChatGPT respect robots.txt? OpenAI uses it for GPTBot and OAI-SearchBot, but says it may not apply to ChatGPT-User. See what to set.
7 min read
Free tool
AI Crawler Checker: Is GPTBot or ClaudeBot Blocked?
Check if your robots.txt allows or blocks GPTBot, ClaudeBot, PerplexityBot and Google-Extended. Free, no sign-up, with the exact rule behind each result.
Share this tool
Found it useful? Send it to someone, or link to it from your own page.
Link to this tool from your site
Copy this HTML and paste it into your page, blog post or resource list.
How to use the AI Crawler Checker
- Enter your domain, or switch to Paste robots.txt and paste the file contents.
- Optionally change the page path to test. Use / for your whole site or a path like /blog/my-post.
- Read the status for each bot and the line number of the rule that decides it.
- Follow the prioritized steps. If you want to change anything, copy the generated rules into your robots.txt.
What this checker does
It reads a robots.txt file and shows, for each well-known AI bot, whether that bot may fetch the page you are testing. It follows the Robots Exclusion Protocol (RFC 9309): a bot uses the group that names it, otherwise the * group, the longest matching rule wins, Allow wins a tie, and no matching rule means allowed.
It checks permission only. A bot being allowed does not mean an AI product will crawl, use or cite your pages, and the tool cannot tell you that.
Three kinds of AI bots
| Type | Examples | What blocking it means |
|---|---|---|
| AI search | OAI-SearchBot, Claude-SearchBot, PerplexityBot | Per vendor documentation, your pages stay out of that product's search answers. |
| User-requested | ChatGPT-User, Claude-User, Perplexity-User | A person asked the assistant to open a page. Some vendors say robots.txt may not apply. |
| Training | GPTBot, ClaudeBot, CCBot, Google-Extended | Signals that your content should not be used to train models. Vendors say this is separate from search. |
Bot names and policies change. The result table links the vendor documentation for the bots we checked against it, and you should confirm there before relying on a rule.
Common problems it catches
- A
User-agent: *block withDisallow: /that also shuts out AI search bots you meant to allow. - Rules placed before any
User-agentline, which crawlers ignore. - Misspelled directives such as
Dissallow. - A robots.txt URL that returns an HTML page or a server error.
- No Sitemap line.
Privacy and limits
For a website check, our server fetches only /robots.txt and /llms.txt from the public domain you enter, with a short timeout and a size limit. It refuses IP addresses, local and internal names, domains that do not resolve to public servers, and non-standard ports. We do not store the address or the file. Pasted text is analyzed in your browser. If a site blocks our request, paste the file instead.
The /llms.txt line is informational. It is a community proposal, and we do not know of a major AI company that has confirmed using it.
Looking at other pages of your site? Try the meta title and description checker, the keyword density checker or the keyword cannibalization checker.
Frequently asked questions
How do I check if GPTBot can crawl my website?
Enter your domain above. The tool reads your robots.txt, finds the rules that apply to GPTBot and shows whether it is allowed, with the line number of the rule that decides it. You can also open yourdomain.com/robots.txt and look for a GPTBot group.
Does blocking GPTBot remove my site from ChatGPT search?
According to OpenAI, no. GPTBot is the training crawler and OAI-SearchBot controls ChatGPT search results. OpenAI says sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers.
Does Google-Extended affect my Google Search rankings?
Google says it does not. Google-Extended is a robots.txt token that controls whether content Google crawls can be used for Gemini training and grounding, and it does not affect inclusion in Google Search.
Should I block AI crawlers?
It depends on your goals. Allowing AI search bots lets those products use your pages in answers. Blocking training bots is a separate choice about whether your content is used to train models. Neither choice guarantees traffic or citations.
Do AI bots always follow robots.txt?
No. Major crawlers say they respect it, but OpenAI and Perplexity say user-requested fetchers such as ChatGPT-User and Perplexity-User may not follow it. robots.txt is a request. To truly restrict content, use authentication or server rules.
Learn more