Should I add GPTBot and ClaudeBot to my robots.txt?

What it is

The names we look for are GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, Claude-User, anthropic-ai, PerplexityBot, Google-Extended, Applebot-Extended, CCBot, Bytespider and meta-externalagent.

A crawler with no group of its own falls back to the wildcard `User-agent: *` group. That is a valid outcome, so this Check is a Warning when no name is present, never a Fail.

Naming an agent is not the same as admitting it. A named group with `Disallow: /` is an explicit refusal, and we record it as a stated rule either way.

Why an agent needs it

Silence is ambiguous. A wildcard group written for search engines in 2015 is now the rule that applies to clients fetching pages on behalf of a buyer.

Writing the groups out makes your position readable and deliberate. It also separates crawlers that gather training data from clients fetching one page for one person.

The Fix

Add one group per agent you have a view on. `User-agent: GPTBot` followed by `Allow: /` admits it; followed by `Disallow: /` refuses it.

Decide separately for training crawlers and for clients fetching a page for a named person. Google-Extended and Applebot-Extended govern training, ChatGPT-User and Claude-User are user-driven fetches.

Keep the wildcard group as the fallback, and make sure your CDN bot rules say the same thing. A robots.txt that admits an agent your edge blocks is a contradiction.

This one is a change in your own markup or edge configuration, so the Fix Pack cannot write it for you. A Report names every page it affects, with the Evidence.