Technical eligibility · Branch T.D, Platform crawlers and AI eligibility
T-14 Anthropic crawler roles must be managed separately
Claude-SearchBot, ClaudeBot and Claude-User have different purposes. Search, training and user-directed access must not be mixed up under a single blanket rule.
- Scope
- Anthropic and Claude crawler governance
- Priority
- High priority.
- Quality bucket
- Technical eligibility
- Legacy coverage
- GEO-06
Why it matters
A single blanket rule for all Anthropic bots will either shut out Claude search visibility or allow training you did not intend. Each bot serves a different purpose, so one setting cannot express three different decisions. Handling them separately keeps your search, training and user-directed access choices clear and testable.
How to implement
- Decide a policy separately for Claude-SearchBot, ClaudeBot and Claude-User.
- If you want Claude search visibility, test that Claude-SearchBot can reach your pages.
- Set the Claude-User policy to match your need for user-directed access.
- Manage the training preference on ClaudeBot on its own.
- Review robots.txt, server and WAF logs, and status codes for each bot.
Checklist
- Make a separate policy decision for each Anthropic bot.
- If Claude search visibility is wanted, test Claude-SearchBot access.
- If user-directed access is wanted, set the Claude-User policy accordingly.
- Manage the training preference separately on ClaudeBot.
How to audit
- Review robots.txt.
- Review the server and WAF logs.
- Analyse status codes by bot.
How UltimateSEO checks this automatically
Automated. Checked automatically: robots.txt groups for ClaudeBot, Claude-SearchBot and Claude-User are evaluated separately with the effective group rules.
States used: PASS, FAIL, UNKNOWN, NOT_APPLICABLE or BLOCKED. UNKNOWN is never shown as a pass. How states work.
Sources
Part of The UltimateSEO Framework v4.1, revised 2026-10-07. Evidence states, control states and the publish gate are explained on the hub.