T-13 OpenAI search, training and user-directed crawler roles must be managed separately

Why it matters

Treating the three OpenAI bots as one makes it easy to block search visibility by accident while trying to refuse training, or to allow training while believing only search is permitted. A WAF that challenges or rate-limits OAI-SearchBot can quietly keep a site out of ChatGPT Search. Separate policies let each decision be made, tested and explained on its own.

How to implement

1. Write a separate robots.txt policy for OAI-SearchBot, GPTBot and ChatGPT-User.
2. If you want search visibility, allow OAI-SearchBot on your critical public paths.
3. If you do not want training, add a separate Disallow rule for GPTBot.
4. Check that the CDN and WAF are not returning 403s, challenges or aggressive rate limits to OAI-SearchBot.
5. Split the three bots into separate categories in your server and edge logs.
6. Record the three roles and the agreed policy in the security runbook.

Checklist

• State your OAI-SearchBot policy explicitly in robots.txt.
• If search visibility is wanted, allow the critical public paths.
• Verify that the CDN and WAF are not returning 403s, challenges or aggressive rate limits.
• If verification against the IP information published by OpenAI is needed, update the edge policy.
• Monitor OAI-SearchBot fetches separately in the server logs.
• Set the business training policy with the legal and product teams.
• If search visibility is wanted, keep OAI-SearchBot allowed.
• If training is not wanted, write a separate Disallow rule for GPTBot.
• Document the robots policy in the security runbook.
• Split OAI-SearchBot, GPTBot and ChatGPT-User into separate categories in the edge logs.
• When there is a search visibility problem, check OAI-SearchBot first.
• Set the ChatGPT-User policy according to the need for user-directed access.
• Document the roles of the three bots for the security team.

How to audit

• Check the live robots.txt.
• Review the edge and WAF logs.
• Review the HTTP status distribution of search bot requests.
• Review the robots.txt policy bot by bot.
• Segment the server logs by bot.
• Segment the logs by user-agent.
• Compare the robots policy with actual traffic.

Related controls

• T-12 Google generative Search eligibility must be built on core Search eligibility P0
• T-14 Anthropic crawler roles must be managed separately P1
• T-15 PerplexityBot access must be managed deliberately P1
• T-16 Bing crawl, indexing and AI telemetry must be set up together P1

Sources

• [O1] OpenAI crawler documentation
• [O2] OpenAI Publishers and Developers FAQ