SEO & AI visibility
Robots rules for search and AI crawlers
Evaluate the exact file, bot and URL you want to control. Robots rules are matched by user agent against the effective group, so one stray Disallow can silence the crawlers that feed AI answers. Test the live file after every change, and remember robots.txt manages crawling only: use noindex for indexing decisions.
Separate crawling from access control
A robots file describes crawling preferences. It does not protect private content from people, indexing via other sources or clients that do not follow those rules. Private reports require authentication. Use noindex appropriately for pages you do not want indexed, while ensuring crawlers can observe that directive.
Choose the correct bot
Googlebot relates to Google Search crawling and the eligibility of content for Search AI features. Google-Extended has a different role for Google’s specified AI training and grounding uses. OpenAI search, training and user-triggered retrieval also use different identifiers. One permission should not be described as controlling every AI experience.
Test a specific path
Paste your actual file into the bot-access tool and enter the page URL. Inspect specific groups and wildcard groups. Then check the deployed HTTP response and any CDN or firewall rules. An allowed result from the supplied file does not prove that the live server accepts the crawler.
See the measurement methodology for product-specific coverage and limitations.