Technical eligibility · Branch T.A, Crawl, HTTP and indexable content
T-11 WAF and CDN must be configured so they do not block real crawlers
Even when robots.txt is open, logs must confirm that real crawlers are not blocked at the edge by 401, 403, 429 or challenge responses.
- Scope
- All public content
- Priority
- Gate. A FAIL stops publish or launch.
- Quality bucket
- Technical eligibility
- Legacy coverage
- GEO-09
Why it matters
An open robots.txt means little if the WAF or CDN quietly turns crawlers away with errors or challenges. Search and AI crawlers then cannot read the pages, so the site drops out of results and answers without any visible warning. Robots.txt alone cannot show edge behaviour, which is why logs have to confirm that the OpenAI, Anthropic and Perplexity crawlers are getting through.
How to implement
- Separate verified crawler traffic in your edge logs.
- Track 401, 403, 429 and challenge rates for each user-agent.
- Build allowlists with each platform's official verification or IP methods, not the user-agent string alone.
- Write separate security policies for public SEO pages and private areas.
- Re-check the logs after every WAF or CDN change.
Checklist
- Separate verified crawler traffic in edge logs.
- Monitor 403, 401, 429 and challenge rates by user-agent.
- Do not build an allowlist from the user-agent string alone. Use the platform's official verification or IP methods.
- Design the security policy separately for public SEO pages and private areas.
How to audit
- Review edge and WAF logs.
- Review bot analytics.
- Check the platform's official bot verification methods.
How UltimateSEO checks this automatically
Partly automated. The crawler fetches with a Googlebot-like user agent and the default user agent and compares status codes; server logs or WAF rules from the customer are needed to confirm no real crawler is blocked. The crawl gives a signal; a person decides the final state with business evidence.
States used: PASS, FAIL, UNKNOWN, NOT_APPLICABLE or BLOCKED. UNKNOWN is never shown as a pass. How states work.
Sources
Part of The UltimateSEO Framework v4.1, revised 2026-10-07. Evidence states, control states and the publish gate are explained on the hub.