Technical eligibility · Branch T.A, Crawl, HTTP and indexable content
T-02 robots.txt and noindex must be used for different purposes
robots.txt controls crawling and noindex controls indexing. For security, authentication must be used instead of either.
- Scope
- All public content
- Priority
- Gate. A FAIL stops publish or launch.
- Quality bucket
- Technical eligibility
- Legacy coverage
- SEO-34
Why it matters
Mixing up the two controls leads to pages that stay in search results or content that is exposed. A robots.txt block stops crawlers from seeing a noindex tag, and neither one keeps sensitive material private. Google's technical requirements keep the two jobs separate for this reason.
How to implement
- Group your URLs by purpose: indexable, not to be indexed, crawl-heavy and sensitive.
- For pages that should stay out of the index, allow crawling and add noindex.
- Use robots.txt disallow only to reduce crawl load.
- Put sensitive content behind authentication rather than relying on robots.txt or noindex.
Checklist
- For pages that should not be indexed: allow crawling and add noindex.
- To reduce crawl load: use a robots.txt disallow.
- For sensitive content: use authentication, because robots is not a security tool.
How to audit
- Compare robots.txt, meta and X-Robots policies by URL group.
How UltimateSEO checks this automatically
Automated. Checked automatically: robots.txt rules against meta robots and X-Robots-Tag headers on crawled URLs, flagging pages that are both disallowed and noindexed or otherwise in conflict.
States used: PASS, FAIL, UNKNOWN, NOT_APPLICABLE or BLOCKED. UNKNOWN is never shown as a pass. How states work.
Sources
Part of The UltimateSEO Framework v4.1, revised 2026-10-07. Evidence states, control states and the publish gate are explained on the hub.