Robots.txt and Noindex Checklist: Verify Search Visibility Controls
Decide whether a page should be crawled, indexed or kept private before changing its controls. Those are different goals. A robots.txt rule, a noindex directive and an authenticated access barrier do not provide the same protection.
Write down the intended outcome
For each tested URL, record whether it is a public page intended for search, a public utility page intentionally excluded from search or private content requiring restricted access. Include staging and preview environments as separate systems.
Do not assume an older instruction to discourage indexing still describes the launch plan. Confirm the current owner-approved intent and inspect the actual live page. This checklist is about verifying that intent, not making every page indexable by default.
Inspect the crawl rules
Request the robots.txt file on the actual hostname. Review the rules relevant to the tested crawler and path. Record the observed file and time, including any platform-generated behavior.
Google explains that blocking crawling with robots.txt is not a reliable way to keep a web-page URL out of search results. A blocked URL can still be known from links. Use appropriate indexing controls for public exclusions, and access controls for content that must remain private.
| Check | Evidence to record | Next decision |
|---|---|---|
| Intent | Public/indexable, public/excluded or private | Approve the desired behavior first |
| Crawl access | Relevant robots.txt rules and access response | Check whether the crawler can reach the page |
| Index directive | Robots meta tags and X-Robots-Tag headers | Find template or server exclusions |
| Private access | Authentication or equivalent restriction | Do not rely on search directives for privacy |
| Observation | Live inspection and processed report dates | Track changes without assuming instant processing |
Check page and response directives
Inspect the rendered head for robots or googlebot directives and the HTTP response for X-Robots-Tag. Different layers can publish different instructions. Record the component responsible for each value before editing.
Google supports noindex in a page meta tag or HTTP response header, but the crawler needs access to see the directive. A robots.txt block can prevent that observation. A noindex entry inside robots.txt is not Google’s supported implementation.
Correct only the intended scope
If a launched public page accidentally retains a staging exclusion, correct the responsible template, plugin or server setting through its supported workflow. Keep deliberately excluded or private pages under their planned controls.
Test more than the homepage: articles, category pages, downloads and unusual templates may use different outputs. Watch for a server-level header inherited by an entire section. Conversely, a single-page correction should not silently change the visibility of the whole site.
Retest and allow for processing
After an authorized correction, repeat the public request and record the visible directives. Use Search Console’s appropriate inspection features where available, distinguishing a live test from previously processed data. Do not interpret an old report as proof that the new output was ignored.
Finish with the intended policy, responsible component, tested URL groups and outstanding search observations. Removing an accidental noindex makes a page eligible for consideration; it does not guarantee inclusion or ranking. Keep crawl access, index eligibility and actual search performance as separate checks.
Compare robots rules with visible noindex output
A hypothetical page has a noindex directive but its path is also blocked from crawling. Verify both controls against the intended outcome: Google needs access to process a page’s noindex directive. Follow the source guidance and change only the appropriate scope. For private staging, retain real access controls; search directives do not provide authorization or protect confidential content.
Sources and scope
Official documentation checked October 6, 2026: Google: Block indexing with noindex; Google: Developer SEO guidance; Google: Robots meta and headers. This is HRL’s verification checklist, not a report of a completed migration or a guarantee of crawling, indexing or rankings.
Related guides
Protect staging · Verify sitemap coverage · Compare canonical signals