When you want a page to stay out of Google Search, robots.txt and noindex often come up together. They solve different problems. Robots.txt tells compliant crawlers which URLs they may crawl; noindex tells a search engine not to include an accessible page in its index. Treating them as interchangeable can leave an unwanted URL visible or prevent a crawler from seeing your intended instruction.
Crawling and indexing are separate decisions
Crawling is a request to fetch a URL. Indexing is the subsequent decision to process that content for potential appearance in search. An accessible URL may never be indexed. Conversely, an uncrawled URL blocked by robots.txt may still be discovered through links and shown with limited information. Read technical reports carefully: “blocked from crawling” and “excluded from indexing” do not describe the same state.
When should you use robots.txt?
Use robots.txt to manage crawler access to paths that do not need frequent crawling, such as certain site-search result patterns. A rule such as User-agent: * followed by Disallow: /search/ asks crawlers that respect the protocol not to request URLs under /search/. The file belongs at the site root, for example https://example.com/robots.txt.
Robots.txt is not a privacy or authentication control. Never use it as the only protection for a customer account or confidential document. It also does not support a Google noindex directive; putting “noindex” into robots.txt is not a substitute for supported metadata.
When is noindex appropriate?
If a page should remain available to visitors but should not appear in search, use an HTML directive such as <meta name="robots" content="noindex">. For non-HTML files, an HTTP X-Robots-Tag header can serve a similar purpose. An outdated campaign confirmation page is one possible example.
Crucially, crawlers need access to the page to read its noindex signal. Blocking that URL through robots.txt at the same time can prevent the signal from being seen. Do not build contradictory rules and assume they reinforce each other.
Worked example: an account URL
Goal: Keep /member/account out of search. Weak approach: Add only Disallow: /member/ and assume the URL cannot be discovered. Better approach: If personal information is involved, require authentication. If the URL is intentionally public but should not be indexed, keep it crawlable and add noindex. The correct solution depends on whether you need privacy, crawl management, or index exclusion.
For parameter-heavy filters such as /products?color=blue, review site navigation and canonicalization as well. A single directive will not fix an uncontrolled URL structure.
Testing checklist after a change
- Does the target URL return the expected HTTP status?
- Does /robots.txt block its path?
- Is there a meta robots or X-Robots-Tag directive?
- Is canonical pointing at an unexpected URL?
- What does Search Console URL Inspection report?
- Has the search engine had time to recrawl?
Start with our Robots.txt Tester and Indexability Checker. For site-wide patterns, use Site Crawler and inspect the affected URLs rather than guessing from one example.
How canonical, sitemaps and redirects fit in
Noindex is not a replacement for canonical on duplicate pages. Read our canonical implementation guide to understand preferred URLs. An XML sitemap should primarily contain URLs you want discovered and indexed. When a page moves permanently, a well-configured redirect may be more appropriate than leaving multiple URL versions accessible.
Frequently asked questions
Will robots.txt remove a URL from Google?
Not necessarily. It controls crawling, not guaranteed removal from results.
Does noindex improve Core Web Vitals?
No. Indexing directives and performance optimization address different concerns.
How quickly do changes take effect?
Reprocessing depends on crawling schedules. Validate your setup and monitor Search Console; do not assume immediate removal.
Further reading: Google robots.txt documentation and Google noindex guidance. Test decisions on your own URL set before rolling them out site-wide.
Join the discussion
Share your experience or ask a question about this guide.