Noindex vs Nofollow vs Disallow: Robots Directives Explained
Noindex vs nofollow vs disallow explained: what each robots directive does, when to use noindex follow, whether nofollow links help SEO and common mistakes.
Noindex, nofollow and disallow are three different instructions to search engines, and mixing them up is one of the most common technical SEO mistakes. Noindex keeps a page out of search results. Nofollow is about links. Disallow stops crawling. Use the wrong one and pages either vanish from Google or stay in it when they shouldn’t.
This guide explains what each directive does, when to use them, whether nofollow links help SEO and the mistakes we see most often in audits. Fixing them usually takes a developer or a few technical SEO services hours, but deciding which one to use is the hard part.
Noindex vs nofollow vs disallow at a glance
| Directive | Where it goes | What it does | Use it for |
|---|---|---|---|
| noindex | Meta robots tag or X-Robots-Tag header | Keeps the page out of search results | Thank-you pages, internal search results, thin tag pages |
| nofollow (page level) | Meta robots tag | Asks Google not to follow any links on the page | Rarely needed |
| rel=“nofollow” (link level) | On a single link | Says you don’t vouch for that link | Untrusted links |
| rel=“sponsored” | On a single link | Marks paid or affiliate links | Ads, sponsorships, affiliate links |
| rel=“ugc” | On a single link | Marks user-generated links | Comments, forum posts |
| Disallow | robots.txt | Stops crawling of matching URLs | Admin areas, endless filters, internal tools |
What is noindex?
A noindex directive tells search engines not to show a page in their results. It goes in the page’s HTML as a meta robots tag, or in the HTTP header as an X-Robots-Tag for files like PDFs.
The key rule: Google has to crawl the page to see the noindex. If the page is also blocked in robots.txt, Google never sees the tag, and the URL can stay in the index.
Noindex, follow
“Noindex, follow” asks search engines to keep the page out of results but still follow its links. It’s common on paginated archives and tag pages. Google has said that pages left noindexed for a long time may eventually be treated as noindex, nofollow, so don’t rely on noindexed pages to pass link value long term.
What is nofollow?
Nofollow exists at two levels:
- Page level (meta robots nofollow): asks Google not to follow any link on the page. Rarely the right choice.
- Link level (rel=“nofollow”): marks a single link as one you don’t vouch for.
Since 2019, Google treats nofollow, sponsored and ugc as hints rather than strict commands.
Do nofollow links help SEO?
Mostly, nofollow links don’t pass ranking value the way followed links do. But because Google treats nofollow as a hint, some may still be used. More importantly, they can bring real visitors and brand visibility, which is why a natural link profile includes a mix of both. We cover the bigger picture in white hat link building for client sites.
What is disallow?
A disallow rule in robots.txt tells crawlers not to request matching URLs. It controls crawling, not indexing. If other sites link to a disallowed URL, Google can still index the URL, usually without a description, because it can’t see the page.
Use disallow for sections that waste crawling, such as admin areas, internal search and endless filter combinations. Don’t use it to hide pages from search; that’s what noindex is for. Our guide to crawlability and indexability explains how the two interact.
When to use each
| Goal | Use |
|---|---|
| Keep a page out of search results | noindex (and allow crawling) |
| Stop Google wasting time on useless URLs | Disallow in robots.txt |
| Mark a paid or affiliate link | rel=“sponsored” |
| Mark links in comments or forums | rel=“ugc” |
| Link to a page you don’t vouch for | rel=“nofollow” |
| Remove a page from Google quickly | noindex plus a removal request in Search Console |
How to check robots directives on a page
- View the page source and search for “robots” to find any meta robots tag.
- Check the HTTP headers for an X-Robots-Tag, which is easy to miss on PDFs and images.
- Open robots.txt and check whether any rule matches the URL.
- Use URL Inspection in Search Console to see whether Google can crawl and index the page, and which canonical it chose.
- Crawl the site to find every page with noindex or nofollow, so nothing important is excluded by mistake.
Common mistakes with robots directives
- Noindex plus disallow on the same page. Google can’t see the noindex, so the page may stay indexed.
- Noindex in robots.txt. Not supported since 2019.
- Staging noindex left on after launch. A whole site can disappear from Google this way.
- Nofollow on internal links to “save” link value. It doesn’t work and stops value flowing to your own pages.
- Disallowing CSS or JavaScript files, which stops Google rendering pages properly.
Check these in every audit. They’re part of the crawling and indexing section of our site audit checklist.