The basics of SEO #4: Preventing a page from being indexed

NicolasBrondinBernard

Author

@NicolasBrondinBernard

Because not all of your pages are worth ranking first on Google!

SEO HTML

Article published on 06/04/2021, last updated on 10/08/2026

Read article #3 of the series titled "Analyzing your SEO evolution"

Before starting to optimize the SEO of these pages, it's worth knowing which pages should preferably not be indexed and how to do it. This is what we're going to discover today.

Why refuse the indexing of a page?

There are three main reasons for wanting to prevent search engine bots from indexing some of your pages:

The first is that you don't want to present a useless page to a user, and risk losing them as soon as they arrive. This is the case for thank-you pages, legal notice pages, and other pages that are part of the site's lifecycle but that you don't want to present to a visitor on their first visit.

The second is that you need to be careful with pages submitted to search engines, because if you leave too many pages with weak content, a simple form (login, for example), a list of sponsored links, the ranking algorithm risks lowering the site's reputation because of the content it will have crawled.

And the last is that the content of certain pages can be almost identical to others, risking, for example, giving the impression of duplicate content. For example, you can find this kind of page on blogs with only one author, where the author's articles page will have the same content as the blog's homepage.

How to do it?

There are two methods, one consisting of enabling indexing page by page, and the other consisting of creating a file grouping together a set of pattern-based rules, called robots.txt:

With a meta tag

This solution is viable if the number of pages you need to avoid indexing is limited, or if none of the URLs of these pages follow a precise pattern. In this case, simply add the following tag inside the element of your page:

<head>
    ...
    <meta name="robots" content="noindex, nofollow">
    ...
</head>

The "noindex" directive tells robots not to index the page, while "nofollow" prevents the indexing of links. It's possible to mix both directives using their opposite values "index" and "follow".

The robots.txt file

The robots.txt file must be created at the root of the site (the file name is case-sensitive), and will be automatically read by crawlers. It contains groups of rules to indicate the indexing (or not) of certain pages filtered by URL patterns:

# robots.txt
User-agent: * # any robot
Disallow: /private/

User-agent: *
Disallow: /contact

In addition to the pattern, it's possible to select only certain specific crawlers. For more detailed information on robots.txt, here is a specialized article on the moz.com website

Hello I'm Nik sur Unsplash

Finished reading this article?

Our newsletter1 email a week to understand it all

Go further

No spam. Only free content, news, and ever more resources to level up your skills!

Join +1500 developers

No comments yet