Deindex pages from your site to improve SEO

We sometimes think that having a maximum number of pages indexed on search engines is a good thing in terms of SEO.

But it's not that simple.

It's true that the number of available pages is one of the criteria for pleasing Google and co... But it's about content pages, web documents containing information that users are actually likely to search for!

If your site references more legal pages (notices, terms and conditions, GDPR, etc.) and administrative pages (contact, login, sign up, etc.) than content pages, this could harm your SEO.

To solve this problem, it's possible to tell indexing bots that certain pages don't need to be indexed on search engines.

Here's how to do it!

Deindexing a page (meta tag)

You can tell search engine "crawlers" that a particular page should not be indexed, and to do this, you simply need to use the "robots" meta tag. Like this:

<META NAME="robots" CONTENT="noindex, nofollow">

noindex

As its name suggests, the "noindex" value will prevent the page from appearing in search results, simple and effective.

nofollow

Even if the page is not indexed, the bots will still crawl it looking for links to index other pages on the site. If you don't want any link contained in the page to be followed by the bot, you'll need to use the "nofollow" value.

Deindexing several pages (robots.txt file)

Rather than choosing to deindex pages one by one, you can opt for a more global approach to make one or more folders "invisible" to crawlers.

To do this, you simply need to create a robots.txt file and place it at the root of your site (next to your index.html), and specify the folders to ignore like this:

User-agent: *
Disallow: /wp-admin/
Disallow: /wp-includes/

You can specify specific "User-agents", to target a particular search engine, but the "*" character lets you target all bots at once!

If you want to test your robots.txt file, Google provides a tool designed for this in its webmaster tools: https://support.google.com/webmasters/answer/6062598