# How to crawl a web page in NodeJS with Puppeteer and Cheerio

Sometimes it's essential to retrieve regularly updated information from a website that doesn't expose any public API, or simply to browse all the pages of a particular website like a search engine bot in order to index them.

Disclaimer: Be careful, most of the information published on websites remains the property of those sites and is not intended to be collected by a third party without authorization.

If you still need to retrieve such information, you'll need to retrieve the source code of the pages in question (crawling), then search for the data you need (parsing) within this mass of information.

We're going to see how to do all this in NodeJS today, with minimal code that you can reuse and adapt as you see fit.

The method

To retrieve our web page and the data we're interested in, we're going to use two different tools: Puppeteer and CheeriosJS. If you're not familiar with these two libraries, don't worry, I'll explain everything.

Puppeteer

Puppeteer is a NodeJS library that provides an API to automate actions on one (or several) Chrome/Chromium instance. In other words, thanks to this library you'll be able to perform all the actions you would with a regular browser, but controlled programmatically.

Puppeteer includes many features by default in addition to loading pages, such as taking screenshots, generating a PDF from a web page, etc...

By default, Puppeteer runs in "headless" mode, meaning it doesn't display any window or graphical rendering.

CheerioJS

Cheerio is simply a server-side implementation of the famous jQuery library. By passing it HTML source code along with a query, Cheerio is able to return the requested information in a simple and very fast way!

You didn't think you'd be reusing jQuery someday, did you?

The code

Here's minimal code to get these two libraries working together. By running this program, Puppeteer will load the main page of my blog, then pass the source code to Cheerio so it can parse all the links present on the page.

Since the code is simple and commented on almost every line, I won't go into more detail than what's already there!

const puppeteer = require('puppeteer'),
    $ = require('cheerio');

(async function(){
	
    //Chrome instance launching, do not launch an instance for each page
    let browser = await puppeteer.launch();
    //Create new blank page in browser
    let page = await browser.newPage();
    //Open the desired url
    await page.goto("https://blog.nicolas.brondin-bernard.com/");
    //Once page loaded, get the source code
    let html = await page.content();
    //Use cheerio to parse every link attribute
    let links = $("a", html);
    links.each(function(i,link){
        console.log($(link).attr("href"));
    })
    //Don't forget to close the browser instance
    await browser.close(); 

})();

You should see the list of links present on my blog's homepage appear after a few seconds directly in your terminal.

As you can see, the code is very short and easy to understand. Puppeteer is a very powerful tool, don't hesitate to check out the documentation to discover everything you can do with it!

And to go further, I invite you to read my article titled "5 advanced techniques for building an efficient web crawler".