# 5 Advanced Techniques for Building an Effective Web Crawler

Two weeks ago I published an article titled "How to crawl a web page in NodeJS with Puppeteer and Cheerio" in which I provided a basic crawler snippet.

If you're not yet familiar with website crawling, I recommend reading it before continuing with this article!

As you'll have understood, the method for retrieving information from a random website is very simple, but if you want to be able to crawl larger sites (or simply a wider range of them), you risk running into some technical difficulties.

For my part, when I set up my Kalico project in 2018, my goal was to retrieve the prices of numerous items across different e-commerce platforms, including giants like Amazon and eBay.

The problem

For an e-commerce site (in particular), it's normal to do everything possible to prevent outside scripts from automatically collecting a large amount of information.

At this point in time, data represents a significant part of companies' value, especially for tech giants.

This is why most of these companies implement technical measures to prevent this kind of collection as much as possible. The problem is that by putting overly strict filters in place, these platforms could risk accidentally blocking real users, which they can't afford to do.

Each platform will therefore rely on various data and metadata left by a "visitor" to try to detect whether or not this is a real person or a script.

On our end, our goal will be to play a game of cat and mouse, using techniques like the ones below to allow us to pass for a regular user most of the time.

Technical solutions

1 - Rotating the user-agent

The user-agent is a piece of metadata present in an HTTP request header that provides information about the browser and its version, the hardware, the operating system, etc.

For example, my current user-agent is the following: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:81.0) Gecko/20100101 Firefox/81.0

Some sites will cross-reference several pieces of metadata like this one (in addition to the IP) in order to detect if a user is behind too large a number of requests, for example.

This is notably what's called "fingerprinting," which I discussed in a previous article.

To counter this practice, the first method is very simple and consists of injecting a different user-agent for each new request, taken from a predefined list or randomly generated on demand.

Example:

"Mozilla/5.0 (iPhone; CPU iPhone OS 11_4_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.0 Mobile/15E148 Safari/604.1"
"Mozilla/5.0 (iPhone; CPU iPhone OS 9_0_1 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13A404 Safari/601.1"
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36"
"Mozilla/5.0 (Linux; U; Android 2.2) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"
"Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_6; en-en) AppleWebKit/533.19.4 (KHTML, like Gecko) Version/5.0.3 Safari/533.19.4"

2 - Using multiple proxies

Obviously, even if the user-agents are different, it's very easy for a site to block an address when it detects too large a number of requests coming from a single IP.

It's therefore preferable to have a list of proxies that we'll assign to different instances of the headless browser (puppeteer) in order to prevent the server's address from being detected within a few minutes.

Proxies can come from free lists found online, or you can also set them up yourself using cloud instances, for example, which notably allows you to achieve better performance.

An additional strength of proxies: it's possible to use different server locations (such as different countries), which is what I did to make it easier to retrieve prices in the correct currencies on Amazon!

3 - Random click delays

Once all the crawler's information has been "anonymized" prior to loading the page, it's much more complicated for a site to detect a bot.

But some sites use behavioral analysis to detect whether the user is acting "like a human" or not.

For example, a visitor who clicks on 30 links in 5 seconds becomes suspicious because it's humanly impossible, but the time between each click matters too!

Because yes, if you tell your script to wait 5 seconds between each click, the site will be able to detect that these actions were automated. The solution is therefore to add a random delay (for example, between 5 and 20 seconds) for each click.

4 - Requesting HTML in different ways

Some sites don't rely on behavior detection or fingerprinting, but instead focus their efforts on making the page's content difficult to retrieve.

For example, you might come across sites that generate their element ids and classes on the fly with each new request, making the process of selecting an element very complicated.

One solution is to analyze beforehand what the site can return and prepare several requests (with Cheerio, for example) to test different page layouts.

It's even sometimes possible to come across sites that are almost impossible to crawl automatically!

5 - Avoiding honeypots

The honeypot technique consists of leaving an unsecured element that a regular user would never interact with but which will serve to trap a bot.

For crawlers, this consists of an invisible link on the page that will instantly block the crawler if it clicks on it. To avoid these honeypots, you can:

• Avoid links hidden by CSS (opacity:0, display: none, or visibility: hidden)
• Avoid empty links (with no content)
• Avoid links containing a 1px by 1px image

Honeypots are hard to avoid, but if you implement the 4 previous techniques, you will have already avoided 99% of your crawlers' problems!