Setting up a configurable proxy with NodeJS in just a few minutes

In 2019, I worked on a shopping assistant for high-tech products called Kalico, whose goal was to always be up to date on new phone releases, but also on the evolution of their prices.

While trying to retrieve prices using a crawler on e-commerce sites' pages, I quickly ran into a problem I had never encountered before. Some sites tried to detect the geographic location of my "fake" client in order to tell them whether the product could be delivered to their area.

The problem was that my api was running on a Heroku server, located in France, and when I tried to crawl the links of UK and US stores, the products were always unavailable for shipping (logically) and therefore refused to display the local price.

My solution was therefore to set up three Amazon machines located in France, the United Kingdom and the United States in order to run simple proxies to bypass the platforms' localization.

After testing several Javascript libraries, running under NodeJS (in order to keep a homogeneous infrastructure), I discovered a complete, open-source, and very easily deployable solution: Anyproxy.

Anyproxy

Anyproxy is (as its name suggests) a fully configurable proxy, developed and maintained by the Chinese e-commerce giant: Alibaba.

Installation can be done directly from npm and starting the software only requires a single command line:

$ npm install -g anyproxy
$ anyproxy

Note: AnyProxy is configured to listen by default on port 8001 (proxy) and 8002 (interface), be sure to check that your ports are open if you are on a remote server

Once launched, you can access the web interface on your http://localhost:8002 and start passing requests through your proxy!

AnyProxy's specifics

AnyProxy is highly configurable, notably for:

• SSL certificate support and https requests
• Adding specific routing rules using a simple .js file
• Being launched inside an application, like a classic NodeJS module

To learn how to configure your proxy, I'll refer you to the official documentation: http://anyproxy.io/en/

Note that I saw a few bits of text still written in Chinese but never in the technical documentation.

Security

If you leave remote access to your proxy open, I strongly advise you to not open the web interface port in production since there is no authentication.

But above all, remember to never leave an open proxy running without a filter, I explained the possible consequences in my article "The day Amazon put the pressure on me...", the simplest solution is to add an authentication rule like the one below:

//rule.js
module.exports = {
  // introduction
  summary: 'Authentication',
  // intercept before send request to server
  beforeSendRequest(requestDetails) {
    console.log(requestDetails.requestOptions.headers);
    if(requestDetails.requestOptions.headers.authorization !== 'Basic test:token'){
      return {
        response: {
          statusCode: 401,
          header: { 'content-type': 'text/html' },
          body: 'You are not authorized to use this proxy'
        }
      };
    }
    return {};
  }
};

To activate this rule, all you have to do is restart AnyProxy with a --rule parameter as shown below!

$ anyproxy --rule ./rule.js