An Introduction To Web Scraping With Node.js (JavaScript)

Are you looking for ways to make use of new technologies? Web extraction or web scraping provides you with a way to collect structured web data in an automated manner. For instance, web scraping is used in the E-commerce world to monitor the pricing of competitors’ services and products. Other use cases of web scraping are given below.

You will learn how to do web scraping with Node.js. But first, we look into what Node.js is. It is an open-source server-side platform to develop server-side and networking applications. Its library is very fast in code execution because its applications do not buffer any data. They simply output the data in chunks. Given below are some of the areas where we can use Node.js.

​​Following are the reasons to use Node.js for web scraping.

npm install --save request request-promise cheerio

const rp = require('request-promise');
const cheerio = require('cheerio');

const options = {
 uri: `https://www.yourURLhere.com`,
 transform: function (body) {
   return cheerio.load(body);
 }
};

You can make your request using the code below.

rp(OPTIONS)
   .then(function (data) {
       // REQUEST SUCCEEDED: DO SOMETHING
   })
   .catch(function (err) {
       // REQUEST FAILED: ERROR OF SOME KIND
   });

We are passing the options object to request-promise in the above code. We then wait to see if our request succeeds or fails. 

Now, we create our own version of the code in which we use arrow functions as shown below.

rp(options)
 .then(($) => {
   console.log($);
 })
 .catch((err) => {
   console.log(err);
 });

{ [Function: initialize]
 fn:
  initialize {
    constructor: [Circular],
    _originalRoot:
     { type: 'root',
       name: 'root',
       namespace: 'http://www.w3.org/1999/xhtml',
       attribs: {},
       ...

This is how you can make your first scrape using Node.js. Here’s the full code we did so far.

const rp = require('request-promise');
const cheerio = require('cheerio');
const options = {
  uri: `https://www.google.com`,
  transform: function (body) {
    return cheerio.load(body);
  }
};

rp(options)
  .then(($) => {
    console.log($);
  })
  .catch((err) => {
    console.log(err);
  });

You can use cheerio to extract the data you want. The implementation of cheerio’s selector is nearly the same as that of jQuery’s. You can use the selector method for traversing and selecting elements in the document. You can also use it to get and set data. For instance, you want to scrape the following HTML of a website.

<ul id="cities">
 <li class="large">Europe</li>
 <li id="medium">Turkey</li>
 <li class="small">Salem</li>
</ul>

$('.large').text()
// Europe
$('#medium').text()
// Turkey
$('li[class=small]').html()

$('li').each(function(i, elem) {
 cities[i] = $(this).text();
});

This is how you can scrape data from websites using Node.js. You can also use additional methods for extracting the data of child elements of a list or the HTML of a website.

You know that proxies act as intermediaries between clients who request resources and the server who provides the resources. There are three different types of proxies, as shown below.

import requests
proxy = 'http://114.121.248.251:8080'
url = 'https://ipecho.net/plain'
# Sending a GET request to the url and
# passing the proxy as a parameter.
page = requests.get(url,
                   proxies={"http": proxy, "https": proxy})
 # Printing the content of the requested url.
print(page.text)

You will get the below output.