Need help with node-crawler?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

6.0K Stars 856 Forks MIT License 457 Commits 42 Opened issues


Web Crawler/Spider for NodeJS + server-side jQuery ;-)

Services available


Need anything else?

Contributors list


npm package

build status Coverage Status Dependency Status NPM download NPM quality Gitter

Most powerful, popular and production crawling/scraping package for Node, happy hacking :)


  • Server-side DOM & automatic jQuery insertion with Cheerio (default) or JSDOM,
  • Configurable pool size and retries,
  • Control rate limit,
  • Priority queue of requests,
  • forceUTF8
    mode to let crawler deal for you with charset detection and conversion,
  • Compatible with 4.x or newer version.

Here is the CHANGELOG

Thanks to Authuir, we have a Chinese docs. Other languages are welcomed!

Table of Contents

Get started


$ npm install crawler

Basic usage

const Crawler = require('crawler');

const c = new Crawler({ maxConnections: 10, // This will be called for each crawled page callback: (error, res, done) => { if (error) { console.log(error); } else { const $ = res.$; // $ is Cheerio by default //a lean implementation of core jQuery designed specifically for the server console.log($('title').text()); } done(); } });

// Queue just one URL, with default callback c.queue('');

// Queue a list of URLs c.queue(['','']);

// Queue URLs with custom callbacks & parameters c.queue([{ uri: '', jQuery: false,

// The global callback won't be called
callback: (error, res, done) => {
    if (error) {
    } else {
        console.log('Grabbed', res.body.length, 'bytes');


// Queue some HTML code directly without grabbing (mostly for tests) c.queue([{ html: '

This is a test

' }]);

Slow down


to slow down when you are visiting web sites.
const Crawler = require('crawler');

const c = new Crawler({ rateLimit: 1000, // maxConnections will be forced to 1 callback: (err, res, done) => { console.log(res.$('title').text()); done(); } });

c.queue(tasks);//between two tasks, minimum time gap is 1000 (ms)

Custom parameters

Sometimes you have to access variables from previous request/response session, what should you do is passing parameters as same as options:

    uri: '',
    parameter1: 'value1',
    parameter2: 'value2',
    parameter3: 'value3'

then access them in callback via


Crawler picks options only needed by request, so don't worry about the redundance.

Raw body

If you are downloading files like image, pdf, word etc, you have to save the raw response body which means Crawler shouldn't convert it to string. To make it happen, you need to set encoding to null

const Crawler = require('crawler');
const fs = require('fs');

const c = new Crawler({ encoding: null, jQuery: false,// set false to suppress warning message. callback: (err, res, done) => { if (err) { console.error(err.stack); } else { fs.createWriteStream(res.options.filename).write(res.body); }



c.queue({ uri: '', filename: 'nodejs-1920x1200.png' });


If you want to do something either synchronously or asynchronously before each request, you can try the code below. Note that direct requests won't trigger preRequest.

const c = new Crawler({
    preRequest: (options, done) => {
        // 'options' here is not the 'options' you pass to 'c.queue', instead, it's the options that is going to be passed to 'request' module
        // when done is called, the request will start
    callback: (err, res, done) => {
        if (err) {
        } else {

c.queue({ uri: '', // this will override the 'preRequest' defined in crawler preRequest: (options, done) => { setTimeout(() => { console.log(options); done(); }, 1000); } });


Send request directly

In case you want to send a request directly without going through the scheduler in Crawler, try the code below.

takes the same options as
, please refer to options for detail. The difference is when calling
must be defined explicitly, with two arguments
, which are the same as that of
of method
    uri: '',
    skipEventRequest: false, // default to true, direct requests won't trigger Event:'request'
    callback: (error, response) => {
        if (error) {
        } else {

Work with Http2

Node-crawler now supports http request. Proxy functionality for http2 request does not be included now. It will be added in the future.

    //unit test work with httpbin http2 server. It could be used for test
    uri: '',
    method: 'GET',
    http2: true, //set http2 to be true will make a http2 request
    callback: (error, response, done) => {
        if (error) {
            return done();

    console.log(`inside callback`);
    return done();


Work with bottleneck

Control rate limit for with limiter. All tasks submit to a limiter will abide the

restrictions of the limiter.
is the minimum time gap between two tasks.
is the maximum number of tasks that can be running at the same time. Limiters are independent of each other. One common use case is setting different limiters for different proxies. One thing is worth noticing, when
is set to a non-zero value,
will be forced to 1.
const crawler = require('crawler');

const c = new Crawler({ rateLimit: 2000, maxConnections: 1, callback: (error, res, done) => { if (error) { console.log(error); } else { const $ = res.$; console.log($('title').text()); } done(); } });

// if you want to crawl some website with 2000ms gap between requests c.queue(''); c.queue(''); c.queue('');

// if you want to crawl some website using proxy with 2000ms gap between requests for each proxy c.queue({ uri:'', limiter:'proxy_1', proxy:'proxy_1' }); c.queue({ uri:'', limiter:'proxy_2', proxy:'proxy_2' }); c.queue({ uri:'', limiter:'proxy_3', proxy:'proxy_3' }); c.queue({ uri:'', limiter:'proxy_1', proxy:'proxy_1' });

Normally, all limiter instances in limiter cluster in crawler are instantiated with options specified in crawler constructor. You can change property of any limiter by calling the code below. Currently, we only support changing property 'rateLimit' of limiter. Note that the default limiter can be accessed by

c.setLimiterProperty('default', 'rateLimit', 3000)
. We strongly recommend that you leave limiters unchanged after their instantiation unless you know clearly what you are doing.
const c = new Crawler({});
c.setLimiterProperty('limiterName', 'propertyName', value);


Event: 'schedule'

Emitted when a task is being added to scheduler.

crawler.on('schedule', (options) => {
    options.proxy = 'http://proxy:port';

Event: 'limiterChange'

Emitted when limiter has been changed.

Event: 'request'

Emitted when crawler is ready to send a request.

If you are going to modify options at last stage before requesting, just listen on it.

crawler.on('request', (options) => {
    options.qs.timestamp = new Date().getTime();

Event: 'drain'

Emitted when queue is empty.

crawler.on('drain', () => {
    // For example, release a connection to database.
    db.end();// close connection to MySQL


Enqueue a task and wait for it to be executed.


Size of queue, read-only

Options reference

You can pass these options to the Crawler() constructor if you want them to be global or as items in the queue() calls if you want them to be specific to that item (overwriting global options)

This options list is a strict superset of mikeal's request options and will be directly passed to the request() method.

Basic request options


  • callback(error, res, done)
    : Function that will be called after a request was completed
    • error
      : Error
    • res
      : http.IncomingMessage A response of standard IncomingMessage includes
      • res.statusCode
        : Number HTTP status code. E.G.
      • res.body
        : Buffer | String HTTP response content which could be a html page, plain text or xml document e.g.
      • res.headers
        : Object HTTP response headers
      • res.request
        : Request An instance of Mikeal's
        instead of http.ClientRequest
        • res.request.uri
          : urlObject HTTP request entity of parsed url
        • res.request.method
          : String HTTP request method. E.G.
        • res.request.headers
          : Object HTTP request headers
      • res.options
        : Options of this task
      • $
        : jQuery Selector A selector for html or xml document.
    • done
      : Function It must be called when you've done your work in callback.

Schedule options

  • options.maxConnections
    : Number Size of the worker pool (Default 10).
  • options.rateLimit
    : Number Number of milliseconds to delay between each requests (Default 0).
  • options.priorityRange
    : Number Range of acceptable priorities starting from 0 (Default 10).
  • options.priority
    : Number Priority of this request (Default 5). Low values have higher priority.

Retry options

  • options.retries
    : Number Number of retries if the request fails (Default 3),
  • options.retryTimeout
    : Number Number of milliseconds to wait before retrying (Default 10000),

Server-side DOM options

  • options.jQuery
    : Boolean|String|Object Use
    with default configurations to inject document if true or 'cheerio'. Or use customized
    if an object with Parser options. Disable injecting jQuery selector if false. If you have memory leak issue in your project, use 'whacko', an alternative parser,to avoid that. (Default true)

Charset encoding

  • options.forceUTF8
    : Boolean If true crawler will get charset from HTTP headers or meta tag in html and convert it to UTF8 if necessary. Never worry about encoding anymore! (Default true),
  • options.incomingEncoding
    : String With forceUTF8: true to set encoding manually (Default null) so that crawler will not have to detect charset by itself. For example,
    incomingEncoding: 'windows-1255'
    . See all supported encodings


  • options.skipDuplicates
    : Boolean If true skips URIs that were already crawled, without even calling callback() (Default false). This is not recommended, it's better to handle outside
    use seenreq

Http headers

  • options.rotateUA
    : Boolean If true,
    should be an array and rotate it (Default false)
  • options.userAgent
    : String|Array, If
    is false, but
    is an array, crawler will use the first one.
  • options.referer
    : String If truthy sets the HTTP referer header
  • options.removeRefererHeader
    : Boolean If true preserves the set referer during redirects
  • options.headers
    : Object Raw key-value of http headers


  • options.http2
    : Boolean If true, request will be sent in http2 protocol (Default false)

Https socks5

const Agent = require('socks5-https-client/lib/Agent');
const c = new Crawler({
    // rateLimit: 2000,
    maxConnections: 20,
    agentClass: Agent, //adding socks5 https agent
    method: 'GET',
    strictSSL: true,
    agentOptions: {
        socksHost: 'localhost',
        socksPort: 9050
    // debug: true,
    callback: (error, res, done) => {
        if (error) {

Work with Cheerio or JSDOM

Crawler by default use Cheerio instead of JSDOM. JSDOM is more robust, if you want to use JSDOM you will have to require it

in your own script before passing it to crawler.

Working with Cheerio

jQuery: true //(default)
jQuery: 'cheerio'
jQuery: {
    name: 'cheerio',
    options: {
        normalizeWhitespace: true,
        xmlMode: true

These parsing options are taken directly from htmlparser2, therefore any options that can be used in

are valid in cheerio as well. The default options are:
    normalizeWhitespace: false,
    xmlMode: false,
    decodeEntities: true

For a full list of options and their effects, see this and htmlparser2's options. source

Work with JSDOM

In order to work with JSDOM you will have to install it in your project folder

npm install jsdom
, and pass it to crawler.
const jsdom = require('jsdom');
const Crawler = require('crawler');

const c = new Crawler({ jQuery: jsdom });

How to test

Crawler uses

to mock http request, thus testing no longer relying on http server.
$ npm install
$ npm test
$ npm run cover # code coverage

Alternative: Docker

After installing Docker, you can run:

# Builds the local test environment
$ docker build -t node-crawler .

Runs tests

$ docker run node-crawler sh -c "npm install && npm test"

You can also ssh into the container for easier debugging

$ docker run -i -t node-crawler bash

Rough todolist

  • Introducing zombie to deal with page with complex ajax
  • Refactoring the code to be more maintainable
  • Make Sizzle tests pass (JSDOM bug?
  • Promise support
  • Commander support
  • Middleware support

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.