node-fetch

by node-fetch

node-fetch / node-fetch

A light-weight module that brings Fetch API to Node.js

5.7K Stars 599 Forks Last release: about 2 months ago (v2.6.1) MIT License 540 Commits 41 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Node Fetch

A light-weight module that brings Fetch API to Node.js.

Build status Coverage status Current version Install size Mentioned in Awesome Node.js Discord

Consider supporting us on our Open Collective:

Open Collective

Motivation

Instead of implementing

XMLHttpRequest
in Node.js to run browser-specific Fetch polyfill, why not go from native
http
to
fetch
API directly? Hence,
node-fetch
, minimal code for a
window.fetch
compatible API on Node.js runtime.

See Jason Miller's isomorphic-unfetch or Leonardo Quixada's cross-fetch for isomorphic usage (exports

node-fetch
for server-side,
whatwg-fetch
for client-side).

Features

  • Stay consistent with
    window.fetch
    API.
  • Make conscious trade-off when following WHATWG fetch spec and stream spec implementation details, document known differences.
  • Use native promise and async functions.
  • Use native Node streams for body, on both request and response.
  • Decode content encoding (gzip/deflate/brotli) properly, and convert string output (such as
    res.text()
    and
    res.json()
    ) to UTF-8 automatically.
  • Useful extensions such as redirect limit, response size limit, explicit errors for troubleshooting.

Difference from client-side fetch

  • See known differences:
  • If you happen to use a missing feature that
    window.fetch
    offers, feel free to open an issue.
  • Pull requests are welcomed too!

Installation

Current stable release (

3.x
)
$ npm install node-fetch

Loading and configuring the module

// CommonJS
const fetch = require('node-fetch');

// ES Module import fetch from 'node-fetch';

If you want to patch the global object in node:

const fetch = require('node-fetch');

if (!globalThis.fetch) { globalThis.fetch = fetch; }

For versions of Node earlier than 12, use this

globalThis
polyfill.

Upgrading

Using an old version of node-fetch? Check out the following files:

Common Usage

NOTE: The documentation below is up-to-date with

3.x
releases, if you are using an older version, please check how to upgrade.

Plain text or HTML

const fetch = require('node-fetch');

(async () => { const response = await fetch('https://github.com/'); const body = await response.text();

console.log(body);

})();

JSON

const fetch = require('node-fetch');

(async () => { const response = await fetch('https://api.github.com/users/github'); const json = await response.json();

console.log(json);

})();

Simple Post

const fetch = require('node-fetch');

(async () => { const response = await fetch('https://httpbin.org/post', {method: 'POST', body: 'a=1'}); const json = await response.json();

console.log(json);

})();

Post with JSON

const fetch = require('node-fetch');

(async () => { const body = {a: 1};

const response = await fetch('https://httpbin.org/post', {
    method: 'post',
    body: JSON.stringify(body),
    headers: {'Content-Type': 'application/json'}
});
const json = await response.json();

console.log(json);

})();

Post with form parameters

URLSearchParams
is available on the global object in Node.js as of v10.0.0. See official documentation for more usage methods.

NOTE: The

Content-Type
header is only set automatically to
x-www-form-urlencoded
when an instance of
URLSearchParams
is given as such:
const fetch = require('node-fetch');

const params = new URLSearchParams(); params.append('a', 1);

(async () => { const response = await fetch('https://httpbin.org/post', {method: 'POST', body: params}); const json = await response.json();

console.log(json);

})();

Handling exceptions

NOTE: 3xx-5xx responses are NOT exceptions, and should be handled in

then()
, see the next section.

Wrapping the fetch function into a

try/catch
block will catch all exceptions, such as errors originating from node core libraries, like network errors, and operational errors which are instances of FetchError. See the error handling document for more details.
const fetch = require('node-fetch');

try { fetch('https://domain.invalid/'); } catch (error) { console.log(error); }

Handling client and server errors

It is common to create a helper function to check that the response contains no client (4xx) or server (5xx) error responses:

const fetch = require('node-fetch');

const checkStatus = res => { if (res.ok) { // res.status >= 200 && res.status < 300 return res; } else { throw MyCustomError(res.statusText); } }

(async () => { const response = await fetch('https://httpbin.org/status/400'); const data = checkStatus(response);

console.log(data); //=&gt; MyCustomError

})();

Handling cookies

Cookies are not stored by default. However, cookies can be extracted and passed by manipulating request and response headers. See Extract Set-Cookie Header for details.

Advanced Usage

Streams

The "Node.js way" is to use streams when possible. You can pipe

res.body
to another stream. This example uses stream.pipeline to attach stream error handlers and wait for the download to complete.
const util = require('util');
const fs = require('fs');
const streamPipeline = util.promisify(require('stream').pipeline);

(async () => { const response = await fetch('https://assets-cdn.github.com/images/modules/logos_page/Octocat.png');

if (response.ok) {
    return streamPipeline(response.body, fs.createWriteStream('./octocat.png'));
}

throw new Error(`unexpected response ${response.statusText}`);

})();

Buffer

If you prefer to cache binary data in full, use buffer(). (NOTE: buffer() is a

node-fetch
only API)
const fetch = require('node-fetch');
const fileType = require('file-type');

(async () => { const response = await fetch('https://octodex.github.com/images/Fintechtocat.png'); const buffer = await response.buffer(); const type = fileType.fromBuffer(buffer)

console.log(type);

})();

Accessing Headers and other Meta data

const fetch = require('node-fetch');

(async () => { const response = await fetch('https://github.com/');

console.log(response.ok);
console.log(response.status);
console.log(response.statusText);
console.log(response.headers.raw());
console.log(response.headers.get('content-type'));

})();

Extract Set-Cookie Header

Unlike browsers, you can access raw

Set-Cookie
headers manually using
Headers.raw()
. This is a
node-fetch
only API.
const fetch = require('node-fetch');

(async () => { const response = await fetch('https://example.com');

// Returns an array of values, instead of a string of comma-separated values
console.log(response.headers.raw()['set-cookie']);

})();

Post data using a file stream

const {createReadStream} = require('fs');
const fetch = require('node-fetch');

const stream = createReadStream('input.txt');

(async () => { const response = await fetch('https://httpbin.org/post', {method: 'POST', body: stream}); const json = await response.json();

console.log(json)

})();

Post with form-data (detect multipart)

const fetch = require('node-fetch');
const FormData = require('form-data');

const form = new FormData(); form.append('a', 1);

(async () => { const response = await fetch('https://httpbin.org/post', {method: 'POST', body: form}); const json = await response.json();

console.log(json)

})();

// OR, using custom headers // NOTE: getHeaders() is non-standard API

const options = { method: 'POST', body: form, headers: form.getHeaders() };

(async () => { const response = await fetch('https://httpbin.org/post', options); const json = await response.json();

console.log(json)

})();

node-fetch also supports spec-compliant FormData implementations such as formdata-node:

const fetch = require('node-fetch');
const FormData = require('formdata-node');

const form = new FormData(); form.set('greeting', 'Hello, world!');

fetch('https://httpbin.org/post', {method: 'POST', body: form}) .then(res => res.json()) .then(json => console.log(json));

Request cancellation with AbortSignal

You may cancel requests with

AbortController
. A suggested implementation is
abort-controller
.

An example of timing out a request after 150ms could be achieved as the following:

const fetch = require('node-fetch');
const AbortController = require('abort-controller');

const controller = new AbortController(); const timeout = setTimeout(() => { controller.abort(); }, 150);

(async () => { try { const response = await fetch('https://example.com', {signal: controller.signal}); const data = await response.json();

    useData(data);
} catch (error) {
    if (error.name === 'AbortError') {
        console.log('request was aborted');
    }
} finally {
    clearTimeout(timeout);
}

})();

See test cases for more examples.

API

fetch(url[, options])

  • url
    A string representing the URL for fetching
  • options
    Options for the HTTP(S) request
  • Returns: Promise<Response>

Perform an HTTP(S) fetch.

url
should be an absolute url, such as
https://example.com/
. A path-relative URL (
/file/under/root
) or protocol-relative URL (
//can-be-http-or-https.com/
) will result in a rejected
Promise
.

Options

The default values are shown after each option key.

{
    // These properties are part of the Fetch Standard
    method: 'GET',
    headers: {},            // Request headers. format is the identical to that accepted by the Headers constructor (see below)
    body: null,             // Request body. can be null, a string, a Buffer, a Blob, or a Node.js Readable stream
    redirect: 'follow',     // Set to `manual` to extract redirect headers, `error` to reject redirect
    signal: null,           // Pass an instance of AbortSignal to optionally abort requests

// The following properties are node-fetch extensions
follow: 20,             // maximum redirect count. 0 to not follow redirect
compress: true,         // support gzip/deflate content encoding. false to disable
size: 0,                // maximum response body size in bytes. 0 to disable
agent: null,            // http(s).Agent instance or function that returns an instance (see below)
highWaterMark: 16384,   // the maximum number of bytes to store in the internal buffer before ceasing to read from the underlying resource.
insecureHTTPParser: false   // Use an insecure HTTP parser that accepts invalid HTTP headers when `true`.

}

Default Headers

If no values are set, the following request headers will be sent automatically:

| Header | Value | | ------------------- | ------------------------------------------------------ | |

Accept-Encoding
|
gzip,deflate,br
(when
options.compress === true
)
| |
Accept
|
*/*
| |
Connection
|
close
(when no
options.agent
is present)
| |
Content-Length
| (automatically calculated, if possible) | |
Transfer-Encoding
|
chunked
(when
req.body
is a stream)
| |
User-Agent
|
node-fetch
|

Note: when

body
is a
Stream
,
Content-Length
is not set automatically.

Custom Agent

The

agent
option allows you to specify networking related options which are out of the scope of Fetch, including and not limited to the following:
  • Support self-signed certificate
  • Use only IPv4 or IPv6
  • Custom DNS Lookup

See

http.Agent
for more information.

In addition, the

agent
option accepts a function that returns
http
(s)
.Agent
instance given current URL, this is useful during a redirection chain across HTTP and HTTPS protocol.
const http = require('http');
const https = require('https');

const httpAgent = new http.Agent({ keepAlive: true }); const httpsAgent = new https.Agent({ keepAlive: true });

const options = { agent: function(_parsedURL) { if (_parsedURL.protocol == 'http:') { return httpAgent; } else { return httpsAgent; } } };

Custom highWaterMark

Stream on Node.js have a smaller internal buffer size (16kB, aka

highWaterMark
) from client-side browsers (>1MB, not consistent across browsers). Because of that, when you are writing an isomorphic app and using
res.clone()
, it will hang with large response in Node.

The recommended way to fix this problem is to resolve cloned response in parallel:

const fetch = require('node-fetch');

(async () => { const response = await fetch('https://example.com'); const r1 = await response.clone();

return Promise.all([res.json(), r1.text()]).then(results =&gt; {
    console.log(results[0]);
    console.log(results[1]);
});

})();

If for some reason you don't like the solution above, since

3.x
you are able to modify the
highWaterMark
option:
const fetch = require('node-fetch');

(async () => { const response = await fetch('https://example.com', { // About 1MB highWaterMark: 1024 * 1024 });

return res.clone().buffer();

})();

Insecure HTTP Parser

Passed through to the

insecureHTTPParser
option on http(s).request. See
http.request
for more information.

Class: Request

An HTTP(S) request containing information about URL, method, headers, and the body. This class implements the Body interface.

Due to the nature of Node.js, the following properties are not implemented at this moment:

  • type
  • destination
  • referrer
  • referrerPolicy
  • mode
  • credentials
  • cache
  • integrity
  • keepalive

The following node-fetch extension properties are provided:

  • follow
  • compress
  • counter
  • agent
  • highWaterMark

See options for exact meaning of these extensions.

new Request(input[, options])

(spec-compliant)

  • input
    A string representing a URL, or another
    Request
    (which will be cloned)
  • options
    [Options][#fetch-options] for the HTTP(S) request

Constructs a new

Request
object. The constructor is identical to that in the browser.

In most cases, directly

fetch(url, options)
is simpler than creating a
Request
object.

Class: Response

An HTTP(S) response. This class implements the Body interface.

The following properties are not implemented in node-fetch at this moment:

  • Response.error()
  • Response.redirect()
  • type
  • trailer

new Response([body[, options]])

(spec-compliant)

Constructs a new

Response
object. The constructor is identical to that in the browser.

Because Node.js does not implement service workers (for which this class was designed), one rarely has to construct a

Response
directly.

response.ok

(spec-compliant)

Convenience property representing if the request ended normally. Will evaluate to true if the response status was greater than or equal to 200 but smaller than 300.

response.redirected

(spec-compliant)

Convenience property representing if the request has been redirected at least once. Will evaluate to true if the internal redirect counter is greater than 0.

Class: Headers

This class allows manipulating and iterating over a set of HTTP headers. All methods specified in the Fetch Standard are implemented.

new Headers([init])

(spec-compliant)

  • init
    Optional argument to pre-fill the
    Headers
    object

Construct a new

Headers
object.
init
can be either
null
, a
Headers
object, an key-value map object or any iterable object.
// Example adapted from https://fetch.spec.whatwg.org/#example-headers-class
const { Headers } = require('node-fetch');

const meta = { 'Content-Type': 'text/xml', 'Breaking-Bad': '<3' }; const headers = new Headers(meta);

// The above is equivalent to const meta = [['Content-Type', 'text/xml'], ['Breaking-Bad', '<3']]; const headers = new Headers(meta);

// You can in fact use any iterable objects, like a Map or even another Headers const meta = new Map(); meta.set('Content-Type', 'text/xml'); meta.set('Breaking-Bad', '<3'); const headers = new Headers(meta); const copyOfHeaders = new Headers(headers);

Interface: Body

Body
is an abstract interface with methods that are applicable to both
Request
and
Response
classes.

The following methods are not yet implemented in node-fetch at this moment:

  • formData()

body.body

(deviation from spec)

Data are encapsulated in the

Body
object. Note that while the Fetch Standard requires the property to always be a WHATWG
ReadableStream
, in node-fetch it is a Node.js
Readable
stream
.

body.bodyUsed

(spec-compliant)

  • Boolean

A boolean property for if this body has been consumed. Per the specs, a consumed body cannot be used again.

body.arrayBuffer()

body.blob()

body.json()

body.text()

(spec-compliant)

  • Returns:
    Promise

Consume the body and return a promise that will resolve to one of these formats.

body.buffer()

(node-fetch extension)

  • Returns:
    Promise

Consume the body and return a promise that will resolve to a Buffer.

Class: FetchError

(node-fetch extension)

An operational error in the fetching process. See ERROR-HANDLING.md for more info.

Class: AbortError

(node-fetch extension)

An Error thrown when the request is aborted in response to an

AbortSignal
's
abort
event. It has a
name
property of
AbortError
. See ERROR-HANDLING.MD for more info.

TypeScript

Since

3.x
types are bundled with
node-fetch
, so you don't need to install any additional packages.

For older versions please use the type definitions from DefinitelyTyped:

$ npm install --save-dev @types/node-fetch

Acknowledgement

Thanks to github/fetch for providing a solid implementation reference.

Team

| David Frank | Jimmy Wärting | Antoni Kepinski | Richie Bendall | Gregor Martynus | | ----------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------- | | David Frank | Jimmy Wärting | Antoni Kepinski | Richie Bendall | Gregor Martynus |

Former

License

MIT

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.