Need help with node2nix?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.
svanderburg

Description

Generate Nix expressions to build NPM packages

231 Stars 53 Forks MIT License 197 Commits 64 Opened issues

Services available

Need anything else?

node2nix

Deploy NPM Package Manager (NPM) packages with the Nix package manager!

Using

node2nix
instead of the "vanilla" NPM is useful for a variety of reasons: * To deploy NPM packages on NixOS and to manage complex software installations (that include non-NPM managed dependencies) by using a universal deployment solution (Nix). * To integrate with other tools in the Nix-ecosystem: NixOS to manage an entire system from a single declarative specification, NixOps to deploy networks of machines (bare metal and in the cloud), and Disnix to manage service-oriented systems.

Table of Contents

Table of Contents

Prerequisites

To be able to convert Git dependencies, the presence of the

nix-hash
command-line utility (that is included with the Nix package manager) is required.

Installation

There are two ways this package can installed.

To install this package through the Nix package manager, obtain a copy of Nixpkgs and run:

$ nix-env -f '' -iA nodePackages.node2nix

Alternatively, this package can also be installed through NPM by running:

$ npm install -g node2nix

Building a development version

A development version can be deployed by checking out the Git repository and running:

$ nix-env -f release.nix -iA package.x86_64-linux

The above command installs the development

node2nix
executable into the Nix profile of the user.

Alternatively, you can the use NPM to install the project dependencies:

$ npm install

The project only uses JavaScript dependencies and, as such, will also work on NixOS.

Usage

node2nix
can be used for a variety of use cases.

Deploying a Node.js development project

The primary use case of

node2nix
is to deploy a development project as a NPM package.

What Node.js developers typically do in a development setting is opening the source code folder and running:

$ npm install

The above command-line instruction deploys all dependencies declared in the

package.json
configuration file so that the application can be run.

With

node2nix
you can use the Nix package manager for exactly the same purpose. Running the following command generates a collection of Nix expressions from
package.json
:
$ node2nix

The above command generates three files:

node-packages.nix
capturing the packages that can be deployed including all its required dependencies,
node-env.nix
contains the build logic and
default.nix
is a composition expression allowing users to deploy the package.

By running the following Nix command with these expressions, the project can be built:

$ nix-build -A package

The above instruction places a

result
symlink in the current working dir pointing to the build result. An executable (that is part of the project) can be started as follows:
$ ./result/bin/node2nix

Generating a tarball from a Node.js development project

The expressions that are generated by

node2nix
(shown earlier) can also be used to generate a tarball from the project:
$ nix-build -A tarball

The above command-line instruction produces a tarball that is placed in the following location:

$ ls result/tarballs/node2nix-1.0.1.tgz

The above tarball can be distributed to any user that has the NPM package manager installed, and installed by running:

$ npm install node2nix-1.0.1.tgz

Deploying a development environment of a Node.js development project

Besides deploying a development project, it may also be useful to only install the project's dependencies and spawning a shell session in which they can be found.

The following command-line instruction uses the earlier generated expressions to deploy all the dependencies and opens a development environment:

$ nix-shell -A shell

Within this shell session, files can be modified and run without any hassle. For example, the following command should work:

$ node bin/node2nix.js --help

Using the Node.js environment in other Nix derivations

The

node_modules
environment generated by
node2nix
can also be used in downstream Nix expressions. This can be useful when you want to build a project that requires running a build system such as Webpack in your Node.js environment.

This environment can be found on the

nodeDependencies
attribute of the generated output. It contains two important paths:
lib/node_modules
and
bin
. The former should be copied or symlinked into your build directory, and the latter should be added to the PATH. (You can also use the
NODE_PATH
environment variable, but see the warnings about that below.)

Here's an example derivation showing this technique:

let
  nodeDependencies = (pkgs.callPackage ./my-app {}).nodeDependencies;

in

stdenv.mkDerivation { name = "my-webpack-app"; src = ./my-app; buildInputs = [nodejs]; buildPhase = '' ln -s ${nodeDependencies}/lib/node_modules ./node_modules export PATH="${nodeDependencies}/bin:$PATH"

# Build the distribution bundle in "dist"
webpack
cp -r dist $out/

''; }

Deploying a collection of NPM packages from the NPM registry

The secondary use of

node2nix
is deploying existing NPM packages from the NPM registry.

Deployment of packages from the registry is driven by a JSON specification that looks as follows:

[
  "async",
  "underscore",
  "slasp",
  { "mocha" : "1.21.x" },
  { "mocha" : "1.20.x" },
  { "nijs": "0.0.18" },
  { "node2nix": "git://github.com/svanderburg/node2nix.git" }
]

The above specification is basically an array of objects. For each element that is a string, the

latest
version is obtained from the registry.

To obtain a specific version of a package, an object must defined in which the keys are the name of the packages and the values the versions that must be obtained.

Any version specification that NPM supports can be used, such as version numbers, version ranges, HTTP(S) URLs, Git URLs, and GitHub identifiers.

Nix expressions can be generated from this JSON specification as follows:

$ node2nix -i node-packages.json

And by using the generated Nix expressions, we can install

async
with Nix as follows:
$ nix-env -f default.nix -iA async

For every package for which the latest version has been requested, we can directly refer to the name of the package to deploy it.

For packages for which a specific version has been specified, we must refer to it using an attribute that name that is composed of its name and version specifier.

The following command can be used to deploy the first specific version of

mocha
declared in the JSON configuration:
$ nix-env -f default.nix -iA '"mocha-1.21.x"'

node2nix
can be referenced as follows:
$ nix-env -f default.nix -iA '"node2nix-git://github.com/svanderburg/node2nix.git"'

Since every NPM package resolves to a package name and version number we can also deploy any package by using an attribute consisting of its name and resolved version number.

This command deploys NiJS version 0.0.18:

$ nix-env -f default.nix -iA '"nijs-0.0.18"'

Using NPM lock files

Node.js 8.x and higher (that includes npm 5.x or higher) can also work with so-called lock files that pinpoint the exact versions used of all dependencies and transitive dependencies, and provides a content addressable cache to speed up deployments.

Some Node.js development projects may include a

package-lock.json
file pinpointing the exact versions of the dependencies and transitive dependencies.
node2nix
can use this file to generate a Nix expression from it so that Nix uses the exact same packages:
$ node2nix -l package-lock.json

Generating packages for specific Node.js versions

By default,

node2nix
generates Nix expressions that should be used in conjuction with Node.js 12.x, which is currently the oldest supported LTS release.

When it is desired, it is also possible to generate expressions for other Node.js versions. For example, Node.js 4.x does not use a flattening/deduplication algorithm, and 6.x that does not support lock files or caching.

The old non-flattening structure can be simulated by adding the

--no-flatten
parameter.

Additionally, to enable all flags to make generation for a certain Node.js work,

node2nix
provides convenience parameters. For example, by using the
-4
parameter, we can generate expressions that can be used with Node.js 4.x:
$ node2nix -4 -i node-package.json

By running the following command, Nix deploys NiJS version 0.0.18 using Node.js 4.x and npm 2.x:

$ nix-env -f default.nix -iA '"nijs-0.0.18"'

Advanced options

node2nix
also has a number of advanced options.

Development mode

By default, NPM packages are deployed in production mode, meaning that the development dependencies are not installed by default. By adding the

--development
command line option, you can also deploy the development dependencies:
$ node2nix --development

Specifying paths

If no options are specified,

node2nix
makes implicit assumptions on the filenames of the input JSON specification and the output Nix expressions. These filenames can be modified with command-line options:
$ node2nix --input package.json --output registry.nix --composition default.nix --node-env node-env.nix

Using alternative NPM registries

You can also use an alternative NPM registry (such as a private one), by adding the

--registry
,
--registry-auth-token
and
--registry-scope
option:
$ node2nix -i node-packages.json --registry http://private.registry.local


$ node2nix
--registry "https://registry.npmjs.org"
--registry "https://npm.pkg.github.com/"
--registry-auth-token "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
--registry-scope "@myorg"

Adding unspecified dependencies

A few exotic NPM packages may have dependencies on native libraries that reside somewhere on the user's host system. Unfortunately, NPM's metadata does not specify them, and as a consequence, it may result in failing Nix builds due to missing dependencies.

As a solution, the generated expressions by

node2nix
are made overridable. The override mechanism can be used to manually inject additional unspecified dependencies.

The easiest way to do this is to create a wrapper Nix expression that imports the generated composition expression from

node2nix
and injects additional dependencies.

Consider the following package collection file (named:

node-packages.json
) that installs one NPM package named
floomatic
:
[
  "floomatic"
]

We can generate Nix expressions from the above specification, by running:

$ node2nix -i node-packages.json

One of floomatic's dependencies is an NPM package named

native-diff-match-patch
that requires the Qt 4.x library and pkgconfig, which are native dependencies not detected by the
node2nix
generator.

With the following wrapper expression (named:

override.nix
), we can inject these dependencies ourselves:
{pkgs ? import  {
    inherit system;
}, system ? builtins.currentSystem}:

let nodePackages = import ./default.nix { inherit pkgs system; }; in nodePackages // { floomatic = nodePackages.floomatic.override { buildInputs = [ pkgs.pkgconfig pkgs.qt4 ]; }; }

The expression does the following:

  • We import the composition expression (
    default.nix
    ) generated by
    node2nix
    .
  • We take the old derivation that builds the
    floomatic
    package, and we add the missing native dependencies as build inputs by defining an override.

With the above wrapper expression, we can correctly deploy floomatic, by running:

$ nix-build override.nix -A floomatic

Wrapping or patching the code or any of its dependencies

Some packages or any of its dependencies may also require some ad-hoc fixes to make them work. In such cases, we can implement a

preRebuild
hook with shell instructions that will be executed before the builder will run
npm rebuild
and
npm install
.

For example, consider the

dnschain
package:
[
  "dnschain"
]

We can generate Nix expressions from the above specification, by running:

$ node2nix -i node-packages.json

dnschain
has a practical problem -- it requires OpenSSL to be in the
PATH
of the user. We can create an
override.nix
expression implementing a
preRebuild
hook that wraps the executable in a script that adds
openssl
to the
PATH
:
{pkgs ? import  {
  inherit system;
}, system ? builtins.currentSystem}:

let nodePackages = import ./default.nix { inherit pkgs system; }; in nodePackages // { dnschain = nodePackages.floomatic.override { preRebuild = '' wrapProgram $out/bin/dnschain --suffix PATH : ${pkgs.openssl.bin}/bin ''; }; }

With the above wrapper expression, we can deploy a wrapped

dnschain
(that is able to find the
openssl
executable), by running:
$ nix-build override.nix -A dnschain

Adding additional/global NPM packages to a packaging process

Sometimes it may also be required to supplement a packaging process with additional NPM packages. For example, when building certain NPM projects, some dependencies have to be installed globally.

A prominent example of such a workflow is a Grunt project. The grunt CLI is typically installed globally, whereas its plugins are installed as development dependencies.

We can automate such a workflow as follows. Consider the following

package.json
example:
{
  "name": "grunt-test",
  "version": "0.0.1",
  "private": "true",
  "devDependencies": {
    "grunt": "*",
    "grunt-contrib-jshint": "*",
    "grunt-contrib-watch": "*"
  }
}

The above configuration declares grunt and two grunt plugins (

jshint
and
watch
) as development dependencies.

We can create a supplemental package specification that refers to additional NPM packages that are supposed to be installed globally:

[
  "grunt-cli"
]

The above configuration (

supplement.json
) states that we need the
grunt-cli
as an additional package, installed globally.

Furtheremore, you can provide specific versions of supplemental packages. Here is example of

supplement.json
with
grunt-cli
version
1.2.0
:
[
  {
    "grunt-cli": "1.2.0"
  }
]

Running the following command-line instruction generates the Nix expressions for the project:

$ node2nix -d -i package.json --supplement-input supplement.json

By overriding the generated expressions, we can instruct the builder to execute

grunt
after the dependencies have been deployed:
{ pkgs ? import  {}
, system ? builtins.currentSystem
}:

let nodePackages = import ./default.nix { inherit pkgs system; }; in nodePackages // { package = nodePackages.package.override { postInstall = "grunt"; }; }

The above expression (

override.nix
) defines a
postInstall
hook that executes grunt after the NPM package has been deployed.

Running the following command executes the packaging process, including the grunt post-processing step:

$ nix-build override.nix -A package

Using private Git repositories

In some development projects, it may be desired to deploy private Git repositories as dependencies. The

fetchgit {}
function in Nixpkgs, however, only supports public repositories.

It is also possible to instruct the generator to use the

fetchgitPrivate {}
function, that adds support for private repositories that can be reached with SSH:
$ node2nix --use-fetchgit-private

Before running the

node2nix
command shown above, you probably want to set up
ssh-agent
first and use
ssh-add
to add a private key to the keychain to prevent the generator from asking for passphrases.

When deploying a project or package, you need to pass an additional parameter that provides an SSH configuration file with a reference to an identify file. The following SSH config file (e.g.

~/ssh_config
) suffices for me:
StrictHostKeyChecking=no
UserKnownHostsFile /dev/null
IdentityFile ~/id_rsa

When deploying a package with Nix, you must propagate the location of the SSH config file as a parameter:

$ nix-build -A package -I ssh-config-file=~/ssh_config

It is also possible to provide the location of the config file by adapting the

NIX_PATH
environment variable, as opposed to using the
-I
parameter:
$ export NIX_PATH=ssh-config-file=~/ssh_config:$NIX_PATH

The above approach also makes it possible to deploy a NPM package with private dependencies as part of a NixOS, NixOps or Disnix configuration.

Disable cache bypassing

In a Nix builder environment, the NPM packages cache is empty and NPM does not seem to trust dependencies that are already stored in the bundled

node_modules/
folder, because they lack the meta data that can be used for integrity checks.

By default,

node2nix
bypasses the cache by augmenting package configuration files with these mandatory meta data fields.

If older versions of NPM are used (any version before 5.x), this meta information is not required. Bypassing the cache can be disabled by providing the

--no-bypass-cache
parameter.

Troubleshooting

This section contains some troubleshooting information for common problems.

Deploying peer dependencies

In NPM version 2.x and older, peer dependencies were automatically deployed if they were not declared as regular dependencies. In newer versions of NPM, this behaviour has changed -- peer dependencies are only used for version checks, but NPM no longer installs them.

Some package deployments may still rely on the old behaviour and will fail to deploy. To generate expressions that install peer dependencies, you can add the

--include-peer-dependencies
parameter:
$ node2nix --include-peer-dependencies

Stripping optional dependencies

When NPM packages with optional dependencies are published to the NPM registry, the optional dependencies become regular runtime dependencies. As a result, when deploying a package with a broken "integrated" optional dependency, the deployment with fail, unlike pure optional dependencies that are allowed to fail.

To fix these package deployments, it is possible to strip the optional dependencies from packages installed from the NPM registry:

$ node2nix --strip-optional-dependencies

Updating the package lock file

When deploying projects that provide a

package-lock.json
file,
node2nix
deployments will typically fail if the corresponding
package.json
configuration has changed after the generation of the lock file, because the dependency tree in the lock file may be incomplete.

To fix this problem,

npm install
must be executed again so that the missing or changed dependencies are updated in the lock file.

Creating a symlink to the node_modules folder in a shell session

In Nix shell sessions, that can be started with

nix-shell -A shell
, conventional Node.js projects will typically work, because there is a
NODE_PATH
environment variable that refers to a Nix store path that provides all the dependencies that the project needs.

However, there are also a variety of Node.js/NPM-based build tools available, such as Grunt, Gulp, Babel or ESLint, that work with plugins that are stored in the

node_modules/
folder of the project.

Unfortunately, these tools do not respect the

NODE_PATH
environment variable and, as a result, fail to work in a Nix shell session.

As a workaround, you can create a symlink in the project's root folder to allow these tools to find their dependencies:

$ ln -s $NODE_PATH node_modules

Keep in mind that the symlink needs to be removed again if you want to deploy the package with

nix-build
or
nix-env
.

Disabling running NPM install

node2nix
tries to mimic NPM's dependency resolver as closely as possible. However, it may happen that there is a small difference and the deployment fails a result.

A mismatch is typically caused by versions that can't be reliably resolved (e.g. due to wildcards) or errors in lifting bundled dependencies. In many cases, the package should still work despite the error.

To prevent the deployment from failing, we can disable the

npm install
step, by overriding the package:
{pkgs ? import  {
  inherit system;
}, system ? builtins.currentSystem}:

let nodePackages = import ./default.nix { inherit pkgs system; }; in nodePackages // { express = nodePackages.express.override { dontNpmInstall = true; }; }

By overriding a package and setting the

dontNpmInstall
parameter to
true
, we skip the install step (which merely serves as a check). The generated expression is actually responsible for obtaining and extracting the dependencies.

API documentation

This package includes API documentation, which can be generated with JSDoc.

License

The contents of this package is available under the MIT license

Acknowledgements

This package is based on ideas and principles pioneered in npm2nix.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.