AssetGraph

AssetGraph is an extensible, node.js-based framework for manipulating and optimizing web pages and web applications. The main core is a dependency graph model of your entire website, where all assets are treated as first class citizens. It can automatically discover assets based on your declarative code, reducing the configuration needs to a minimum.

If you just want to get started with the basics, read Peter Müller - Getting started with Assetgraph.

If you are looking for a prepackaged build system take a look at Assetgraph-builder.

Tools built with AssetGraph

• assetgraph-builder - A static web page build system that post-processes your website with extremely little configuration
• subfont - A tool that supercharges your webfont loading by automatically applying all best practice loading techniques and generating optimal font subsets
• hyperlink - A link checker tool that will ensure all your internal and external links are intact and up to date
• seespee - A Content-Security Policy generator. Point it at a webpage and it will tell you what policy you need as a minimum
• trackingdog - cli for finding the original source location of a line+column in a generated file, utilizing the source map
• bringhome - cli for downloading a web page and its assets to a local directory.

Assets and relations

All web build tools, even those that target very specific problems, have to get a bunch of boring stuff right just to get started, such as loading files from disc, parsing and serializing them, charsets, inlining, finding references to other files, resolution of and updating urls, etc.

The observation that inspired the project is that most of these tasks can be viewed as graph problems, where the nodes are the assets (HTML, CSS, images, JavaScript...) and the edges are the relations between them, e.g. anchor tags, image tags, favorite icons, css background-image properties and so on.

An AssetGraph object is a collection of assets (nodes) and the relations (edges) between them. It's a basic data model that allows you to populate, query, and manipulate the graph at a high level of abstraction. For instance, if you change the url of an asset, all relations pointing at it are automatically updated.

Additionally, each individual asset can be inspected and massaged using a relevant API: jsdom for HTML, PostCSS for CSS, and an Esprima AST for Javascript.

AssetGraph represents inline assets the same way as non-inline ones, so eg. inline scripts, stylesheets, and images specified as

data:

<!--[if !IE]> -->
<style type="text/css">
  body {
    background-image: url(data:image/gif;base64,R0lGODlhAQABAID/AMDAwAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==);
  }
</style>
<!-- <![endif]-->

These are some of the supported assets and associated relation types:

HTML

SVG

style=...

CSS

//# sourceMappingURL=...

background-image: url(...)

@import url(...)

behavior: url(...)

filter: AlphaImageLoader(src='...')

@font-face { src: url(...) }

JavaScript

//# sourceMappingURL=...

'foo/bar.png'.toString('url')

Web manifest

Icon urls,

related_applications

start_url

Cache manifest (appcache)

Entries in the

CACHE

NETWORK

FALLBACK

JSON, XML, PNG, GIF, JPEG, ICO

(none)

Features

• Build an AssetGraph programmatically or load it from disk or a remote server via http.
• Find explicit dependencies between JavaScript and CSS and roll them out as
  and
  tags in your HTML.
• Bundle and inline CSS and JavaScript.
• Create a cache manifest with references to all the assets your web app needs to be usable offline.
• Move all CSS, JavaScript, image assets etc. to a static dir and rename them to md5.extension so the web server can be configured to set a far-future Cache-Control.
• Help getting your static assets on a CDN by allowing you to easily rewrite all references to them.
• Use Graphviz to visualize your dependencies at any step.
• Using the separate assetgraph-sprite transform: Optimize CSS background images by creating sprite images. The spriting is guided by a set of custom CSS properties with a

  -ag-sprite

  prefix.

Installation

Make sure you have node.js and npm installed, then run:

$ npm install assetgraph

Querying the graph

AssetGraph supports a flexible syntax for finding assets and relations in a populated graph using the

findAssets

findRelations

Get an array containing all assets in the graph:

var allAssets = assetGraph.findAssets();

Find assets by type:

var htmlAssets = assetGraph.findAssets({ type: 'Html' });

Find assets of different named types:

var jsAndCss = assetGraph.findAssets({ type: { $in: ['Css', 'JavaScript' ] });

Find assets by matching a regular expression against the url:

var localImageAssets = assetGraph.findAssets({
  url: { $regex: /^file:.*\.(?:png|gif|jpg)$/ },
});

Find assets by predicate function:

var orphanedJavaScriptAssets = assetGraph.findAssets(function (asset) {
  return (
    asset.type === 'JavaScript' &&
    assetGraph.findRelations({ to: asset }).length === 0
  );
});

Find all HtmlScript (

var allHtmlScriptRelations = assetGraph.findRelations({ type: 'HtmlScript' });

Query objects have "and" semantics, so all conditions must be met for a multi-criteria query to match:

var textBasedAssetsOnGoogleCom = assetGraph.findAssets({
  isText: true,
  url: { $regex: /^https?:\/\/(?:www\.)google\.com\// },
});

Find assets by existence of incoming relations:

var importedCssAssets = assetGraph.findAssets({
  type: 'Css',
  incomingRelations: { $elemMatch: { type: 'CssImport' } },
});

Relation queries can contain nested asset queries when querying the

to

from

Find all HtmlAnchor (

assetGraph.findRelations({
  type: 'HtmlAnchor',
  to: { isImage: true, url: { $regex: /^file:/ } },
});

Transforms and workflows

AssetGraph comes with a collection of premade "transforms" that you can use as high level building blocks when putting together your build procedure. Most transforms work on a set of assets or relations and usually accept a query object so they can be scoped to work on only a specific subset of the graph.

Usually you'll start by loading some initial assets from disc or via http using the

loadAssets

populate

Thus the skeleton looks something like this:

var AssetGraph = require('assetgraph');

const assetGraph = new AssetGraph({ root: '/the/root/directory/' });
await assetGraph.loadAssets('*.html'); // Load all Html assets in the root dir
await assetGraph.populate({ followRelations: { type: 'HtmlAnchor' } }); // Follow 
// More work...
await assetGraph.writeAssetsToDisc({ type: 'Html' }); // Overwrite existing files

// Done!

In the following sections the built-in transforms are documented individually:

assetGraph.addCacheManifest([queryObj])

Add a

CacheManifest

Html

Html

queryObj

HtmlAnchor

assetGraph.bundleRelations(queryObj[, strategyName])

Bundle the

Css

JavaScript

queryObj

The

strategyName

oneBundlePerIncludingAsset

(the default)

Each unique asset pointing to one or more of the assets being bundled will get its own bundle. This can lead to duplication if eg. several

Html

sharedBundles

Create as many bundles as needed, optimizing for combined byte size of the bundles rather than http requests. Warning: Not as well tested as

oneBundlePerIncludingAsset

Note that a conditional comment within an

Html

ie.css

all.css

The created bundles will be placed at the root of the asset graph with names derived from their unique id (for example

file://root/of/graph/124.css

assetGraph.compressJavaScript([queryObj[, compressorName[, compressorOptions]]])

Compresses all

JavaScript

queryObj

The

compressorName

uglifyJs

(the default and the fastest)

The excellent UglifyJS compressor. If provided, the

compressorOptions

ast_squeeze

yuicompressor

Yahoo's YUICompressor though Tim-Smart's node-yuicompressor module. If provided, the

compressorOptions

require('yui-compressor').compile

closurecompiler

Google's Closure Compiler through Tim-Smart's node-closure module. If provided, the

compressorOptions

require('closure-compiler').compile

assetGraph.convertCssImportsToHtmlStyles([queryObj])

Finds all

Html

queryObj

CssImport

@import url(...)

HtmlStyle

Effectively the inverse of

assetGraph.convertHtmlStylesToInlineCssImports

Example:

is turned into:

assetGraph.convertHtmlStylesToInlineCssImports([queryObj])

Finds all

Html

queryObj

HtmlStyle

CssImport

@import url(...)

CssImports

Example:

is turned into:

This is a workaround for the limit of 31 stylesheets in Internet Explorer <= 8. This transform allows you to have up to 31*31 stylesheets in the development version of your HTML and still have it work in older Internet Explorer versions.

assetGraph.drawGraph(fileName)

Uses the Graphviz

dot

fileName

.svg

Requires Graphviz to be installed,

sudo apt-get install graphviz

assetGraph.executeJavaScriptInOrder(queryObj[, context])

Experimental: For each JavaScript asset in the graph (or those matched by queryObj), find all reachable

JavaScript

If the

context

assetGraph.externalizeRelations([queryObj])

Finds all inline relations in the graph (or those matched by

queryObj

For example:

could be turned into:

assetGraph.inlineCssImagesWithLegacyFallback([queryObj[, options]])

Finds all

Html

queryObj

Css

CssImage

background-image

data:

1. If

   options.sizeThreshold

   is specified, images with a greater byte size won't be inlined.

2. To avoid duplication, images referenced by more than one

   CssImage

   relation won't be inlined.

3. A

   CssImage

   relation pointing at an image with an

   inline

   GET parameter will always be inlined (eg.

   background-image: url(foo.png?inline);

   ). This takes precedence over the first two criteria.

4. If

   options.minimumIeVersion

   is specified, the

   data:

   url length limitations of that version of Internet Explorer will be honored.

If any image is inlined an Internet Explorer-only version of the stylesheet will be created and referenced from the

Html

For example:

await assetGraph.inlineCssImagesWithLegacyFallback(
  { type: 'Html' },
  { minimumIeVersion: 7, sizeThreshold: 4096 }
);

where

assetGraph

and

foo.css

body {
  background-image: url(small.gif);
}

will be turned into:

where

1234.css

foo.css

data:

body {
  background-image: url(data:image/gif;base64,R0lGODlhAQABAID/AMDAwAAAACH5BAEAAAAALAAAAAABAAEAAAICRAEAOw==);
}

The file name

1234.css

assetGraph.inlineRelations([queryObj])

Inlines all relations in the graph (or those matched by

queryObj

HtmlScript

HtmlStyle

CssImage

Example:

await assetGraph.inlineRelations({ type: { $in: ['HtmlStyle', 'CssImage'] } });

where

assetGraph

and

foo.css

body {
  background-image: url(small.gif);
}

will be turned into:

Note that

foo.css

CssImage

assetGraph.loadAssets(fileName|wildcard|url|Asset[, ...])

Add new assets to the graph and make sure they are loaded, returning a promise that fulfills with an array of the assets that were added. Several syntaxes are supported, for example:

const [aHtml, bCss] = await assetGraph.loadAssets('a.html', 'b.css'); // Relative to assetGraph.root
await assetGraph.loadAssets({
  url: 'http://example.com/index.html',
  text: 'var foo = bar;', // The source is specified, won't be loaded
});

file://

await assetGraph.loadAssets('file:///foo/bar/*.html'); // Wildcard expansion
await assetGraph.loadAssets('*.html'); // assetGraph.root must be file://...

assetGraph.mergeIdenticalAssets([queryObj])

Compute the MD5 sum of every asset in the graph (or those specified by

queryObj

For example:

await assetGraph.mergeIdenticalAssets({ type: { $in: ['Png', 'Css'] } });

where

assetGraph

Html

will be turned into the following if

foo.png

bar.png

and the

bar.png

assetGraph.minifyAssets([queryObj])

Minify all assets in the graph, or those specified by

queryObj

Html

and

Xml

Pure-whitespace text nodes are removed immediately.

Json

,

JavaScript

, and

Css

The asset gets marked as minified (

isPretty

false

asset.parseTree

JavaScript

compact

assetGraph.compressJavaScript

Compare to

assetGraph.prettyPrintAssets

transforms.populate(options)

Add assets to the graph by recursively following "dangling relations". This is the preferred way to load a complete web site or web application into an

AssetGraph

assetGraph.loadAssets

The

options

from

: queryObj

Specifies the set assets of assets to start populating from (defaults to all assets in the graph).

followRelations

: queryObj

Limits the set of relations that are followed. The default is to follow all relations.

onError

: function (err, assetGraph, asset)

If there's an error loading an asset and an

onError

concurrency

: Number

The maximum number of assets that can be loading at once (defaults to 100).

Example:

const assetGraph = new AssetGraph();
await assetGraph.loadAssets('a.html');
await assetGraph.populate({
  followRelations: {
    type: 'HtmlAnchor',
    to: { url: { $regex: /\/[bc]\.html$/ } },
  },
});

If

a.html

b.html

b.html

c.html

assetGraph.populate

c.html

d.html

d.html

assetGraph.prettyPrintAssets([queryObj])

Pretty-print all assets in the graph, or those specified by

queryObj

JavaScript

Css

Html

Xml

Json

The asset gets marked as pretty printed (

isPretty

true

asset.parseTree

Xml

Html

Compare to

assetGraph.minifyAssets

Example:

// Pretty-print all Html and Css assets:
await assetGraph.prettyPrintAssets({ type: { $in: ['Html', 'Css'] } });

assetGraph.removeRelations([queryObj, [options]])

Remove all relations in the graph, or those specified by

queryObj

The

options

detach

: Boolean

Whether to also detach the relations (remove their nodes from the parse tree of the source asset). Only supported for some relation types. Defaults to

false

removeOrphan

: Boolean

Whether to also remove assets that become "orphans" as a result of removing their last incoming relation.

assetGraph.setHtmlImageDimensions([queryObj])

Sets the

width

height

img

HtmlImage

queryObj

Example:

const AssetGraph = require('assetgraph');

const assetGraph = new AssetGraph();
await assetGraph.loadAssets('hasanimage.html');
await assetGraph.populate();
// assetGraph.findAssets({type: 'Html'})[0].text === ''
await assetGraph.setHtmlImageDimensions();
// assetGraph.findAssets({type: 'Html'})[0].text === ''

assetGraph.writeStatsToStderr([queryObj])

Dumps an ASCII table with some basic stats about all the assets in the graph (or those matching

queryObj

Example:

       Ico   1   1.1 KB
       Png  28 196.8 KB
       Gif 145 129.4 KB
      Json   2  60.1 KB
       Css   2 412.6 KB
JavaScript  34   1.5 MB
      Html   1   1.3 KB
    Total: 213   2.2 MB

assetGraph.writeAssetsToDisc(queryObj, outRoot[, root])

Writes the assets matching

queryObj

outRoot

file://

root

outRoot

root

Directories will be created as needed.

Example:

const AssetGraph = require('assetgraph');

const assetGraph = new AssetGraph({root: 'http://example.com/'});
await assetGraph.loadAssets(
  'http://example.com/bar/quux/foo.html',
  'http://example.com/bar/baz.html'
);
// Write the two assets to /my/output/dir/quux/foo.html and /my/output/dir/baz.html:
await assetGraph.writeAssetsToDisc({type: 'Html'} 'file:///my/output/dir/', 'http://example.com/bar/');

assetGraph.writeAssetsToStdout([queryObj])

Writes all assets in the graph (or those specified by

queryObj

License

AssetGraph is licensed under a standard 3-clause BSD license -- see the

LICENSE