Need help with AutoWebPerf?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

GoogleChromeLabs
273 Stars 24 Forks Apache License 2.0 283 Commits 10 Opened issues

Description

AutoWebPerf provides a flexible and scalable framework for running web performance audits with arbitrary audit tools including PageSpeedInsights, WebPageTest and more.

Services available

!
?

Need anything else?

Contributors list

AutoWebPerf (AWP)

quicklink

AutoWebPerf provides a flexible and scalable framework for running web performance audits with arbitrary audit tools like WebPageTest and PageSpeedInsights. This library enables developers to collect metrics consistently and store metrics to a preferred data store such as local JSON files, Google Sheets, BigQuery, or an in-house SQL database.

Check out https://web.dev/autowebperf for introduction.

How it works

AutoWebPerf takes a list of Tests from an arbitrary data store platform, such as local JSONs, Google Sheets, BigQuery, or a self-hosted SQL database. With the list of Tests, it executes audits based on each Test config, collects metrics from individual data sources into a list of Results.

The process of running an audit through an measurement tool (e.g. WebPageTest) is defined in the individual Gatherer. The logic of reading and writing with a data platform (e.g. local JSON) is implemented in a Connector.

Feature highlights

  • A library of web audit automation that can be plugged-in to any platforms, like Google Sheets, GCP App Engine, or simply a cron job that writes to JSON file.
  • Providing the ability to run recurring tests with customizable frequency (e.g. daily, weekly, monthly, etc), network conditions, and other audit configs, etc.
  • Metric gatherers are designed as modules that are decoupled with the output data format and automation logic.
  • Connector modules are designed to read Test list and write audit results to specific data format or platforms. e.g. a connector for CSV files. (See
    src/connectors/csv-connector
    for details)

How does this compare to the rest of Google's speed measurement tools?

AutoWebPerf serves as a performance audit aggregator that automates the process of performance audit and metrics collection through multiple speed measurement tools including WebPageTest, PageSpeedInsights, and Chrome UX Report. As each individual speed measurement tool provides audit metrics, AutoWebPerf aggregates the results and writes to any preferred data storage platform, such as local JSONs, cloud-based database, or GoogleSheets.

Quickstart

First, clone AWP repo and run npm install:

git clone https://github.com/GoogleChromeLabs/AutoWebPerf.git
npm install

Once finished, check the install by running a single test with the following command:

./awp run examples/tests.json output/results.json
This command uses the example file in
examples/tests.json
and returns the results to
output/results.json
.

To start recurring tests, you'll need to include a

recurring.frequency
property in the test file and set the next trigger in the test file. To setup the next trigger time and to run a one-off test, use this command after adding the
recurring.frequency
property to your tests:
./awp recurring examples/tests-recurring.json output/results.json
If this was successful, the trigger time will have updated base on your chosen frequency, and a result would have been written to
output/results.json
.

Once the trigger time is correctly set, you can have your tests auto-run on the next triger time with the

continue
command:
./awp continue examples/tests-recurring.json output/results.json
This will automatically run each test at the frequency specified. More information can be found below in the "Run recurring tests" section below.

More Examples

Single URL: To test a single URL through PageSpeedInsights: ``` ./awp run url:https://www.thinkwithgoogle.com/ json:output/results.json

**Pick Gatherer:** to test a single URL with a specific gatherer like PageSpeedInsights or WebPageTest:

./awp run --gatherers=psi url:https://web.dev json:output/results.json ```

CSV file: To run tests defined in a CSV file and write results to a JSON file:

./awp run csv:examples/tests.csv json:output/results.json

PageSpeedInsights API: To run PageSpeedInsights tests with an API Key:

PSI_APIKEY=SAMPLE_KEY ./awp run examples/tests.json output/results.json

WebPageTest API: To run WebPageTest tests:

WPT_APIKEY=SAMPLE_KEY ./awp run examples/tests-wpt.json output/results.json

Override vs. Append: To run tests and override existing results in the output file

./awp run examples/tests.json output/results.json --override-results

Available gatherers:

Available connectors:

  • JSON connector - reads or writes to local JSON files. This is the default connector if a conenctor name is not specified. For example:
    ./awp run examples/tests.json output/results.json
    

Alternatively, to specify using the JSON connector for the

Tests
path and the
Results
path:
./awp run json:/examples/tests.json json:output/results.json
  • CSV connector - reads or writes to local CSV files. To specify using the CSV connector for the

    Tests
    path and the
    Results
    path:
    ./awp run csv:/examples/tests.csv csv:output/results.csv
    
  • URL connector - generates just one

    Test
    with a specific URL for audit. To run an audit with just one
    Test
    with a specific URL:
    ./awp run url:https://example.com csv:output/results.csv
    

Please note that this connector only works with

Tests
path, not for the
Results
path.

Using AWP with Node CLI

Run tests

You can run the following anytime for printing CLI usages:

./awp --help

To run tests, you can run the following CLI command with given Tests JSON, like

examples/tests.json
, which contains an array of tests. You can check out the
examples/tests.json
for the data structure of Tests.
./awp run examples/tests.json output/results.json

This will generate the result object(s) in the given path to

results.json
.

By default, AWP will use JSON as the default connector for both reading tests and writing results. Alternatively, you can specify a different connector in the format of

:
.

E.g. to run tests defined in CSV and write results in JSON:

./awp run csv:examples/tests.csv json:output/results.csv

Retrieve test results

For some audit platforms like WebPageTest, each test may take a few minutes to fetch actual results. For these type of asynchronous audits, each Result will stay in "Submitted" status. You will need to explicitly retrieve results later.

Run the following to retrieve the final metrics of results in the

results.json
.
./awp retrieve examples/tests.json output/results.json

This will fetch metrics for all audit platforms and update to the Result object in the

output/results.json
. You can check out
examples/results.json
for details in Result objects.

Run recurring tests

If you'd like to set up recurring tests, you can define the

recurring
object that contains
frequency
for that Test.
./awp recurring examples/tests-recurring.json output/results.json

This will generate the Result object in the

results.json
and updates the next trigger time to its original Test object in the
tests.json
. E.g. the updated Test object would look like the following, with the updated
nextTriggerTimestamp
.
{
  "label": "web.dev",
  "url": "https://web.dev",
  "recurring": {
    "frequency": "Daily",
    "nextTriggerTimestamp": 1599692305567,
    "activatedFrequency": "Daily"
  },
  "psi": {
    "settings": {
      "locale": "en-GB",
      "strategy": "mobile"
    }
  }
}

The

nextTriggerTimestamp
will be updated to the next day based on the previous timestamp. This is to prevent repeated runs with the same Test and to guarantee that this Test is executed only once per day.

Set up a cron job to run recurring tests

In most Unix-like operating system, you can set up a cron job to run the AWP CLI periodically.

For example, in macOS, you can run the following commands to set up a daily cron job with AWP:

# Edit the cronjob with a text editor.
EDITOR=nano crontab -e

Add the following line to the crontab for a daily run at 12:00 at noon. Note that this is based on the system time where it runs AWP.

0 12 * * * PSI_APIKEY=SAMPLE_KEY cd ~/workspace/awp && ./awp run examples/tests.json csv:output/results-recurring.csv

Run tests with extensions

An extension is a module to assist AWP to run tests with additional process and computation. For example,

budgets
extension is able to add performance budgets and compute the delta between the targets and the result metrics.

To run with extensions:

./awp run examples/tests.json output/results.json --extensions=budgets

Tests and Results

Define the Tests

The list of tests is simply an array of Tests objects, like the sample Tests below. Or check out

src/examples/tests.js
for a detailed example of Tests list.
[{
  "label": "Test-1",
  "url": "example1.com",
  "webpagetest": {
    ...
  }
}, {
  "label": "Test-2",
  "url": "example2.com",
  "psi": {
    ...
  }
}]

Each

Test
object defines which audits to run by defining
gatherers
property. For example, the first
Test
has a
webpagetest
property which defines the configuration of running a WebPageTest audit. The second
Test
has a
psi
property that defines how to run PageSpeedInsights audit.

Generate the Results

After running tests, a list of

Results
is generated like below. Each
Result
contains its corresponding metrics to the predefined
gatherers
such as WebPageTest and PageSpeedInsights. See the example below.
[{
  "label": "Test-1",
  "url": "example1.com",
  "webpagetest": {
    "metrics": {
      FirstContentfulPaint: 900,
      ...
    }
  }  
}, {
  "label": "Test-2",
  "url": "example2.com",
  "psi": {
    "metrics": {
      FirstContentfulPaint: 900,
      ...
    }
  }  
}]

Environmental Variables

Some conenctors or gatherers may require one or more environmental variables, such as API keys or the path to service account. For example, running with the CrUX API gatherer requires the CrUX API key.

To pass the environmental variables in the CLI, run the command with the regular usage of environment vars:

CRUX_APIKEY= ./awp run url:https://wev.dev/ json:output/results.json

Gatherers

AWP supports the following audit gatherers. Please check out the corresponding documentations for details.

WebPageTest

The WebPageTest gatherer runs Tests through either the public WebPageTest endpoints or a custom private WebPageTest instance.

See docs/webpagetest.md for more details for the usage of WebPageTest gatherer.

PageSpeed Insights

The PageSpeed Insights gatherer runs Tests through the public PageSpeed Insights API.

See docs/psi.md for more details for the usage of PSI gatherer.

Chrome UX Report API (CrUX API)

The CrUX API gatherer collects performance metrics through the Chrome UX Report API.

See docs/cruxapi.md for more details for the usage of CrUX API gatherer.

Chrome UX Report History (CrUX via BigQuery)

The CrUX BigQuery gatherer collects performance metrics through the Chrome UX Report with its public Google BigQuery project. Please noet that you would need set up a Google Cloud project in order to query the public BigQuery table.

See docs/cruxbigquery.md for more details for the usage of CrUX API gatherer.

Design

AWP is designed with modules, including modules for running audits with WebPageTest, PageSpeedInsights, or other tools, and modules for reading/writing data from data platforms such as JSON, GoogleSheets or a Cloud service.

In a high-level view, there are three types of modules: - Gatherer - A Gatherer runs audits and generates metrics. - Connector - A Connector reads test configs from and writes results to a data platform, such as a local JSON file or with Google Sheets. - Extension - An Extension adds additional metrics or information, either before or after running audits.

The AWP engine uses two major JavaScript Object structures for running audits and collecting metrics.

  • Test - An object that contains the audit configuration for one test task, such as URL, audit methods, or extension config. You can refer to
    examples/tests.json
    for an actual Test object.
  • Result - An object that contains audit configuration, metrics and overall status. You can refer to
    examples/results.json
    for an actual Result object.

Audit steps

In order to deal with asynchronous audit tool like WebPageTest, AWP breaks the audit cycle into three steps of actions:

  • Run - This action takes a list of
    Tests
    and generates a list of
    Results
    objects.
  • Recurring - Similar to Run, this action takes a list of
    Tests
    , generates a list of
    Results
    , and updates nextTriggerTimestamp for each recurring
    Test
    . This action is useful when running with periodic or timer-based tasks such as cron job.
  • Retrieve - This action scans the list of Results, and collects metrics when the results are not in
    Retrieved
    status.

AWP Config

To set up modules and their configurations, an overall AWP Config is required as a JavaScript Object.

AWP Config has the following required properties: -

connector
: The name of connector. -
helper
: A helper for a specific connector, including API Handler and other helper functions, which will be used in gatherers and extensions. -
dataSources
: An array of audit sources, such as
webpagetest
or
psi
. Each of the data source needs to have a corresponding Gatherer file in the
src/gatherers
folder. -
extensions
: An array of extensions. Each extension needs to have a corresponding Extension file in
src/extensions
.

Other optional properties: -

verbose
: Whether to print verbose messages. -
debug
: Whether to print debug messages.

The following config example comes from the

examples/awp-config.js
:
{
  connector: 'JSON',
  helper: 'Node',
  dataSources: ['webpagetest'],
  json: { // Config for JSON Connector.
    tests: 'tests.json',
    results: 'results.json',
  },
  extensions: [
    'budgets',
  ],
  budgets: { // Config for Budgets extension.
    dataSource: 'webpagetest',
  },
  verbose: true,
  debug: false,
}

With the example config above, it will use

JSON
connector which reads and writes Tests and Results as JSON files. See
examples/tests.json
and
examples/results.json
for examples.

In addition to fundamental properties, there are a few additional properties used by modules: -

json
property as the configuration for JSONConnector. -
budgets
property as the configuration for BudgetsExtension

Usage of AutoWebPerf core

Examples of creating a new instance of AWP:

let awp = new AutoWebPerf({
  connector: 'JSON',
  helper: 'Node',
  dataSources: ['webpagetest'],
  extensions: extensions,
  json: { // Config for JSON connector.
    tests: argv['tests'],
    results: argv['results'],
  },
  verbose: verbose,
  debug: debug,
});
To submit all tests:
awp.run();

To submit specific tests using filters: This will run the test which has id=1 and selected=true properties.

awp.run({
  filters: ['id="1"', 'selected'],
});

To retrieve all pending results, filtering with status !== "Retrieved".

awp.retrieve({
  filters: ['status!=="Retrieved"'],
});
- For more advanced usage of PatternFilter, please check out
src/utils/pattern-filter.js
with more examples.

To run recurring tests:

// This will run the actual audit and update the nextTriggerTimestamp.
awp.recurring();

To run tests with specific extensions:

// This will override the extension list defined in the awpConfig.
awp.run({
  extensions: ['budgets']
})

Gatherer Modules

A Gatherer class extends

src/gatherers/gatherer.js
and overrides the following methods:
  • constructor(config, apiHelper, options)
    :
    • config
      : The config defined in a property with this gatherer's name in the AWP config. Some audit tools like WebPageTest or PageSpeedInsights require API keys. The API key for the gatherer is located in the
      config.apiKey
      .
    • options
      : Additional settings like
      verbose
      and
      debug
      .
  • run(test, options)
    :
    • test
      : A
      Test
      object for this audit run. The data required for this gatherer (e.g. settings or metadata) will be in the property with the gatherer's name. E.g. the data for WebPageTest will be in
      webpagetest
      of this
      Test
      object.
    • options
      : Additional settings.
  • retrieve(result, options)
    :
    • result
      : A
      Result
      object to retrieve metrics with. The data required for this gatherer will be in the property with the gatherer's name. E.g. the data and metrics will be in
      webpagetest
      of this
      Result
      object.
    • options
      : Additional settings like
      verbose
      and
      debug
      .

Connector Modules

A Connector class extends

src/connectors/connector.js
and overrides the following methods:
  • constructor(config, apiHandler)
    :
    • config
      : The config defined in a property with this connector's name in the AWP config.
    • apiHandler
      : The API handler instance used for making API calls.
  • getConfig()
    : The method to return the Connector's additional config object. This config object depends on where this Connector stores its additional settings including API keys for gatherers. For example, JSONConnector uses the
    tests.json
    and reads additional settings from the
    config
    property, including API keys for each gatherers.
  • getTestList(options)
    : The method to return the list of
    Tests
    as an array.
  • updateTestList(newTests, options)
    : The method to update
    Tests
    list, given the list of new
    Tests
    .
  • getResultList(options)
    : The method to return the list of
    Results
    as an array.
  • appendResultList(newResults, options)
    : The method to append new
    Results
    to the end of the current
    Results
    list.
  • updateResultList(newResults, options)
    : The method to update existing
    Results
    in the current
    Results
    list.

Extension Modules

A Extension class extends

src/extensions/extension.js
and overrides the following methods:
  • constructor(config)
    :
    • config
      : The config defined in a property with this extension's name in the AWP config.
  • beforeRun(context)
    : The method before executing Run step for a
    Test
    .
    • context.test
      : The corresponding
      Test
      object.
  • afterRun(context)
    : The method after executing Run step for a
    Test
    .
    • context.test
      : The corresponding
      Test
      object.
    • context.result
      : The corresponding
      Result
      object.
  • beforeAllRuns(context)
    : The method before executing Run step.
    • context.tests
      : All
      Test
      objects in this Run.
  • afterAllRuns(context)
    : The method after executing Run step.
    • context.tests
      : All
      Test
      objects in this Run.
    • context.results
      : All
      Result
      objects in this Run.
  • beforeRetrieve(context)
    : The method before executing Retrieve step for a
    Result
    .
    • context.result
      : The corresponding
      Result
      object.
  • afterRetrieve(context)
    : The method after executing Retrieve step for a
    Result
    .
    • context.result
      : The corresponding
      Result
      object.
  • beforeAllRetrieves(context)
    : The method before executing Retrieve step.
    • context.result
      : The corresponding
      Result
      object.
  • afterAllRetrieves(context)
    : The method after executing Retrieve step.
    • context.result
      : The corresponding
      Result
      object.

Test Object

A standard

Test
object contains the following properties:

(You can refer to

examples/tests.json
for an example.)
  • selected
    : Whether to perform Run for this
    Test
    .
  • label
    : Name of this
    Test
    .
  • url
    : URL to audit.
  • recurring
    : Settings for recurring audit.
    • frequency
      : The frequency string defined in
      src/common/frequency.js
      . E.g. 'Daily', 'Weekly' or 'Monthly'.

Gatherer-specific settings will be in their own property with the Gatherer's name in lower case. For example, the settings for WebPageTests will be:

  • webpagetest
    • settings
      : Setting object contains audit location, connection, etc.
    • metadata
      : Metadata object contains WebPageTests's ID, JSON URL, etc.

Result Object

A standard

Result
object contains the following properties:
  • selected
    : Whether to perform Retrieve for this
    Result
    .
  • id
    : Auto-generated unique ID for this
    Result
    .
  • type
    :
    Single
    or
    Recurring
    audit.
  • status
    :
    Submitted
    ,
    Retrieved
    or
    Error
    . Refer to
    src/common/status.js
    for details.
  • label
    : String label for this
    Result
    . This label inherits from its original
    Test
    object.
  • url
    : Audited URL.
  • createdTimestamp
    : When this
    Result
    is created.
  • modifiedTimestamp
    : When this
    Result
    is last modified.

Standardized Metrics

All metric names used in AWP are required to follow the names, case sensitive. See the full list of standardized metrics in

src/common/metrics.js
  • Timing metrics

    • TimeToFirstByte
    • FirstPaint
    • FirstMeaningfulPaint
    • FirstContentfulPaint
    • VisualComplete
    • SpeedIndex
    • DOMContentLoaded
    • LoadEvent
    • TimeToInteractive
    • TotalBlockingTime
    • FirstCPUIdle
    • FirstInputDelay
    • LargestContentfulPaint
  • Resource Size

    • HTML
    • Javascript
    • CSS
    • Fonts
    • Images
    • Videos
  • Resource Count

    • DOMElements
    • Connections
    • Requests
  • Resource Scores

    • Performance
    • ProgressiveWebApp

Source Folder Structure

All source codes for major functions are located in

src
folder. Files are organized into the following subfolders:
  • common
    : Common classes and definitions, such as Status, Frequency, Metrics, etc.
  • connectors
    : Connector classes.
  • extensions
    : Extension classes.
  • gatherers
    : Gatherer classes.
  • utils
    : Utilities and tools.

Unit Test

Run the following commands to run unit test:

npm test

To run individual test spec, you can install Jest NPM module to your local machine:

npm install -g jest
jest test/some-module.test.js

Unit Test Design

The Unit Test is based on Jest unit test framework. All unit tests are located in the

./test
folder, and are organized into its own corresponding subfolders, as the same structure as in the
src
folder.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.