Need help with metabench?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

157 Stars 15 Forks Other 389 Commits 11 Opened issues


A simple framework for compile-time benchmarks

Services available


Need anything else?

Contributors list

# 43,090
247 commits
# 108,624
111 commits
# 228,253
9 commits
# 80,761
1 commit
# 163,341
1 commit
# 460,642
1 commit
# 535,122
1 commit
# 667,785
1 commit

Metabench Travis status Appveyor status

A simple framework for compile-time microbenchmarks


Metabench is a single, self-contained CMake module making it easy to create compile-time microbenchmarks. Compile-time benchmarks measure the performance of compiling a piece of code instead of measuring the performance of running it, as regular benchmarks do. The micro part in microbenchmark means that Metabench can be used to benchmark precise parts of a C++ file, such as the instantiation of a single function. Writing benchmarks of this kind is very useful for C++ programmers writing metaprogramming-heavy libraries, which are known to cause long compilation times. Metabench was designed to be very simple to use, while still allowing fairly complex benchmarks to be written.

Metabench is also a collection of compile-time microbenchmarks written using the

module. The benchmarks measure the compile-time performance of various algorithms provided by different metaprogramming libraries. The benchmarks are updated nightly with the latest version of each library, and the results are published at


Metabench requires CMake 3.1 or higher and Ruby 2.1 or higher. Metabench is known to work with CMake's Unix Makefiles and Ninja generators.


To use Metabench, make sure you have the dependencies listed above and simply drop the

file somewhere in your CMake search path for modules. Then, use
to include the module in your CMake file, add individual datasets to be benchmarked using
, and finally specify which datasets should be put together into a chart via
. For example, a minimal CMake file using Metabench would look like:
# Make sure Metabench can be found when writing include(metabench)
list(APPEND CMAKE_MODULE_PATH "path/to/metabench/directory")

Actually include the module


Add new datasets

metabench_add_dataset(dataset1 "path/to/dataset1.cpp.erb" "[1, 5, 10]") metabench_add_dataset(dataset2 "path/to/dataset2.cpp.erb" "(1...15)") metabench_add_dataset(dataset3 "path/to/dataset3.cpp.erb" "(1...20).step(5)")

Add a new chart

metabench_add_chart(chart DATASETS dataset1 dataset2 dataset3)

This will create a target named

, which, when run, will gather benchmark data from each
and output JSON files for easy integration with other tools. A HTML file is generated for easy visualization of the datasets as a NVD3 chart. To understand what the
files are, read what follows.

The principle

Benchmarking the compilation time of a single

file is rather useless, because one could simply run the compiler and time that single execution instead. What is really useful is to have a means of running variations of the same
file automatically. For example, we might be interested in benchmarking the compilation time for creating a
with many elements in it. To do so, we could write the following test case:

int main() { auto tuple = std::make_tuple(1, 2, 3, 4, 5); }

We would run the compiler and time the compilation, and then change the test case by augmenting the number of elements in the tuple:


int main() { auto tuple = std::make_tuple(1, 2, 3, 4, 5, 6, 7, 8, 9, 10); }

We would measure the compilation time for this file, and repeat the process until satisfactory data has been gathered. This tedious task of generating different (but obviously related)

files and running the compiler to gather timings is what Metabench automates. It does this by taking a
file written using the ERB template system, and generating a family of
files from that template. It then compiles these
files and gathers benchmark data from these compilations.

Concretely, you start by writing a

file (say
) that may contain ERB markup:

int main() { auto tuple = std::make_tuple(); }

Code contained inside

 is just normal Ruby code. When the file
will be rendered, the contents of 
 will be replaced with the
result of evaluating this Ruby code, which will look like:

int main() { auto tuple = std::make_tuple(1, 2, 3, ..., n); }

The ERB markup language has many other features; we encourage readers to take a look at the Wikipedia page. What happens is that Metabench will generate a

file for different values of
, and will gather benchmark data for each of these values. Now, this isn't the whole story. More often than not, we're only interested in benchmarking part of a C++ file. Indeed, if we benchmark the whole file in our example above, we'll end up measuring the time required to
 header in addition to the time
required for creating the 
. While this might be negligible in our example, this situation arises in nontrivial examples, and would make the resulting data nearly worthless. Hence, we have to tell Metabench what part(s) of the file it should measure. This is done by guarding the relevant part(s) of the code with a preprocessor

int main() { #if defined(METABENCH) auto tuple = std::make_tuple(); #endif }

What Metabench will actually do is compile the file once with the macro defined (and hence with the content of the block), and once without it. It will then subtract the time for compiling the file without the content of the block to the time for compiling the whole file, which should represent a good approximation of the time for compiling what's inside the block.

On the C++ side of things, the

file will be compiled (to benchmark it) as if it were located in the directory containing the
file, so that relative include paths can be used. Furthermore, it will be compiled as if the
file were part of a CMake executable added in the same directory as the call to
. This way, any variable or property set in CMake will also apply when benchmarking the file. In other words, Metabench tries to create the illusion that the code is actually compiled as if it were written in the

This is it for the basic usage of the module! The

directory contains a fully working example of using Metabench to create benchmarks. For a more involved example, you can take a look at the benchmark suite in the
directory. Note that only the most basic usage of Metabench was covered here. To know all the features provided by the module, you should read the reference documentation provided as comments inside the CMake module.

A note on benchmark resolution

Like any measurement tool, Metabench has a limited resolution. For example, when the code being measured (inside the

pair) takes only a few milliseconds to compile, the timings reported by Metabench may be completely inside the noise. Typically, the resolution of timings taken by Metabench is similar to that of the
command. A good technique to make sure the results of a benchmark are not inside the noise is to reduce the relative uncertainty of the measurement. This can be done by increasing the total compilation time of the measured block, by repeating the same thing (or a similar one) multiple times:

int main() { #if defined(METABENCH) auto tuple1 = std::make_tuple(); auto tuple2 = std::make_tuple(); auto tuple3 = std::make_tuple(); auto tuple4 = std::make_tuple(); #endif }


Metabench was initially developed inside the Boost.Hana library as a mean to benchmark compile-time algorithms. After seeing that a self-standing framework would be useful to the general C++ community, it was decided to extract it into its own project.


Please see

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.