Onceover

The gateway drug to automated infrastructure testing with Puppet

Onceover is a tool to automatically run basic tests on an entire Puppet control repository.

It includes automatic parsing of the

Puppetfile

environment.conf

New in v3.19.1: I've reversed the decision to have onceover use

site.pp

manifest

environment.conf

site.pp

manifest

Table of Contents

• Overview
• Quick Start
• Configuration
  • onceover.yaml
  • Factsets
  • Hiera
  • Puppetfile
• Spec testing
  • Adding your own spec tests
• Using Workarounds
• Extra tooling
  • Plugins
  • Overriding Onceover's Templates
  • Accessing Onceover in a traditional RSpec test
  • Accessing fact sets in a traditional RSpec test
  • Accessing Roles in a traditional RSpec test
  • Filtering
  • Extra Configuration
  • Ruby Warnings
  • Rake tasks
    • generate_fixtures

Overview

This gem provides a toolset for testing Puppet control repository (ie. Repos used with r10k).

The main purpose of this project is to provide a set of tools to help smooth out the process of setting up and running

rspec-puppet

Due to the fact that controlrepos are fairly standardised in nature it seemed ridiculous that you would need to set up the same testing framework that we would normally use within a module for a controlrepo. This is because at this level we are normally just running very basic tests that cover a lot of code. It would also mean that we would need to essentially duplicated our

Puppetfile

.fixtures.yml

This toolset requires some configuration before it can be used so definitely read that section before getting started.

Quick Start

Note: This assumes you are inside the root of your control-repo.

1. Add

   onceover

   to your

   Gemfile

   :

   gem 'onceover'
   

2. Run Bundler to install

   onceover

   bundle install
   

3. Set up your configuration

   bundle exec onceover init
   

4. Run your spec tests!

   bundle exec onceover run spec
   

Configuration

This project uses one main config file to determine what Puppet classes we should be testing and how: onceover.yaml.

As

onceover

onceover.yaml

Usually located at

spec/onceover.yaml

ONCEOVER_YAML

Hopefully this config file will be fairly self explanatory once you see it, but basically this is the place where we define what classes we want to test and the factsets that we want to test them against.

Main settings

• classes

  A list (array) of classes that we want to test

  Classes are specified as string or regular expressions. The recommended setting to have a good coverage is

  /^role::/

  if you use roles and profiles method and if you dont, you should.

• nodes

  A list (array) of nodes that we want to test against.

  The nodes that we list here map directly to a factset, e.g.

  Debian-7.8-64

• node_groups

  An hash of named node groups.

  We can create groups by simply specifying an array of nodes to be in the group, or use the subtractive include/exclude syntax.

  Note: A node group named

  all_nodes

  is automatically created by

  onceover

  .

  Important: The names used for the actual

  class_groups

  and

  node_groups

  must be unique.

• class_groups

  An hash of named class groups.

  We can create groups by simply specifying an array of classes (string or regexp) to be in the group, or use the subtractive include/exclude syntax.

  Note: A class group named

  all_classes

  is automatically created by

  onceover

  .

  Important: The names used for the actual

  class_groups

  and

  node_groups

  must be unique.

• test_matrix

  An array of hashes with the following format:

    - {nodes_to_test}: # The name of a node or node group
        classes: '{classes_to_test}' # the name of a class or
        tests: '{all_tests|acceptance|spec}' # acceptance deprecated/broken, set to spec
        {valid_option}: {value} # See below
  

  Valid options:

  • tags

    Default:

    nil

  One or many tags that tests in this group should be tagged with. This allows you to run only certain tests using the

  --tags

  command line parameter.

Advanced settings

• functions

  Default:

  nil

  In this section we can add functions that we want to mock when running spec tests.

  Each function takes the

  return

  arguments, e.g.

  functions:
    puppetdb_query:
      returns: []
  

• before

  Default:

  nil

  A block to run before each spec test.

  For example, this can be used when the functions to stub are conditional, the following stub function

  x

  if the OS is windows, stub function

  y

  if the fact

  java_installed

  is true.

  Note: The facts are available through the

  node_facts

  hash and the trusted facts as

  trusted_facts

  .

  Example:

  yaml
  before:
    - "Puppet::Util::Platform.stubs(:'windows?').returns(node_facts['kernel'] == 'windows')"
  

• after

  Default:

  nil

  A block to run after each spec test.

  Note: The facts are available through the

  node_facts

  hash and the trusted facts as

  trusted_facts

  .

  Exmaple:

  yaml
  after:
    - "puts 'Test finished running'"
  

• include_spec_files

  Default:

  [**/*]

  Glob to select additionnal files to be run during

  onceover

  spec tests.

  Default is to select all files located in the repo spec directory, usually

  spec/

  . If you have some RSpec tests that depend on a different RSpec configuration than

  onceover

  or want, for example, to have a different job in your CI to run your own unit tests, you can use this option to select which spec files to run during

  onceover

  spec tests.

• opts

  Default:

  {}

  This setting overrides defaults for the

  Onceover::Controlrepo

  class'

  opts

  hash.

  Example:

  yaml
  opts:
    :facts_dirs:                           # Remember: `opts` keys are symbols!
      - 'spec/factsets'                    # Limit factsets to files in this repository
    :debug: true                           # Set the `logger.level` to debug
    :profile_regex: '^(profile|site)::'    # Profiles include a legacy module named `site::`
    :facts_files:                          # Factset filenames use the extension`.facts` instead of `.json`
      - 'spec/factsets/*.facts'
    :manifest: 'manifests/site.pp'         # Manifest to use while compiling (nil by default)
  

Include/Exclude syntax

This can be used with either

node_groups

class_groups

node_groups:
  windows_nodes: # This has to be defined first
    - sevrer2008r2
    - server2012r2
  non_windows:
    include: 'all_nodes' # Start with all nodes
    exclude: 'windows_nodes' # Then remove the windows ones from that list

It's important to note that in order to reference a group using the include/exclude syntax is has to have been defined already i.e. it has to come above the group that references it

Examples

Minimal

classes:
  - /^role::/

nodes:

Debian-10-amd64

test_matrix:

all_nodes:
  classes: 'all_classes'
  tests: 'spec'

Advanced

In the example below we have referred to

centos6a

centos7b

all_nodes

non_windows_servers

centos_severs

A full example:

classes:
  - 'roles::backend_dbserver'
  - 'roles::frontend_webserver'
  - 'roles::load_balancer'
  - 'roles::syd_f5_load_balancer'
  - 'roles::windows_server'
  - '/^role/'                     # Note that this regex format requires `/`

nodes:

centos6a
centos7b
server2008r2a
ubuntu1404a
ubuntu1604a

node_groups:
  centos_severs:
    - centos6a
    - centos7b
  ubuntu_servers:
    - ubuntu1404a
    - ubuntu1604a
  non_windows_servers:
    include: 'all_nodes'
    exclude: 'server2008r2a'
class_groups:
  windows_roles:
    - 'roles::windows_server'
    - 'roles::backend_dbserver'
    - '/^roles::win/'
  non_windows_roles:
    include: 'all_classes'
    exclude: 'windows_roles'
test_matrix:

all_nodes:
  classes: 'all_classes'
  tests: 'spec'
non_windows_servers:
  classes: 'non_windows_roles'
ubuntu_servers:
  classes: 'all_classes'
  tests: 'spec'
centos_severs:
  classes: 'roles::frontend_webserver'
  tests: 'spec'
  runs_before_idempotency: 2
  tags:
- 'frontend'


functions:
  query_resources:
    returns: []
  profile::custom_function:
    returns: ["one", "two"]
opts:
  :facts_dirs:
    - spec/factsets
  :profile_regex: '^(profile|site)::'   # Note that this regex doesn't use /

Factsets

Factsets are used by the onceover gem to generate spec tests, which compile a given class against a certain set of facts.

This gem comes with a few pre-canned factsets, these are listed under the

nodes

onceover.yaml

onceover init

You can, and its a good pratice, add your own factsets by putting them in

spec/factsets/*.json

To generate these factsets all we need to do is log onto a real machine that has puppet installed and run:

puppet facts > fact_set_name.json

Its recommended to run this on each of the types of nodes that you run in your infrastructure to have good coverage.

Factsets are named based on their filename, i.e.

myfactset

onceover.yaml

spec/factsets/myfactset.json

Trusted Facts

You can add trusted facts to the factsets by creating a new section called trusted:

{
  "name": "node.puppetlabs.net",
  "trusted": {
    "pp_role": "agent",
    "pp_datacenter": "puppet",
  },
  "values": {
    "aio_agent_build": "1.10.4",
    "aio_agent_version": "1.10.4",
    "architecture": "x86_64",

Notice that the

extensions

$trusted['extensions']['pp_role']

Hiera

If you have hiera data inside your controlrepo (or somewhere else)

onceover

hiera.yaml

hiera-eyaml

hiera.yaml

yaml

It is also worth noting that any hiera hierarchies that are based on custom facts will not work unless those facts are part of your factsets. Trusted facts will also not work at all as the catalogs are being compiled without the node's certificate. In these instances it may be worth creating a hierarchy level that simply includes dummy data for testing purposes in order to avoid hiera lookup errors.

Puppetfile

Organisations often reference modules from their own git servers in the

Puppetfile

mod "puppetlabs-apache",
  :git => "https://git.megacorp.com/pup/puppetlabs-apache.git",
  :tag => "v5.4.0"

Under the hood,

onceover

r10k

Puppetfile

git

r10k

git

onceover

r10k

INFO     -> Updating module /Users/dylan/control-repo/.onceover/etc/puppetlabs/code/environments
/production/modules/apache
ERROR    -> Command exited with non-zero exit code:
Command: git --git-dir /Users/dylan/.r10k/git/ssh---git.megacorp.com-pup-puppetlabs_apache.git fetch origin --prune
Stderr:
ssh_askpass: exec(/usr/bin/ssh-askpass): No such file or directory
Host key verification failed.

fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Exit code: 128

Then the approach to debug it would be to run the command that Onceover suggested:

git --git-dir /Users/dylan/.r10k/git/ssh---git.megacorp.com-pup-puppetlabs_apache.git fetch origin --prune

In this case, running the command interactively gives us a prompt to add the server to our

~/.ssh/known_hosts

$ git --git-dir /Users/dylan/.r10k/git/ssh---git.megacorp.com-pup-puppetlabs_apache.git fetch origin --prune
The authenticity of host 'git.megacorp.com (123.456.789.101)' can't be established.
...
Warning: Permanently added 'git.megacorp.com,123.456.789.101' (RSA) to the list of known hosts.

The other way to resolve this would have been to install the

ssh_askpass

r10k configuration

If you have custom r10k config that you want to use, place the

r10k.yaml

• {repo root}/r10k.yaml

• {repo root}/spec/r10k.yaml

A good use of this is enabling multi threading by creating the following in

r10k.yaml

# spec/r10k.yaml
---
pool_size: 20

Creating the config file

If your

hiera.yaml

spec/hiera.yaml

Setting the

datadir

| Hiera Version | Config File Location | Required datadir | |---------------|----------------------|------------------| | 3 |

spec

data

data

spec

../data

data

spec

../data

hiera-eyaml

If you are using the

hiera-eyaml

hiera-eyaml

Recommended Solution: I recommend that if you are using

hiera-eyaml

1. Duplicate your

   hiera.yaml

   file so that there is a copy in the

   spec/

   directory
2. Change the

   datadir

   setting as described above
3. Remove the eyaml backend entirely and just use the base yaml backend. For hiera 5 this will look like:

  ---
  version: 5
  defaults:
    datadir: "../data"
    data_hash: yaml_data

This means that for testing, hiera will just return the encrypted string for anything that is encrypted using eyaml. This usually isn't a problem for catalog compilation and will allow tests to pass.

Spec testing

Once you have your

onceover.yaml

To run the tests:

onceover run spec

This will do the following things:

1. Create a temporary directory under

   .onceover

2. Clone all repos in the Puppetfile into the temporary directory
3. Generate tests that use rspec-puppet
4. Run the tests

Adding your own spec tests

When using this gem adding your own spec tests is exactly the same as if you were to add them to a module, simply create them under

spec/{classes,defines,etc.}

it { should compile }

Exposing Puppet output

If you want to see Puppet's output, you can set the

SHOW_PUPPET_OUTPUT

true

SHOW_PUPPET_OUTPUT=true onceover run spec

Using workarounds

There may be situations where you cannot test everything that is in your puppet code, some common reasons for this include:

• Code is destined for a Puppet Master but the VM image is not a Puppet Master which means we can't restart certain services etc.
• A file is being pulled from somewhere that is only accessible in production
• Something is trying to connect to something else that does not exist

Fear not! There is a solution for this, it's also a good way to practice writing nasty puppet code. For this exact purpose I have added the ability for onceover to include extra bits of code in the tests to fix things like this. All you need to do is put a file/s containing puppet code here:

spec/pre_conditions/*.pp

What this will do is it will take any puppet code from any files it finds in that directory and have it executed alongside the code that you are actually testing. For example if we are testing some code that notifies the

pe-puppetserver

file { '/etc/puppetlabs/puppet/puppet.conf':
  ensure  => file,
  content => '#nothing',
  notify  => Service['pe-puppetserver'], # This will fail without the PE module!
}

To fix this we can add the service to the pre_conditions to make sure that our catalogs can compile e.g.

# spec/pre_conditions/services.pp
service { 'pe-puppetserver':
  ensure => 'running',
}

You can also mock out defined resources or types that you cannot gain access to easily, such as

puppet_enterprise::mcollective::client

  define puppet_enterprise::mcollective::client (
    $activemq_brokers,
    $logfile     = '/var/log',
    $create_user = true,
  ) {

  }

or

  define pe_ini_setting (
    $ensure  = present,
    $path,
    $section,
    $setting,
    $value,
  ) {
  }

Resource collectors are likely to come in handy here too. They allow you to override values of resources that match given criteria. This way we can override things for the sake of testing without having to change the code.

NOTE: If you want to access the class or factset that onceover is running against just use the

$onceover_class

$onceover_node

Extra Tooling

Plugins

Onceover now allows for plugins. To use a plugin simply install a gem with a name that starts with

onceover-

Useful plugins:

• onceover-codequality Check lint and syntax
• onceover-octocatalog-diff See the differences between two versions of a catalog

If you want to write your own plugin, take a look at onceover-helloworld to help you get started.

Inspecting and updating the Puppetfile

Onceover comes with some extra commands for interacting with the Puppetfile in useful ways. These are:

onceover show puppetfile

This will display all the current versions of all modules that are in the Puppetfile alongside the latest versions and whether or not they are out of date. This is a useful took for making sure your modules don't get too stale.

onceover update puppetfile

This takes your Puppetfile and actually modifies all of the module versions in there to the latest versions and saves the file. This is useful for setting up automated Puppetfile updates, just get Jenkins or Bamboo to:

1. Check out the Controlrepo
2. Run onceover to get a passing baseline
3. Update the Puppetfile with the latest versions of all modules
4. Run Onceover again
5. Create a pull request if all tests pass

Overriding Onceover's Templates

Onceover uses templates to create a bunch of files in the

.onceover

spec/templates/

spec/templates/spec_helper.rb.erb

Accessing Onceover in a traditional RSpec test

If you would like to use

onceover.yaml

it { should_compile }

# spec/classes/role_spec.rb
require 'spec_helper'
require 'onceover/controlrepo'
require 'helpers/shared_examples'

Onceover::Controlrepo.new.spec_tests do |class_name,node_name,facts,pre_conditions|
  describe class_name do
    context "on #{node_name}" do
      let(:facts) { facts }
      let(:pre_condition) { pre_conditions }
  it_behaves_like 'soe'
end
  end
end

This will use the

soe

onceover.yaml

Note: Onceover will automatically run any extra Rspec tests that it finds in the normal directories

spec/{classes,defines,unit,functions,hosts,integration,types}

Accessing fact sets in a traditional RSpec test

We can access all of our fact sets using

Onceover::Controlrepo.facts

Onceover::Controlrepo.facts.each do |facts|
  context "on #{facts['fqdn']}" do
    let(:facts) { facts }
    it { should compile }
  end
end

Other (possibly less useful) methods

The following code will test all roles that onceover can find (ignoring the ones configured in

onceover.yaml

require 'spec_helper'
require 'onceover/controlrepo'
Onceover::Controlrepo.roles.each do |role|
  describe role do
    Onceover::Controlrepo.facts.each do |facts|
      context "on #{facts['fqdn']}" do
        let(:facts) { facts }
        it { should compile }
      end
    end
  end
end

This will iterate over each role in the controlrepo and test that it compiles with each set of facts.

The same can also be done with profiles just by using the profiles method instead:

require 'spec_helper'
require 'onceover'
Onceover::Controlrepo.profiles.each do |profile|
  describe profile do
    Onceover::Controlrepo.facts.each do |facts|
      context "on #{facts['fqdn']}" do
        let(:facts) { facts }
        it { should compile }
      end
    end
  end
end

It is not limited to just doing simple "It should compile" tests. You can put any tests you want in here.

Also since the

profiles

roles

facts

facts

Filtering

You can also filter your fact sets based on the value of any fact, including structured facts. (It will drill down into nested hashes and match those too, it's not just a dumb equality match)

Just pass a hash to the

facts

require 'spec_helper'
require 'onceover'

describe 'profile::windows_appserver' do
  Onceover::Controlrepo.facts({
    'kernel' => 'windows'
    }).each do |facts|
    context "on #{facts['fqdn']}" do
      let(:facts) { facts }
      it { should compile }
    end
  end
end

Extra Configuration

You can modify the regexes that the gem uses to filter classes that it finds into roles and profiles. Just set up a Controlrepo object and pass regexes to the below settings.

repo = Onceover::Controlrepo.new()
repo.role_regex = /.*/ # Tells the class how to find roles, will be applied to repo.classes
repo.profile_regex = /.*/ # Tells the class how to find profiles, will be applied to repo.classes

Note that you will need to call the

roles

profiles

repo.roles

Ruby Warnings

When running onceover with

--debug

warning: constant ::Fixnum is deprecated

--debug

RUBYOPT

-W0

export RUBYOPT='-W0'

It is also worth noting if you want to use

RUBYOPT

--debug

Rake tasks

I have included a couple of little rake tasks to help get you started with testing your control repos. Set them up by adding this to your

Rakefile

require 'onceover/rake_tasks'

The tasks are as follows:

generate_fixtures

bundle exec rake generate_fixtures

This task will go though your Puppetfile, grab all of the modules in there and convert them into a

.fixtures.yml

environment.conf

.fixtures.yml

Puppetfile

forge "http://forgeapi.puppetlabs.com"

Modules from the Puppet Forge
mod "puppetlabs/stdlib", "4.6.0"
mod "puppetlabs/apache", "1.5.0"

environment.conf

modulepath = site:modules:$basemodulepath
environment_timeout = 0

Then the

.fixtures.yml

---
fixtures:
  symlinks:
    profiles: site/profiles
    roles: site/roles
  forge_modules:
    stdlib:
      repo: puppetlabs/stdlib
      ref: 4.6.0
    apache:
      repo: puppetlabs/apache
      ref: 1.5.0

Notice that the symlinks are not the ones that we provided in

environment.conf

Developing Onceover

Install gem dependencies:

bundle install

Clone the submodules

git submodule init && git submodule update --recursive

Execute tests

bundle exec rake

Contributors

Cheers to all of those who helped out:

• @jessereynolds
• @op-ct
• @GeoffWilliams
• @beergeek
• @jairojunior
• @natemccurdy
• @aardvark
• @Mandos
• @Nekototori
• @LMacchi
• @tabakhase
• @binford2k
• @raphink
• @tequeter
• @alexjfisher
• @smortex
• @16c7x
• @neomilium
• @chlawren