Elasticsearch Puppet Module

<!-- -->

Table of Contents

1. Module description - What the module does and why it is useful
2. Setup - The basics of getting started with Elasticsearch
   • The module manages the following
   • Requirements
3. Usage - Configuration options and additional functionality
4. Advanced features - Extra information on advanced usage
5. Reference - An under-the-hood peek at what the module is doing and how
6. Limitations - OS compatibility, etc.
7. Development - Guide for contributing to the module
8. Support - When you need help with this module

Module description

This module sets up Elasticsearch instances with additional resource for plugins, templates, and more.

This module is actively tested against Elasticsearch 2.x, 5.x, and 6.x.

WARNING: The 7.x major release of this module contains breaking changes!

In order to simplify the management of Elasticsearch moving forward, and add support for both Elasticsearch 6.x and 7.x, support for running multiple instances of Elasticsearch has been removed.

This module also does not currently handle the migration from the instance based configuration to the new single deployment model.
Therefore in-place upgrades from version 6.x of this module to 7.x, or migrations from multi-instance to single deployment is not currently supported.
We hope to add support for this in a future release.

Therefore please ensure that you test this major release in your environment before using it in production!

Setup

The module manages the following

• Elasticsearch repository files.
• Elasticsearch package.
• Elasticsearch configuration file.
• Elasticsearch service.
• Elasticsearch plugins.
• Elasticsearch snapshot repositories.
• Elasticsearch templates.
• Elasticsearch ingest pipelines.
• Elasticsearch index settings.
• Elasticsearch users, roles, and certificates.
• Elasticsearch licenses.
• Elasticsearch keystores.

Requirements

• The stdlib Puppet library.
• richardc/datacat
• Augeas
• puppetlabs-java_ks for certificate management (optional).

Beginning with Elasticsearch 7.0.0, a Java JDK has been bundled as part of the elasticsearch package.
However there still needs to be a version of Java present on the system being managed in order for Puppet to be able to run various utilities.
We recommend managing your Java installation with the puppetlabs-java module.

Repository management

When using the repository management, the following module dependencies are required:

• General: Elastic/elastic_stack
• Debian/Ubuntu: Puppetlabs/apt
• OpenSuSE/SLES: puppet/zypprepo

Beginning with Elasticsearch

Declare the top-level

elasticsearch

include ::java

class { 'elasticsearch': }

Usage

Main class

Most top-level parameters in the

elasticsearch

Install a specific version

class { 'elasticsearch':
  version => '7.9.3'
}

Note: This will only work when using the repository.

Automatically restarting the service (default set to false)

By default, the module will not restart Elasticsearch when the configuration file, package, or plugins change. This can be overridden globally with the following option:

class { 'elasticsearch':
  restart_on_change => true
}

Or controlled with the more granular options:

restart_config_change

restart_package_change

restart_plugin_change.

Automatic upgrades (default set to false)

class { 'elasticsearch':
  autoupgrade => true
}

Removal/Decommissioning

class { 'elasticsearch':
  ensure => 'absent'
}

Install everything but disable service(s) afterwards

class { 'elasticsearch':
  status => 'disabled'
}

API Settings

Some resources, such as

elasticsearch::template

class { 'elasticsearch':
  api_protocol            => 'http',
  api_host                => 'localhost',
  api_port                => 9200,
  api_timeout             => 10,
  api_basic_auth_username => undef,
  api_basic_auth_password => undef,
  api_ca_file             => undef,
  api_ca_path             => undef,
  validate_tls            => true,
}

Each of these can be set at the top-level

elasticsearch

Dynamically Created Resources

This module supports managing all of its defined types through top-level parameters to better support Hiera and Puppet Enterprise. For example, to manage an index template directly from the

elasticsearch

class { 'elasticsearch':
  templates => {
    'logstash' => {
      'content' => {
        'template' => 'logstash-*',
        'settings' => {
          'number_of_replicas' => 0
        }
      }
    }
  }
}

Plugins

This module can help manage a variety of plugins. Note that

module_dir

From an official repository

elasticsearch::plugin { 'x-pack': }

From a custom url

elasticsearch::plugin { 'jetty':
  url => 'https://oss-es-plugins.s3.amazonaws.com/elasticsearch-jetty/elasticsearch-jetty-1.2.1.zip'
}

Using a proxy

You can also use a proxy if required by setting the

proxy_host

proxy_port

puppet
elasticsearch::plugin { 'lmenezes/elasticsearch-kopf',
  proxy_host => 'proxy.host.com',
  proxy_port => 3128
}

Proxies that require usernames and passwords are similarly supported with the

proxy_username

proxy_password

Plugin name formats that are supported include:

• elasticsearch/plugin/version

  (for official elasticsearch plugins downloaded from download.elastic.co)

• groupId/artifactId/version

  (for community plugins downloaded from maven central or OSS Sonatype)

• username/repository

  (for site plugins downloaded from github master)

Upgrading plugins

When you specify a certain plugin version, you can upgrade that plugin by specifying the new version.

elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.1.1': }

And to upgrade, you would simply change it to

elasticsearch::plugin { 'elasticsearch/elasticsearch-cloud-aws/2.4.1': }

Please note that this does not work when you specify 'latest' as a version number.

ES 6.x and 7.x official plugins

For the Elasticsearch commercial plugins you can refer them to the simple name.

See Plugin installation for more details.

Scripts

Installs scripts to be used by Elasticsearch. These scripts are shared across all defined instances on the same host.

elasticsearch::script { 'myscript':
  ensure => 'present',
  source => 'puppet:///path/to/my/script.groovy'
}

Script directories can also be recursively managed for large collections of scripts:

elasticsearch::script { 'myscripts_dir':
  ensure  => 'directory,
  source  => 'puppet:///path/to/myscripts_dir'
  recurse => 'remote',
}

Templates

By default templates use the top-level

elasticsearch::api_*

elasticsearch::template { 'templatename':
  api_protocol            => 'https',
  api_host                => $::ipaddress,
  api_port                => 9201,
  api_timeout             => 60,
  api_basic_auth_username => 'admin',
  api_basic_auth_password => 'adminpassword',
  api_ca_file             => '/etc/ssl/certs',
  api_ca_path             => '/etc/pki/certs',
  validate_tls            => false,
  source                  => 'puppet:///path/to/template.json',
}

Add a new template using a file

This will install and/or replace the template in Elasticsearch:

elasticsearch::template { 'templatename':
  source => 'puppet:///path/to/template.json',
}

Add a new template using content

This will install and/or replace the template in Elasticsearch:

elasticsearch::template { 'templatename':
  content => {
    'template' => "*",
    'settings' => {
      'number_of_replicas' => 0
    }
  }
}

Plain JSON strings are also supported.

elasticsearch::template { 'templatename':
  content => '{"template":"*","settings":{"number_of_replicas":0}}'
}

Delete a template

elasticsearch::template { 'templatename':
  ensure => 'absent'
}

Ingestion Pipelines

Pipelines behave similar to templates in that their contents can be controlled over the Elasticsearch REST API with a custom Puppet resource. API parameters follow the same rules as templates (those settings can either be controlled at the top-level in the

elasticsearch

Adding a new pipeline

This will install and/or replace an ingestion pipeline in Elasticsearch (ingestion settings are compared against the present configuration):

elasticsearch::pipeline { 'addfoo':
  content => {
    'description' => 'Add the foo field',
    'processors' => [{
      'set' => {
        'field' => 'foo',
        'value' => 'bar'
      }
    }]
  }
}

Delete a pipeline

elasticsearch::pipeline { 'addfoo':
  ensure => 'absent'
}

Index Settings

This module includes basic support for ensuring an index is present or absent with optional index settings. API access settings follow the pattern previously mentioned for templates.

Creating an index

At the time of this writing, only index settings are supported. Note that some settings (such as

number_of_shards

elasticsearch::index { 'foo':
  settings => {
    'index' => {
      'number_of_replicas' => 0
    }
  }
}

Delete an index

elasticsearch::index { 'foo':
  ensure => 'absent'
}

Snapshot Repositories

By default snapshotrepositories use the top-level `elasticsearch::api*` settings to communicate with Elasticsearch. The following is an example of how to override these settings:

elasticsearch::snapshot_repository { 'backups':
  api_protocol            => 'https',
  api_host                => $::ipaddress,
  api_port                => 9201,
  api_timeout             => 60,
  api_basic_auth_username => 'admin',
  api_basic_auth_password => 'adminpassword',
  api_ca_file             => '/etc/ssl/certs',
  api_ca_path             => '/etc/pki/certs',
  validate_tls            => false,
  location                => '/backups',
}

Delete a snapshot repository

elasticsearch::snapshot_repository { 'backups':
  ensure   => 'absent',
  location => '/backup'
}

Connection Validator

This module offers a way to make sure an instance has been started and is up and running before doing a next action. This is done via the use of the

es_instance_conn_validator

puppet
es_instance_conn_validator { 'myinstance' :
  server => 'es.example.com',
  port   => '9200',
}

A common use would be for example :

class { 'kibana4' :
  require => Es_Instance_Conn_Validator['myinstance'],
}

Package installation

There are two different ways of installing Elasticsearch:

Repository

Choosing an Elasticsearch major version

This module uses the

elastic/elastic_stack

class { 'elastic_stack::repo':
  version => 6,
}

class { 'elasticsearch':
  version => '6.8.12',
}

This module defaults to the upstream package repositories, which as of Elasticsearch 6.3, includes X-Pack. In order to use the purely OSS (open source) package and repository, the appropriate

oss

elastic_stack::repo

elasticsearch

class { 'elastic_stack::repo':
  oss => true,
}

class { 'elasticsearch':
  oss => true,
}

Manual repository management

You may want to manage repositories manually. You can disable automatic repository management like this:

class { 'elasticsearch':
  manage_repo => false,
}

Remote package source

When a repository is not available or preferred you can install the packages from a remote source:

http/https/ftp

class { 'elasticsearch':
  package_url => 'https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.4.2.deb',
  proxy_url   => 'http://proxy.example.com:8080/',
}

Setting

proxy_url

elasticsearch::plugin

proxy_url

proxy_url

undef

puppet://

class { 'elasticsearch':
  package_url => 'puppet:///path/to/elasticsearch-1.4.2.deb'
}

Local file

class { 'elasticsearch':
  package_url => 'file:/path/to/elasticsearch-1.4.2.deb'
}

JVM Configuration

When configuring Elasticsearch's memory usage, you can modify it by setting

jvm_options

class { 'elasticsearch':
  jvm_options => [
    '-Xms4g',
    '-Xmx4g'
  ]
}

Service management

Currently only the basic SysV-style init and Systemd service providers are supported, but other systems could be implemented as necessary (pull requests welcome).

Defaults File

The defaults file (

/etc/defaults/elasticsearch

/etc/sysconfig/elasticsearch

File source

class { 'elasticsearch':
  init_defaults_file => 'puppet:///path/to/defaults'
}

Hash representation

$config_hash = {
  'ES_HEAP_SIZE' => '30g',
}

class { 'elasticsearch':
  init_defaults => $config_hash
}

Note:

init_defaults

Advanced features

Security

File-based users, roles, and certificates can be managed by this module.

Note: If you are planning to use these features, it is highly recommended you read the following documentation to understand the caveats and extent of the resources available to you.

Roles

Roles in the file realm can be managed using the

elasticsearch::role

myrole

elasticsearch::role { 'myrole':
  privileges => {
    'cluster' => [ 'monitor' ],
    'indices' => [{
      'names'      => [ '*' ],
      'privileges' => [ 'read' ],
    }]
  }
}

This role would grant users access to cluster monitoring and read access to all indices. See the Security documentation for your version to determine what

privileges

Note: The Puppet provider for

elasticsearch_user

roles.yml

resources { 'elasticsearch_role':
  purge => true,
}

Mappings

Associating mappings with a role for file-based management is done by passing an array of strings to the

mappings

elasticsearch::role

elasticsearch::role { 'logstash':
  mappings   => [
    'cn=group,ou=devteam',
  ],
  privileges => {
    'cluster' => 'manage_index_templates',
    'indices' => [{
      'names'      => ['logstash-*'],
      'privileges' => [
        'write',
        'delete',
        'create_index',
      ],
    }],
  },
}

If you'd like to keep the mappings file purged of entries not under Puppet's control, you should use the following

resources

resources { 'elasticsearch_role_mapping':
  purge => true,
}

Users

Users can be managed using the

elasticsearch::user

mysuser

myrole

elasticsearch::user { 'myuser':
  password => 'mypassword',
  roles    => ['myrole'],
}

The

password

esusers

users

users

elasticsearch::user { 'myuser':
  password => '$2a$10$IZMnq6DF4DtQ9c4sVovgDubCbdeH62XncmcyD1sZ4WClzFuAdqspy',
  roles    => ['myrole'],
}

Note: When using the

esusers

users

elasticsearch::user

notify { 'update password': } ~>
elasticsearch::user { 'myuser':
  password => 'mynewpassword',
  roles    => ['myrole'],
}

Certificates

SSL/TLS can be enabled by providing the appropriate class params with paths to the certificate and private key files, and a password for the keystore.

class { 'elasticsearch' :
  ssl                  => true,
  ca_certificate       => '/path/to/ca.pem',
  certificate          => '/path/to/cert.pem',
  private_key          => '/path/to/key.pem',
  keystore_password    => 'keystorepassword',
}

Note: Setting up a proper CA and certificate infrastructure is outside the scope of this documentation, see the aforementioned security guide for more information regarding the generation of these certificate files.

The module will set up a keystore file for the node to use and set the relevant options in

elasticsearch.yml

System Keys

System keys can be passed to the module, where they will be placed into individual instance configuration directories. This can be set at the

elasticsearch

class { 'elasticsearch':
  system_key => 'puppet:///path/to/key',
}

Licensing

If you use the aforementioned security features, you may need to install a user license to leverage particular features outside of a trial license. This module can handle installation of licenses without the need to write custom

exec

curl

You may instruct the module to install a license through the

elasticsearch::license

class { 'elasticsearch':
  license => $license,
}

The

license

api_*

The native provider for licenses will not print license signatures as part of Puppet's changelog to ensure that sensitive values are not included in console output or Puppet reports. Any fields present in the

license

POST

signature

Data directories

There are several different ways of setting data directories for Elasticsearch. In every case the required configuration options are placed in the

elasticsearch.yml

Default

By default we use:

/var/lib/elasticsearch

Which mirrors the upstream defaults.

Single global data directory

It is possible to override the default data directory by specifying the

datadir

class { 'elasticsearch':
  datadir => '/var/lib/elasticsearch-data'
}

Multiple Global data directories

It's also possible to specify multiple data directories using the

datadir

class { 'elasticsearch':
  datadir => [ '/var/lib/es-data1', '/var/lib/es-data2']
}

See the Elasticsearch documentation for additional information regarding this configuration.

Elasticsearch configuration

The

config

Configuration writeup

The

config

Full hash writeup

Instead of writing the full hash representation:

class { 'elasticsearch':
  config                 => {
   'cluster'             => {
     'name'              => 'ClusterName',
     'routing'           => {
        'allocation'     => {
          'awareness'    => {
            'attributes' => 'rack'
          }
        }
      }
    }
  }
}

Short hash writeup

class { 'elasticsearch':
  config => {
    'cluster' => {
      'name' => 'ClusterName',
      'routing.allocation.awareness.attributes' => 'rack'
    }
  }
}

Keystore Settings

Recent versions of Elasticsearch include the elasticsearch-keystore utility to create and manage the

elasticsearch.keystore

config

elasticsearch::instance

elasticsearch.keystore

elasticsearch-keystore

secrets

For example, to define cloud plugin credentials for all instances:

class { 'elasticsearch':
  secrets => {
    'cloud.aws.access_key' => 'AKIA....',
    'cloud.aws.secret_key' => 'AKIA....',
  }
}

Purging Secrets

By default, if a secret setting exists on-disk that is not present in the

secrets

secrets

purge_secrets

elasticsearch

Notifying Services

Any changes to keystore secrets will notify running elasticsearch services by respecting the

restart_on_change

restart_config_change

Reference

Class parameters are available in the auto-generated documentation pages. Autogenerated documentation for types, providers, and ruby helpers is also available on the same documentation site.

Limitations

This module is built upon and tested against the versions of Puppet listed in the metadata.json file (i.e. the listed compatible versions on the Puppet Forge).

The module has been tested on:

• Amazon Linux 1/2
• Debian 8/9/10
• CentOS 7/8
• OracleLinux 7/8
• Ubuntu 16.04, 18.04, 20.04
• SLES 12

Testing on other platforms has been light and cannot be guaranteed.

Development

Please see the CONTRIBUTING.md file for instructions regarding development environments and testing.

Support

Need help? Join us in #elasticsearch on Freenode IRC or on the discussion forum.