ansible-role-proxmox

by lae

Deploys and configures Proxmox VE 5.x/6.x clusters.

148 Stars 53 Forks Last release: 3 months ago (v1.6.5) MIT License 280 Commits 22 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Build Status Galaxy Role

lae.proxmox

Installs and configures a Proxmox 5.x/6.x cluster with the following features:

  • Ensures all hosts can connect to one another as root
  • Ability to create/manage groups, users, access control lists and storage
  • Ability to create or add nodes to a PVE cluster
  • Ability to setup Ceph on the nodes
  • IPMI watchdog support
  • BYO HTTPS certificate support
  • Ability to use either
    pve-no-subscription
    or
    pve-enterprise
    repositories

Quickstart

The primary goal for this role is to configure and manage a Proxmox VE cluster (see example playbook), however this role can be used to quickly install single node Proxmox servers.

I'm assuming you already have Ansible installed. You will need to use an external machine to the one you're installing Proxmox on (primarily because of the reboot in the middle of the installation, though I may handle this somewhat differently for this use case later).

Copy the following playbook to a file like

install_proxmox.yml
:
- hosts: all
  become: True
  roles:
    - {
        role: geerlingguy.ntp,
        ntp_manage_config: true,
        ntp_servers: [
          clock.sjc.he.net,
          clock.fmt.he.net,
          clock.nyc.he.net
        ]
      }
    - {
        role: lae.proxmox,
        pve_group: all,
        pve_reboot_on_kernel_update: true
      }

Install this role and a role for configuring NTP:

ansible-galaxy install lae.proxmox geerlingguy.ntp

Now you can perform the installation:

ansible-playbook install_proxmox.yml -i $SSH_HOST_FQDN, -u $SSH_USER

If your

SSH_USER
has a sudo password, pass the
-K
flag to the above command. If you also authenticate to the host via password instead of pubkey auth, pass the
-k
flag (make sure you have
sshpass
installed as well). You can set those variables prior to running the command or just replace them. Do note the comma is important, as a list is expected (otherwise it'll attempt to look up a file containing a list of hosts).

Once complete, you should be able to access your Proxmox VE instance at

https://$SSH_HOST_FQDN:8006
.

Support/Contributing

For support or if you'd like to contribute to this role but want guidance, feel free to join this Discord server: https://discord.gg/cjqr6Fg

Deploying a fully-featured PVE 5.x cluster

Create a new playbook directory. We call ours

lab-cluster
. Our playbook will eventually look like this, but yours does not have to follow all of the steps:
lab-cluster/
├── files
│   └── pve01
│       ├── lab-node01.local.key
│       ├── lab-node01.local.pem
│       ├── lab-node02.local.key
│       ├── lab-node02.local.pem
│       ├── lab-node03.local.key
│       └── lab-node03.local.pem
├── group_vars
│   ├── all
│   └── pve01
├── inventory
├── roles
│   └── requirements.yml
├── site.yml
└── templates
    └── interfaces-pve01.j2

6 directories, 12 files

First thing you may note is that we have a bunch of

.key
and
.pem
files. These are private keys and SSL certificates that this role will use to configure the web interface for Proxmox across all the nodes. These aren't necessary, however, if you want to keep using the signed certificates by the CA that Proxmox sets up internally. You may typically use Ansible Vault to encrypt the private keys, e.g.:
ansible-vault encrypt files/pve01/*.key

This would then require you to pass the Vault password when running the playbook.

Let's first specify our cluster hosts. Our

inventory
file may look like this:
[pve01]
lab-node01.local
lab-node02.local
lab-node03.local

You could have multiple clusters, so it's a good idea to have one group for each cluster. Now, let's specify our role requirements in

roles/requirements.yml
:
---
- src: geerlingguy.ntp
- src: lae.proxmox

We need an NTP role to configure NTP, so we're using Jeff Geerling's role to do so. You wouldn't need it if you already have NTP configured or have a different method for configuring NTP.

Now, let's specify some group variables. First off, let's create

group_vars/all
for setting NTP-related variables:
---
ntp_manage_config: true
ntp_servers:
  - lab-ntp01.local iburst
  - lab-ntp02.local iburst

Of course, replace those NTP servers with ones you prefer.

Now for the flesh of your playbook,

pve01
's group variables. Create a file
group_vars/pve01
, add the following, and modify accordingly for your environment.
---
pve_group: pve01
pve_fetch_directory: "fetch/{{ pve_group }}/"
pve_watchdog: ipmi
pve_ssl_private_key: "{{ lookup('file', pve_group + '/' + inventory_hostname + '.key') }}"
pve_ssl_certificate: "{{ lookup('file', pve_group + '/' + inventory_hostname + '.pem') }}"
pve_cluster_enabled: yes
pve_groups:
  - name: ops
    comment: Operations Team
pve_users:
  - name: [email protected]
    email: [email protected]
    firstname: Admin
    lastname: User 1
    groups: [ "ops" ]
  - name: [email protected]
    email: [email protected]
    firstname: Admin
    lastname: User 2
    groups: [ "ops" ]
pve_acls:
  - path: /
    roles: [ "Administrator" ]
    groups: [ "ops" ]
pve_storages:
  - name: localdir
    type: dir
    content: [ "images", "iso", "backup" ]
    path: /plop
    maxfiles: 4
pve_ssh_port: 22

interfaces_template: "interfaces-{{ pve_group }}.j2"

pve_group
is set to the group name of our cluster,
pve01
- it will be used for the purposes of ensuring all hosts within that group can connect to each other and are clustered together. Note that the PVE cluster name will be set to this group name as well, unless otherwise specified by
pve_clustername
. Leaving this undefined will default to
proxmox
.

pve_fetch_directory
will be used to download the host public key and root user's public key from all hosts within
pve_group
. These are then uploaded to each host into the appropriate configuration files.

pve_watchdog
here enables IPMI watchdog support and configures PVE's HA manager to use it. Leave this undefined if you don't want to configure it.

pve_ssl_private_key
and
pve_ssl_certificate
point to the SSL certificates for pvecluster. Here, a file lookup is used to read the contents of a file in the playbook, e.g.
files/pve01/lab-node01.key
. You could possibly just use host variables instead of files, if you prefer.

pve_ssl_letsencrypt
allows to obtain a Let's Encrypt SSL certificate for pvecluster. The Ansible role systemli.letsencrypt needs to be installed first in order to use this function.

pve_cluster_enabled
enables the role to perform all cluster management tasks. This includes creating a cluster if it doesn't exist, or adding nodes to the existing cluster. There are checks to make sure you're not mixing nodes that are already in existing clusters with different names.

pve_groups
,
pve_users
, and
pve_acls
authorizes some local UNIX users (they must already exist) to access PVE and gives them the Administrator role as part of the
ops
group. Read the User and ACL Management section for more info.

pve_storages
allows to create different types of storage and configure them. The backend needs to be supported by Proxmox. Read the Storage Management section for more info.

pve_ssh_port
allows you to change the SSH port. If your SSH is listening on a port other than the default 22, please set this variable. If a new node is joining the cluster, the PVE cluster needs to communicate once via SSH.

pve_manage_ssh
(default true) allows you to disable any changes this module would make to your SSH server config. This is useful if you use another role to manage your SSH server. Note that setting this to false is not officially supported, you're on your own to replicate the changes normally made in sshclusterconfig.yml.

interfaces_template
is set to the path of a template we'll use for configuring the network on these Debian machines. This is only necessary if you want to manage networking from Ansible rather than manually or via each host in PVE. You should probably be familiar with Ansible prior to doing this, as your method may involve setting host variables for the IP addresses for each host, etc.

Let's get that interface template out of the way. Feel free to skip this file (and leave it undefined in

group_vars/pve01
) otherwise. Here's one that I use:
# {{ ansible_managed }}
auto lo
iface lo inet loopback

allow-hotplug enp2s0f0 iface enp2s0f0 inet manual

auto vmbr0 iface vmbr0 inet static address {{ lookup('dig', ansible_fqdn) }} gateway 10.4.0.1 netmask 255.255.255.0 dns-nameservers 10.2.2.4 10.3.2.4 dns-search local bridge_ports enp2s0f0 bridge_stp off bridge_fd 0

allow-hotplug enp2s0f1 auto enp2s0f1 iface enp2s0f1 inet static address {{ lookup('dig', ansible_hostname + "-clusternet.local") }} netmask 255.255.255.0

You might not be familiar with the

dig
lookup, but basically here we're doing an A record lookup for each machine (e.g. lab-node01.local) for the first interface (and configuring it as a bridge we'll use for VM interfaces), and then another slightly modified lookup for the "clustering" network we might use for Ceph ("lab-node01-clusternet.local"). Of course, yours may look completely different, especially if you're using bonding, three different networks for management/corosync, storage and VM traffic, etc.

Finally, let's write our playbook.

site.yml
will look something like this:
---
- hosts: all
  become: True
  roles:
    - geerlingguy.ntp

Leave this out if you're not modifying networking through Ansible

  • hosts: pve01 become: True serial: 1 tasks:

    • name: Install bridge-utils apt: name: bridge-utils

    • name: Configure /etc/network/interfaces template: src: "{{ interfaces_template }}" dest: /etc/network/interfaces register: _configure_interfaces

    • block:

      • name: Reboot for networking changes shell: "sleep 5 && shutdown -r now 'Networking changes found, rebooting'" async: 1 poll: 0

      • name: Wait for server to come back online wait_for_connection: delay: 15 when: _configure_interfaces is changed

  • hosts: pve01 become: True roles:

    • lae.proxmox

Basically, we run the NTP role across all hosts (you might want to add some non-Proxmox machines), configure networking on

pve01
with our separate cluster network and bridge layout, reboot to make those changes take effect, and then run this Proxmox role against the hosts to setup a cluster.

At this point, our playbook is ready and we can run the playbook.

Ensure that roles and dependencies are installed:

ansible-galaxy install -r roles/requirements.yml --force
pip install jmespath dnspython

jmespath
is required for some of the tasks involving clustering.
dnspython
is only required if you're using a
dig
lookup, which you probably won't be if you skipped configuring networking. We pass
--force
to
ansible-galaxy
here so that roles are updated to their latest versions if already installed.

Now run the playbook:

ansible-playbook -i inventory site.yml -e '{"pve_reboot_on_kernel_update": true}'

The

-e '{"pve_reboot_on_kernel_update": true}'
should mainly be run the first time you do the Proxmox cluster setup, as it'll reboot the server to boot into a PVE kernel. Subsequent runs should leave this out, as you want to sequentially reboot servers after the cluster is running.

To specify a particular user, use

-u root
(replacing
root
), and if you need to provide passwords, use
-k
for SSH password and/or
-K
for sudo password. For example:
ansible-playbook -i inventory site.yml -K -u admin1

This will ask for a sudo password, then login to the

admin1
user (using public key auth - add
-k
for pw) and run the playbook.

That's it! You should now have a fully deployed Proxmox cluster. You may want to create Ceph storage on it afterwards (see Ceph for more info) and other tasks possibly, but the hard part is mostly complete.

Example Playbook

This will configure hosts in the group

pve01
as one cluster, as well as reboot the machines should the kernel have been updated. (Only recommended to set this flag during installation - reboots during operation should occur serially during a maintenance period.) It will also enable the IPMI watchdog.
- hosts: pve01
  become: True
  roles:
    - {
        role: geerlingguy.ntp,
        ntp_manage_config: true,
        ntp_servers: [
          clock.sjc.he.net,
          clock.fmt.he.net,
          clock.nyc.he.net
        ]
      }
    - {
        role: lae.proxmox,
        pve_group: pve01,
        pve_cluster_enabled: yes,
        pve_reboot_on_kernel_update: true,
        pve_watchdog: ipmi
      }

Role Variables

[variable]: [default] #[description/purpose]
pve_group: proxmox # host group that contains the Proxmox hosts to be clustered together
pve_fetch_directory: fetch/ # local directory used to download root public keys from each host to
pve_repository_line: "deb http://download.proxmox.com/debian/pve stretch pve-no-subscription" # apt-repository configuration - change to enterprise if needed (although TODO further configuration may be needed)
pve_remove_subscription_warning: true # patches the subscription warning messages in proxmox if you are using the community edition
pve_extra_packages: [] # Any extra packages you may want to install, e.g. ngrep
pve_run_system_upgrades: false # Let role perform system upgrades
pve_run_proxmox_upgrades: true # Let role perform Proxmox VE upgrades
pve_check_for_kernel_update: true # Runs a script on the host to check kernel versions
pve_reboot_on_kernel_update: false # If set to true, will automatically reboot the machine on kernel updates
pve_remove_old_kernels: true # Currently removes kernel from main Debian repository
pve_watchdog: none # Set this to "ipmi" if you want to configure a hardware watchdog. Proxmox uses a software watchdog (nmi_watchdog) by default.
pve_watchdog_ipmi_action: power_cycle # Can be one of "reset", "power_cycle", and "power_off".
pve_watchdog_ipmi_timeout: 10 # Number of seconds the watchdog should wait
pve_zfs_enabled: no # Specifies whether or not to install and configure ZFS packages
# pve_zfs_options: "" # modprobe parameters to pass to zfs module on boot/modprobe
# pve_zfs_zed_email: "" # Should be set to an email to receive ZFS notifications
pve_ceph_enabled: false # Specifies wheter or not to install and configure Ceph packages. See below for an example configuration.
pve_ceph_repository_line: "deb http://download.proxmox.com/debian/ceph-nautilus buster main" # apt-repository configuration. Will be automatically set for 5.x and 6.x (Further information: https://pve.proxmox.com/wiki/Package_Repositories)
pve_ceph_network: "{{ (ansible_default_ipv4.network +'/'+ ansible_default_ipv4.netmask) | ipaddr('net') }}" # Ceph public network
# pve_ceph_cluster_network: "" # Optional, if the ceph cluster network is different from the public network (see https://pve.proxmox.com/pve-docs/chapter-pveceph.html#pve_ceph_install_wizard)
pve_ceph_mon_group: "{{ pve_group }}" # Host group containing all Ceph monitor hosts
pve_ceph_mds_group: "{{ pve_group }}" # Host group containing all Ceph metadata server hosts
pve_ceph_osds: [] # List of OSD disks
pve_ceph_pools: [] # List of pools to create
pve_ceph_fs: [] # List of CephFS filesystems to create
pve_ceph_crush_rules: [] # List of CRUSH rules to create
# pve_ssl_private_key: "" # Should be set to the contents of the private key to use for HTTPS
# pve_ssl_certificate: "" # Should be set to the contents of the certificate to use for HTTPS
pve_ssl_letsencrypt: false # Specifies whether or not to obtain a SSL certificate using Let's Encrypt
pve_groups: [] # List of group definitions to manage in PVE. See section on User Management.
pve_users: [] # List of user definitions to manage in PVE. See section on User Management.
pve_storages: [] # List of storages to manage in PVE. See section on Storage Management.
pve_datacenter_cfg: {} # Dictionary to configure the PVE datacenter.cfg config file.

To enable clustering with this role, configure the following variables appropriately:

pve_cluster_enabled: no # Set this to yes to configure hosts to be clustered together
pve_cluster_clustername: "{{ pve_group }}" # Should be set to the name of the PVE cluster

The following variables are used to provide networking information to corosync. These are known as ring0addr/ring1addr or link0addr/link1addr, depending on PVE version. They should be IPv4 or IPv6 addresses. For more information, refer to the Cluster Manager chapter in the PVE Documentation.

# pve_cluster_addr0: "{{ ansible_default_ipv4.address }}"
# pve_cluster_addr1: "another interface's IP address or hostname"

You can set options in the datacenter.cfg configuration file:

pve_datacenter_cfg:
  keyboard: en-us

You can also configure HA manager groups:

pve_cluster_ha_groups: [] # List of HA groups to create in PVE.

This example creates a group "labnode01" for resources assigned to the lab-node01 host: ``` pveclusterhagroups: - name: lab_node01 comment: "My HA group" nodes: "lab-node01" nofailback: 0 restricted: 0 ```

All configuration options supported in the datacenter.cfg file are documented in the Proxmox manual datacenter.cfg section.

Dependencies

This role does not install NTP, so you should configure NTP yourself, e.g. with the

geerlingguy.ntp
role as shown in the example playbook.

When clustering is enabled, this role makes use of the

json_query
filter, which requires that the
jmespath
library be installed on your control host. You can either
pip install jmespath
or install it via your distribution's package manager, e.g.
apt-get install python-jmespath
.

User and ACL Management

You can use this role to manage users and groups within Proxmox VE (both in single server deployments and cluster deployments). Here are some examples.

pve_groups:
  - name: Admins
    comment: Administrators of this PVE cluster
  - name: api_users
  - name: test_users
pve_users:
  - name: [email protected]
    email: [email protected]
  - name: [email protected]
    email: [email protected]
    firstname: Musee
    lastname: Ullah
    groups: [ "Admins" ]
  - name: [email protected]
    password: "Proxmox789"
    groups:
      - api_users
  - name: [email protected]
    password: "Test456"
    enable: no
    groups:
      - api_users
      - test_users
  - name: [email protected]
    expire: 1514793600
    groups: [ "test_users" ]
    comment: "Temporary user set to expire on 2018年  1月  1日 月曜日 00:00:00 PST"
    email: [email protected]
    firstname: Test
    lastname: User

Refer to

library/proxmox_user.py
link and
library/proxmox_group.py
link for module documentation.

For managing ACLs, a similar module is employed, but the main difference is that most of the parameters only accept lists (subject to change):

pve_acls:
  - path: /
    roles: [ "Administrator" ]
    groups: [ "Admins" ]
  - path: /pools/testpool
    roles: [ "PVEAdmin" ]
    users:
      - [email protected]
    groups:
      - test_users

Refer to

library/proxmox_acl.py
link for module documentation.

Storage Management

You can use this role to manage storage within Proxmox VE (both in single server deployments and cluster deployments). For now, the only supported types are

dir
,
rbd
,
nfs
,
lvm
and
lvmthin
. Here are some examples.
pve_storages:
  - name: dir1
    type: dir
    content: [ "images", "iso", "backup" ]
    path: /ploup
    disable: no
    maxfiles: 4
  - name: ceph1
    type: rbd
    content: [ "images", "rootdir" ]
    nodes: [ "lab-node01.local", "lab-node02.local" ]
    username: admin
    pool: rbd
    krbd: yes
    monhost:
      - 10.0.0.1
      - 10.0.0.2
      - 10.0.0.3
  - name: nfs1
    type: nfs
    content: [ "images", "iso" ]
    server: 192.168.122.2
    export: /data
  - name: lvm1
    type: lvm
    content: [ "images", "rootdir" ]
    vgname: vg1
  - name: lvmthin1
    type: lvmthin
    content: [ "images", "rootdir" ]
    vgname: vg2
    thinpool: data

Refer to

library/proxmox_storage.py
link for module documentation.

Ceph configuration

This section could use a little more love. If you are actively using this role to manage your PVE Ceph cluster, please feel free to flesh this section more thoroughly and open a pull request! See issue #68.

PVE Ceph management with this role is experimental. While users have successfully used this role to deploy PVE Ceph, it is not fully tested in CI (due to a lack of usable block devices to use as OSDs in Travis CI). Please deploy a test environment with your configuration first prior to prod, and report any issues if you run into any.

This role can configure the Ceph storage system on your Proxmox hosts. The following definitions show some of the configurations that are possible.

pve_ceph_enabled: true
pve_ceph_network: '172.10.0.0/24'
pve_ceph_cluster_network: '172.10.1.0/24'
pve_ceph_osds:
  # OSD with everything on the same device
  - device: /dev/sdc
  # OSD with block.db/WAL on another device
  - device: /dev/sdd
    block.db: /dev/sdb1
# Crush rules for different storage classes
# By default 'type' is set to host, you can find valid types at (https://docs.ceph.com/en/latest/rados/operations/crush-map/)
# listed under 'TYPES AND BUCKETS'
pve_ceph_crush_rules:
  - name: ssd
    class: ssd
    type: osd
  - name: hdd
    class: hdd
    type: host
# 2 Ceph pools for VM disks which will also be defined as Proxmox storages
# Using different CRUSH rules
pve_ceph_pools:
  - name: ssd
    pgs: 128
    rule: ssd
    application: rbd
    storage: true
# This Ceph pool uses custom size/replication values
  - name: hdd
    pgs: 32
    rule: hdd
    application: rbd
    storage: true
    size: 2
    min-size: 1
pve_ceph_fs:
# A CephFS filesystem not defined as a Proxmox storage
  - name: backup
    pgs: 64
    rule: hdd
    storage: false
    mountpoint: /srv/proxmox/backup

pve_ceph_network
by default uses the
ipaddr
filter, which requires the
netaddr
library to be installed and usable by your Ansible controller.

Contributors

Musee Ullah (@lae, [email protected]) - Main developer
Fabien Brachere (@Fbrachere) - Storage config support
Gaudenz Steinlin (@gaundez) - Ceph support, etc
Thoralf Rickert-Wendt (@trickert76) - PVE 6.x support, etc
Engin Dumlu (@roadrunner)
Jonas Meurer (@mejo-)
Ondrej Flidr (@SniperCZE)
niko2 (@niko2)
Christian Aublet (@caublet)
Michael Holasek (@mholasek)

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.