Docker Machine Driver for Proxmox VE

This driver can be used to kickstart a VM in Proxmox VE to be used with Docker/Docker Machine.

• Download and copy it into your

  PATH

  (don't forget to

  chmod +x

  ) or build your own driver

• Check if it works:

  $ docker-machine create --driver proxmoxve --help | grep -c proxmox
  35
  

Operation

Now you have two modes of operation: * use an iso to install a Docker distribution (e.g. RancherOS) * use a previously created cloud-init-based image VM template as a base

There are also other options to customize your VM which are not shown here, so please feel free to explore them with

docker-machine create --driver proxmoxve --help

Preparing a special test user in PVE

If you want to test this docker-machine driver, i strongly recommend to secure it properly. Best way to do this to create a special user that has its own pool and storage for creating the test machines. This corresponds to the examples below.

Here is what I use (based on ZFS):

• create a pool for use as

  --proxmoxve-proxmox-pool docker-machine

  pvesh create /pools -poolid docker-machine
  

• create an user

  docker-machine

  with password

  D0ck3rS3cr3t

  pvesh create /access/users -userid [email protected] -password D0ck3rS3cr3t
  

• creating a special ZFS dataset and use it as PVE storage

  zfs create -o refquota=50G rpool/docker-machine-test
  zfs create rpool/docker-machine-test/iso
  pvesh create /storage -storage docker-machine -type zfspool -pool rpool/docker-machine-test
  pvesh create /storage -storage docker-machine-iso -type dir -path /rpool/docker-machine-test/iso -content iso
  pvesh set /pools/docker-machine -storage docker-machine
  pvesh set /pools/docker-machine -storage docker-machine-iso
  

• set proper permissions for the user

  pvesh set /access/acl -path /pool/docker-machine -roles PVEVMAdmin,PVEDatastoreAdmin,PVEPoolAdmin -users [email protected]
  

If you have additional test storages, you can also add them easily:

    pvesh set /pools/docker-machine -storage nfs
    pvesh set /pools/docker-machine -storage lvm
    pvesh set /pools/docker-machine -storage directory

Ceph is currently not directly tested by me, but there are fine people out there wo tried it.

Clone VM

This approach uses a predefined VM template with cloud-init support to be cloned and used. There a lot of ways to do that, here is an adopted one (courtesy of @travisghansen):

#!/bin/bash

set -x
set -e
export IMGID=9007
export BASE_IMG="debian-10-openstack-amd64.qcow2"
export IMG="debian-10-openstack-amd64-${IMGID}.qcow2"
export STORAGEID="docker-machine"
if [ ! -f "${BASE_IMG}" ];then
  wget https://cloud.debian.org/images/cloud/OpenStack/current-10/debian-10-openstack-amd64.qcow2
fi
if [ ! -f "${IMG}" ];then
  cp -f "${BASE_IMG}" "${IMG}"
fi
prepare mounts
guestmount -a ${IMG} -m /dev/sda1 /mnt/tmp/
mount --bind /dev/ /mnt/tmp/dev/
mount --bind /proc/ /mnt/tmp/proc/
get resolving working
mv /mnt/tmp/etc/resolv.conf /mnt/tmp/etc/resolv.conf.orig
cp -a --force /etc/resolv.conf /mnt/tmp/etc/resolv.conf
install desired apps
chroot /mnt/tmp /bin/bash -c "apt-get update"
chroot /mnt/tmp /bin/bash -c "DEBIAN_FRONTEND=noninteractive apt-get install -y net-tools curl qemu-guest-agent nfs-common open-iscsi lsscsi sg3-utils multipath-tools scsitools"
https://www.electrictoolbox.com/sshd-hostname-lookups/
sed -i 's:#UseDNS no:UseDNS no:' /mnt/tmp/etc/ssh/sshd_config
sed -i '/package-update-upgrade-install/d' /mnt/tmp/etc/cloud/cloud.cfg
cat > /mnt/tmp/etc/cloud/cloud.cfg.d/99_custom.cfg << 'EOF'
#cloud-config
Install additional packages on first boot

Default: none

if packages are specified, this apt_update will be set to true

packages may be supplied as a single package name or as a list
with the format [, ] wherein the specifc
package version will be installed.
#packages:
- qemu-guest-agent
- nfs-common
ntp:
  enabled: true
datasource_list: [ NoCloud, ConfigDrive ]
EOF
cat > /mnt/tmp/etc/multipath.conf << 'EOF'
defaults {
    user_friendly_names yes
    find_multipaths yes
}
EOF
enable services
chroot /mnt/tmp systemctl enable open-iscsi.service || true
chroot /mnt/tmp systemctl enable multipath-tools.service || true
restore systemd-resolved settings
mv /mnt/tmp/etc/resolv.conf.orig /mnt/tmp/etc/resolv.conf
umount everything
umount /mnt/tmp/dev
umount /mnt/tmp/proc
umount /mnt/tmp
create template
qm create ${IMGID} --memory 512 --net0 virtio,bridge=vmbr0
qm importdisk ${IMGID} ${IMG} ${STORAGEID} --format qcow2
qm set ${IMGID} --scsihw virtio-scsi-pci --scsi0 ${STORAGEID}:vm-${IMGID}-disk-0
qm set ${IMGID} --ide2 ${STORAGEID}:cloudinit
qm set ${IMGID} --boot c --bootdisk scsi0
qm set ${IMGID} --serial0 socket --vga serial0
qm template ${IMGID}
set host cpu, ssh key, etc
qm set ${IMGID} --scsihw virtio-scsi-pci
qm set ${IMGID} --cpu host
qm set ${IMGID} --agent enabled=1
qm set ${IMGID} --autostart
qm set ${IMGID} --onboot 1
qm set ${IMGID} --ostype l26
qm set ${IMGID} --ipconfig0 "ip=dhcp"

Adapt to fit your needs and run it on your Proxmox VE until it works without any problems and creates a template in your Proxmox VE. You may need to install

libguestfs-tools

After the image is created, you can start to use the machine driver to create new VMs:

#!/bin/sh
set -ex

export PATH=$PWD:$PATH
PVE_NODE="proxmox"
PVE_HOST="192.168.1.5"
PVE_USER="docker-machine"
PVE_REALM="pve"
PVE_PASSWD="D0ck3rS3cr3t"
PVE_STORAGE_NAME="${1:-docker-machine}"
PVE_POOL="docker-machine"
VM_NAME="docker-clone"
docker-machine rm --force $VM_NAME >/dev/null 2>&1 || true
docker-machine --debug 
    create 
    --driver proxmoxve 
    --proxmoxve-proxmox-host $PVE_HOST 
    --proxmoxve-proxmox-node $PVE_NODE 
    --proxmoxve-proxmox-user-name $PVE_USER 
    --proxmoxve-proxmox-user-password $PVE_PASSWD 
    --proxmoxve-proxmox-realm $PVE_REALM 
    --proxmoxve-proxmox-pool $PVE_POOL 
    
    --proxmoxve-provision-strategy clone 
    --proxmoxve-ssh-username 'debian' 
    --proxmoxve-ssh-password 'geheim' 
    --proxmoxve-vm-clone-vmid 9007 
    
    --proxmoxve-debug-resty 
    --proxmoxve-debug-driver 
    
    $* 
    
    $VM_NAME
eval $(docker-machine env $VM_NAME)
docker ps

Rancher OS

• Use a recent, e.g.

  1.5.6

  version of RancherOS and copy the

  rancheros-proxmoxve-autoformat.iso

  to your iso image storage on your PVE
• Create a script with the following contents and adapt to your needs:

PVE_NODE="proxmox-docker-machine"
PVE_HOST="192.168.1.10"

PVE_USER="docker-machine"
PVE_REALM="pve"
PVE_PASSWD="D0ck3rS3cr3t"
PVE_STORAGE_NAME="docker-machine"
PVE_STORAGE_SIZE="4"
PVE_POOL="docker-machine"
SSH_USERNAME="docker"
SSH_PASSWORD="tcuser"
PVE_MEMORY=2
PVE_CPU_CORES=4
PVE_IMAGE_FILE="docker-machine-iso:iso/rancheros-proxmoxve-autoformat-v1.5.5.iso"
VM_NAME="docker-rancher"
docker-machine rm --force $VM_NAME >/dev/null 2>&1 || true
docker-machine --debug 
    create 
    --driver proxmoxve 
    --proxmoxve-proxmox-host $PVE_HOST 
    --proxmoxve-proxmox-node $PVE_NODE 
    --proxmoxve-proxmox-user-name $PVE_USER 
    --proxmoxve-proxmox-user-password $PVE_PASSWD 
    --proxmoxve-proxmox-realm $PVE_REALM 
    --proxmoxve-proxmox-pool $PVE_POOL 
    
    --proxmoxve-vm-storage-path $PVE_STORAGE_NAME 
    --proxmoxve-vm-storage-size $PVE_STORAGE_SIZE 
    --proxmoxve-vm-cpu-cores $PVE_CPU_CORES 
    --proxmoxve-vm-memory $PVE_MEMORY 
    --proxmoxve-vm-image-file "$PVE_IMAGE_FILE" 
    
    --proxmoxve-ssh-username $SSH_USERNAME 
    --proxmoxve-ssh-password $SSH_PASSWORD 
    
    --proxmoxve-debug-resty 
    --proxmoxve-debug-driver 
    
    $VM_NAME
eval $(docker-machine env $VM_NAME)
docker ps

• Run the script

At the first run, it is advisable to not comment out the

debug

Changes

Version 4

• support for using clones+cloud-init (Thanks to @travisghansen)
• enable custom network bridge without vlan tag (Thanks to @guyguy333)
• including args to choice scsi model (Thanks to @bemanuel)
• fix remove error, add further flags (Thanks to @Psayker)

Version 3

• Renaming driver from

  proxmox-ve

  to

  proxmoxve

  due to identification problem with RancherOS's K8S implementation (Thanks to

  @Sellto

  for reporting #16)
• fixing issue with created disk detection (Thanks to

  @Sellto

  for reporting #16)
• Add

  IPAddress

  property needed by rancher to know the ip address of the created VM (Thanks to

  @Sellto

  )
• Change the name of each flag for better display in the rancher

  Node Templates

  (Thanks to

  @Sellto

  )
• Add number of

  CPU cores configuration paramater

  (Thanks to

  @Sellto

  )
• LVM-thin fixes (Thanks to

  @vstconsulting

  )
• Bridge and VLAN tag support (Thanks to

  @bemanuel

  )
• Fixing filename detection including NFS support

Version 2

• exclusive RancherOS support due to their special Proxmox VE iso files
• adding wait cycles for asynchronous background tasks, e.g.

  create

  ,

  stop

  etc.
• use one logger engine
• add guest username, password and ssh-port as new command line arguments
• more and potentially better error handling

Version 1

• Initial Version