Need help with docuum?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

stepchowfun
292 Stars 14 Forks Other 395 Commits 4 Opened issues

Description

Docuum performs least recently used (LRU) eviction of Docker images. 🗑️

Services available

!
?

Need anything else?

Contributors list

# 158,614
Rust
Shell
Docker
ci
168 commits
# 393,851
Go
Shell
redis-c...
Redis
12 commits
# 307,440
HTML
helm-ch...
Shell
C#
12 commits
# 21,659
bitbuck...
html5
vuejs
Clojure
9 commits
# 149,753
PHP
reposit...
Zsh
Vim
7 commits

Docuum: LRU eviction of Docker images

Build status

Docuum performs least recently used (LRU) eviction of Docker images to keep the disk usage below a given threshold.

Docker's built-in

docker image prune --all --filter until=…
command serves a similar purpose. However, the built-in solution isn't ideal since it uses the image creation time, rather than the last usage time, to determine which images to remove. That means it can delete frequently used images, which may be expensive to rebuild.

Docuum is ideal for use cases such as continuous integration (CI) workers, developer workstations, or any other environment in which Docker images accumulate on disk over time. Docuum works well with tools like Toast and Docker Compose.

Docuum is used by Airbnb on its fleet of 1.5k+ CI workers.

How it works

Docker doesn't record when an image was last used. To work around this, Docuum listens for notifications via

docker events
to learn when images are used. It maintains a small piece of state in a local data directory (see this for details about where this directory is on various platforms). That persisted state allows you to freely restart Docuum (or the whole machine) without losing the image usage timestamp data.

When Docuum first starts and subsequently whenever a new Docker event comes in, LRU eviction is performed until the total disk usage due to Docker images is below the given threshold. This design has a few advantages over evicting images based on a fixed time to live (TTL), which is what various other tools in the Docker ecosystem do:

  1. There is no need to configure and tune an interval to run on. Docuum evicts images immediately whenever the disk usage exceeds the threshold without waiting for any timers.
  2. Docuum uses no CPU resources when there is no Docker activity. You can run it on your laptop without worrying about draining your battery.
  3. In order to prevent your disk from filling up, it's more straightforward to set a threshold based on disk usage rather than guessing an appropriate maximum image age.

Docuum also respects the parent-child relationships between images. In particular, it will delete children of a parent before deleting the parent (even if the children were used more recently than the parent), because Docker doesn't allow images with children to be deleted.

Usage

Once Docuum is installed, you can run it manually from the command line as follows:

docuum --threshold '10 GB'

Docuum will then start listening for Docker events. You can use

Ctrl
+
C
to stop it.

You probably want to run Docuum as a daemon, e.g., with launchd, systemd, etc. See the Configuring your operating system to run the binary as a daemon section below for instructions.

Here are the supported command-line options:

USAGE:
    docuum

OPTIONS: -h, --help Prints help information

-k, --keep <regex>...
        Prevents deletion of images for which repository:tag matches <regex>

-t, --threshold <threshold>
        Sets the maximum amount of space to be used for Docker images (default: 10 GB)

-v, --version
        Prints version information

The

--threshold
flag accepts multiple representations, like
10 GB
,
10 GiB
, or
10GB
.

Installation instructions

Installation consists of two steps:

  1. Installing the binary
  2. Configuring your operating system to run the binary as a daemon

Installing the binary

Installation on macOS or Linux (x86-64)

If you're running macOS or Linux on an x86-64 CPU, you can install Docuum with this command:

curl https://raw.githubusercontent.com/stepchowfun/docuum/main/install.sh -LSfs | sh

The same command can be used again to update to the latest version.

The installation script supports the following optional environment variables:

  • VERSION=x.y.z
    (defaults to the latest version)
  • PREFIX=/path/to/install
    (defaults to
    /usr/local/bin
    )

For example, the following will install Docuum into the working directory:

curl https://raw.githubusercontent.com/stepchowfun/docuum/main/install.sh -LSfs | PREFIX=. sh

If you prefer not to use this installation method, you can download the binary from the releases page, make it executable (e.g., with

chmod
), and place it in some directory in your
PATH
(e.g.,
/usr/local/bin
).

Installation on Windows (x86-64)

If you're running Windows on an x86-64 CPU, download the latest binary from the releases page and rename it to

docuum
(or
docuum.exe
if you have file extensions visible). Create a directory called
Docuum
in your
%PROGRAMFILES%
directory (e.g.,
C:\Program Files\Docuum
), and place the renamed binary in there. Then, in the "Advanced" tab of the "System Properties" section of Control Panel, click on "Environment Variables..." and add the full path to the new
Docuum
directory to the
PATH
variable under "System variables". Note that the
Program Files
directory might have a different name if Windows is configured for language other than English.

To update to an existing installation, simply replace the existing binary.

Installation with Homebrew

If you have Homebrew, you can install Docuum as follows:

brew install docuum

You can update an existing installation with

brew upgrade docuum
.

Installation with Cargo

If you have Cargo, you can install Docuum as follows:

cargo install docuum

You can run that command with

--force
to update an existing installation.

Running Docuum in a Docker container on a host capable of running Linux containers

If you prefer not to install Docuum on your system and you're running macOS or Linux on an x86-64 CPU, you can run it in a container:

docker run \
  --init \
  --rm \
  --tty \
  --name docuum \
  --volume /var/run/docker.sock:/var/run/docker.sock \
  --volume docuum:/root \
  stephanmisc/docuum --threshold '10 GB'

If you're on a Windows system configured to run Linux containers, use this command:

docker run `
  --init `
  --rm `
  --tty `
  --name docuum `
  --volume //var/run/docker.sock:/var/run/docker.sock `
  --volume docuum:/root `
  stephanmisc/docuum --threshold '10 GB'

We don't currently publish a Windows-based image, because some Windows machines (namely, those which run containers with process isolation rather than Hyper-V) can only run Windows containers that were built for the exact build of Windows (e.g., 1809) which is running on the host. This makes Windows-based images less portable, and as a result we'd need to publish a separate Windows-based image for each build of Windows we want to support. At this time, we don't have the infrastructure to do that.

The instructions below for configuring your operating system to run Docuum as a daemon assume it's installed as an executable binary. If you prefer to run it as a Docker container, change the relevant service definition to run a Docker command like the relevant one above, with the following adjustments:

  • Omit the
    --tty
    flag. This prevents Docuum from printing colored logs, which you probably don't want for a daemon.
  • Configure Docker as a hard dependency. Ordinarily, Docuum and Docker can be started in any order, and Docuum will patiently wait for Docker to start if needed. However, when running Docuum as a Docker container, then of course Docker must be started first.

Configuring your operating system to run the binary as a daemon

Creating a launchd service on macOS

On macOS, launchd can be used to run Docuum as a daemon. Create a file (owned by root) called

/Library/LaunchDaemons/local.docuum.plist
with the following contents, adjusting the arguments as needed:
    
        Label
        local.docuum
        Program
        /usr/local/bin/docuum
        ProgramArguments
        
            /usr/local/bin/docuum
            --threshold
            10 GB
        
        StandardOutPath
        /var/log/docuum.log
        StandardErrorPath
        /var/log/docuum.log
        EnvironmentVariables
        
            PATH
            /bin:/usr/bin:/usr/local/bin
        
        KeepAlive
        
    

Run

sudo launchctl load /Library/LaunchDaemons/local.docuum.plist
to start the service. You can view the logs with
tail -F /var/log/docuum.log
.

Creating a systemd service on Linux

On most Linux distributions, systemd can be used to run Docuum as a daemon. Create a file (owned by root) called

/etc/systemd/system/docuum.service
with the following contents, adjusting the arguments as needed:
[Unit]
Description=Docuum
After=docker.service
Wants=docker.service

[Service] Environment='THRESHOLD=10 GB' ExecStart=/usr/local/bin/docuum --threshold ${THRESHOLD} Restart=on-failure

[Install] WantedBy=multi-user.target

Run

sudo systemctl enable docuum --now
to enable and start the service. You can view the logs with
sudo journalctl --follow --unit docuum
.

Creating an NSSM service on Windows

On Windows, NSSM, the "Non-Sucking Service Manager", can be used to run Docuum as a daemon. Install NSSM by downloading the binary and adding it to your

PATH
(see the Installation on Windows (x86-64) section for instructions on how to configure this environment variable), then run Windows Terminal as Administrator and enter the following command:
nssm install Docuum

NSSM will then open a configuration window. Configure the following:

  • In the
    Application
    tab, select the path to the Docuum binary. You can optionally add arguments like
    --threshold "10 GB"
    .
  • Optionally, in the
    I/O
    tab, choose where you want the logs to be written.

Then click the

Install service
button. Back in Windows Terminal, run the following to start the service:
nssm start Docuum

If you configured a path for the log file in the

I/O
tab of the installation window, you can view those logs with
Get-Content -Wait docuum.log
(adjusting the file path as needed).

Requirements

  • Docuum requires Docker Engine 17.03.0 or later.
    • If you are using Docker Engine 18.09.0 or later with BuildKit mode enabled, Docker does not create intermediate images for each build step and instead uses a separate "build cache". Docuum will only clean up images, not the Buildkit build cache. BuildKit's built-in garbage collection feature can be used for the build cache (e.g.,
      docker builder prune --all --force --keep-storage '10 GB'
      ). If you are not using BuildKit mode, Docker's caching mechanism uses intermediate images, and Docuum will happily vacuum such images as usual.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.