zchunk

zchunk is a compressed file format that splits the file into independent chunks. This allows you to only download changed chunks when downloading a new version of the file, and also makes zchunk files efficient over rsync.

zchunk files are protected with strong checksums to verify that the file you downloaded is, in fact, the file you wanted.

As of zchunk-1.0, the ABI and API have been marked stable, and the only changes allowed are backwards-compatible additions

Installation

To build and install zchunk, first install meson and run

meson build
cd build
ninja
ninja test
sudo ninja install

If you're building on an operating system where some libraries are stored in /usr/local, you'll need to replace

meson build

CFLAGS=-I/usr/local/include CXXFLAGS=-I/usr/local/include LDFLAGS=-L/usr/local/lib meson build

Using the utilities

To decompress a zchunk file, simply run:

unzck 

To compress a new zchunk file, run:

zck 

To download a zchunk file, run:

zckdl -s  

To read a zchunk header, run:

zck_read_header 

Zchunk dictionaries

By default, each chunk in a zchunk file is compressed independently. However, if you're creating a zchunk file that has any repetitive data, you may be able to reduce the overall file size by using a zstd dictionary. The dictionary takes up extra space at the beginning of the zchunk file, but is used as an identical initial dictionary for compressing each chunk, which can give a significant overall savings.

It is important that all further revisions of the zchunk file use the same dictionary. If the dictionary changes, none of the chunks will match from the old file, and the full new file will be downloaded.

Zchunk can use any zstd dictionary, but also includes a utility to generate the ideal zstd dictionary for a zchunk file.

To create an ideal dictionary for a zchunk file, run:

zck_gen_zdict 

The dictionary will be saved as

You will then need to recompress the file with the dictionary:

zck -D 

Note that

zck_gen_zdict

zstd

Documentation

• Format definition
• Initial announcement
• How zchunk works (with pretty pictures)