Need help with storagetapper?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

268 Stars 55 Forks MIT License 248 Commits 21 Opened issues


StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service

Services available


Need anything else?

Contributors list



Build Status Go Report Card codecov

StorageTapper is a scalable realtime MySQL change data streaming, logical backup and logical replication service.

Storagetapper is deployed in production at Uber and used to produce snapshot and realtime changed data of thousands of MySQL tables across multiple datacenters.

It is also used as a backup service to snapshot hundreds of terrabytes of Schemaless data to HDFS and S3 with optional asymmetric encryption and compression.

It reads data from source transforms according to the specified event format and produces data to destination.

Supported event sources: * MySQL * Schemaless

Supported event destinations: * Kafka * HDFS * S3 * Local file * MySQL (experimental) * Postgres (experimental) * Clickhouse (experimental)

Supported event formats: * Avro * JSON * MsgPack * SQL

Features: * Fields filtering * Snapshot row filtering * Producing realtime changes only, i.e. omitting initial snapshot * Snapshot and changelog events ordering * Periodic snapshots * Snapshot index hinting

Storagetapper keeps it jobs state in MySQL database and automatically distribute jobs between configured number of workers.

It is also aware of node roles and takes snapshot from the slave nodes in order to reduce load on master nodes. It can also optionally further throttles the reads. Binlogs are streamed from master nodes for better SLAs.

Service is dynamically configurable through RESTful API or builtin UI.

Build & Install

Debian & Ubuntu

cd storagetapper
make deb && dpkg -i ../storagetapper_1.0_amd64.deb


cd storagetapper
make && make install



/bin/bash scripts/ # install all dependencies: MySQL, Kafka, HDFS, S3, ...
make test # run all tests
GO111MODULE=on TEST_PARAM="" /bin/bash scripts/ ./pipe # individual test

Non Linux

make test-env
$ make test


Storagetapper loads configuration from the following files and location in the given order:


Available options described in Configuration section


This software is licensed under the MIT License.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.