Decentralized, k-ordered unique IDs in Clojure
Decentralized, k-ordered unique ID generator.
This is a Clojure implementation of Boundary's Erlang Flake ID service.
New flake IDs can be generated with the
generate!fn from flake's core namespace.
Note that in order to prevent generation of duplicate IDs,
init!must be called prior to generating IDs for the first time.
=> (require '[flake.core :as flake]) => (flake/init!) => (map flake/flake->bigint (take 3 (repeatedly flake/generate!))) (25978563106299135585558915252224N 25978563106299135585558915252225N 25978563106299135585558915252226N)
Here we have generated three BigIntegers which are flake IDs. Note how they are ordered.
It may be desirable to encode these IDs in a shorter representation, such as Base62. The utils namespace provides an encoder:
=> (require '[flake.utils :as utils]) => (->> (repeatedly flake/generate!) first flake/flake->bigint utils/base62-encode) "8mwFA958SJ2CZVu9nk"
A flake's middle-most bits are derived from a hardware address, e.g. MAC. If this is not desirable or the caller wishes to have more granular control over which bits are used here, a custom
worker-idmay be provided:
(import '[java.security SecureRandom])
(defn rand-bytes "Return
nrandom bytes in an array." [n] (let [bs (byte-array n)] (.nextBytes (SecureRandom.) bs) bs))
(def worker-id (rand-bytes 6))
(first (repeatedly (partial flake/generate! worker-id)))
The above specifies a random array of bytes to be used as the worker-id.
Flakes are byte sequences composed of 128 bits. These sequences are structured such that the first 64 bits are a timestamp, i.e. the time since an epoch in milliseconds, the next 48 bits are a unique, machine-specific bitset, normally the MAC, and finally the remaining 16 bits are a monotonically increasing short.
A diagram of the flake byte structure:
Exceptions may occur if a single machine generates more than 65,535 flakes within a millisecond. That puts the upper-bound on flake generation per machine at ~65 million flakes per second. If this limit is reached a
IllegalArgumentExceptionwill be raised.
Additionally flake makes an effort to detect drift in system time and will raise
IllegalStateExceptionif time appears to be flowing in the wrong direction. Note that
init!must be used for this to work properly!