storehaus

by twitter

twitter / storehaus

Storehaus is a library that makes it easy to work with asynchronous key value stores

462 Stars 81 Forks Last release: over 3 years ago (0.15.0) Apache License 2.0 960 Commits 30 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Storehaus

Build Status Codecov branch Latest version Chat

Storehaus is a library that makes it easy to work with asynchronous key value stores. Storehaus is built on top of Twitter's Future.

Storehaus-Core

Storehaus's core module defines three traits; a read-only

ReadableStore
a write-only
WritableStore
and a read-write
Store
. The traits themselves are tiny:
package com.twitter.storehaus

import com.twitter.util.{ Closable, Future, Time }

trait ReadableStore[-K, +V] extends Closeable { def get(k: K): Future[Option[V]] def multiGet[K1 <: k set map future override def close time future.unit trait writablestore put v multiput kvs.map kv=""> (kv._1, put(kv)) } override def close(time: Time) = Future.Unit }

trait Store[-K, V] extends ReadableStore[K, V] with WritableStore[K, Option[V]] </:>

The

ReadableStore
trait uses the
Future[Option[V]]
return type to communicate one of three states about each value. A value is either
  • definitely present,
  • definitely missing, or
  • unknown due to some error (perhaps a timeout, or a downed host).

The

ReadableStore
and
Store
companion objects provide a bunch of ways to create new stores. See the linked API documentation for more information.

Combinators

Coding with Storehaus's interfaces gives you access to a number of powerful combinators. The easiest way to access these combinators is by wrapping your store in an

EnrichedReadableStore
or an
EnrichedStore
. Storehaus provides implicit conversions inside of the
ReadableStore
and
Store
objects.

Here's an example of the

mapValues
combinator, useful for transforming the type of an existing store.
import com.twitter.storehaus.ReadableStore
import ReadableStore.enrich

// Create a ReadableStore from Int -> String: val store = ReadableStore.fromMap(Map[Int, String](1 -> "some value", 2 -> "other value"))

// "get" behaves as expected: store.get(1).get // res5: Option[String] = Some(some value)

// calling "mapValues" with a function from V => NewV returns a new ReadableStore[K, NewV]: val countStore: ReadableStore[Int, Int] = store.mapValues { s => s.size }

// This new store applies the function to every value on the way out: countStore.get(1).get // res6: Option[Int] = Some(10)

Storehaus-Algebra

storehaus-algebra
module adds the
MergeableStore
trait. If you're using key-value stores for aggregations, you're going to love
MergeableStore
.
package com.twitter.storehaus.algebra

trait MergeableStore[-K, V] extends Store[K, V] { def monoid: Monoid[V] def merge(kv: (K, V)): Future[Option[V]] = multiMerge(Map(kv)).apply(kv._1) def multiMerge[K1 <: k map v future kvs.map kv=""> (kv._1, merge(kv)) } } </:>

MergeableStore
's
merge
and
multiMerge
are similar to
put
and
multiPut
; the difference is that values added with
merge
are added to the store's existing value and the previous value is returned. Because the addition is handled with a
Semigroup[V]
or
Monoid[V]
from Twitter's Algebird project, it's easy to write stores that aggregate Lists, decayed values, even HyperLogLog instances.

The

MergeableStore
object provides a number of combinators on these stores. For ease of use, Storehaus provides an implicit conversion to an enrichment on

MergeableStore
. Access this by importing
MergeableStore.enrich
.

Other Modules

Storehaus provides a number of modules wrapping existing key-value stores. Enriching these key-value stores with Storehaus's combinators has been hugely helpful to us here at Twitter. Writing your jobs in terms of Storehaus stores makes it easy to test your jobs; use an in-memory

JMapStore
in testing and a
MemcacheStore
in production.

Planned Modules

Here's a list of modules we plan in implementing, with links to the github issues tracking progress on these modules:

Documentation

To learn more and find links to tutorials and information around the web, check out the Storehaus Wiki.

The latest ScalaDocs are hosted on Storehaus's Github Project Page.

Contact

Discussion occurs primarily on the Storehaus mailing list. Issues should be reported on the GitHub issue tracker.

Get Involved + Code of Conduct

Pull requests and bug reports are always welcome!

We use a lightweight form of project governence inspired by the one used by Apache projects. Please see Contributing and Committership for our code of conduct and our pull request review process. The TL;DR is send us a pull request, iterate on the feedback + discussion, and get a +1 from a Committer in order to get your PR accepted.

The current list of active committers (who can +1 a pull request) can be found here: Committers

A list of contributors to the project can be found here: Contributors

Maven

Storehaus modules are available on maven central. The current groupid and version for all modules is, respectively,

"com.twitter"
and
0.13.0
.

Current published artifacts are

  • storehaus-core_2.11
  • storehaus-core_2.10
  • storehaus-algebra_2.11
  • storehaus-algebra_2.10
  • storehaus-memcache_2.11
  • storehaus-memcache_2.10
  • storehaus-mysql_2.11
  • storehaus-mysql_2.10
  • storehaus-hbase_2.11
  • storehaus-hbase_2.10
  • storehaus-redis_2.11
  • storehaus-redis_2.10
  • storehaus-dynamodb_2.11
  • storehaus-dynamodb_2.10
  • storehaus-kafka-08_2.11
  • storehaus-kafka-08_2.10
  • storehaus-mongodb_2.11
  • storehaus-mongodb_2.10
  • storehaus-elasticsearch_2.11
  • storehaus-elasticsearch_2.10
  • storehaus-leveldb_2.11
  • storehaus-leveldb_2.10
  • storehaus-http_2.11
  • storehaus-http_2.10
  • storehaus-cache_2.11
  • storehaus-cache_2.10
  • storehaus-testing_2.11
  • storehaus-testing_2.10

The suffix denotes the scala version.

Testing notes

We use

travis-ci
to set up any underlying stores (e.g. MySQL, Redis, Memcached) for the tests. In order for these tests to pass on your local machine, you may need additional setup.

MySQL tests

You will need MySQL installed on your local machine. Once installed, run the

mysql
commands listed in .travis.yml file.

Redis tests

You will need redis installed on your local machine. Redis comes bundled with an executable for spinning up a server called

redis-server
. The Storehaus redis tests expect the factory defaults for connecting to one of these redis server instances, resolvable on
localhost
port
6379
.

Memcached

You will need Memcached installed on your local machine and running on the default port

11211
.

Authors

Contributors

Here are a few that shine among the many:

License

Copyright 2013 Twitter, Inc.

Licensed under the Apache License, Version 2.0.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.