metafora

by lytics

lytics / metafora

Distributed long running work system in Go

128 Stars 16 Forks Last release: Not found Apache License 2.0 466 Commits 1 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

metafora

Join the chat at https://gitter.im/lytics/metafora Build Status GoDoc

Metafora is a Go library designed to run long-running (minutes to permanent) tasks in a cluster.

IRC:

#lytics/metafora
on irc.gitter.im

Features

  • Distributed - horizontally scalable
  • Elastic - online cluster resizing with automated rebalancing
  • Masterless - work stealing, not assigning, pluggable balancing
  • Fault tolerant - tasks are reassigned if nodes disappear
  • Simple - few states, no checkpointing, no configuration management
  • Extensible - well defined interfaces for implementing balancing and coordinating
  • Exactly-once - designed to enforce one-and-only-one instance of each submitted task is runningref

Metafora is a library for building distributed task work systems. You're responsible for creating a

main()
entrypoint for your application, writing a
metafora.Handler
and
HandlerFunc
to actually process tasks, and then starting Metafora's
Consumer
.

Metafora's task state machine is implemented as a

Handler
adapter. Simply implement your task processor as a
StatefulHandler
function, and create a
metafora.Handler
with
statemachine.New
.

Example

koalemosd is a sample consumer implementation that can be run as a daemon (it requires etcd). koalemosctl is a sample command line client for submitting tasks to

koalemosd
.
# Install etcd as per https://go.etcd.io/etcd#getting-etcd
# Run the following in one terminal:
go get -v -u github.com/lytics/metafora/examples/koalemosd
koalemosd

Run the client in another

go get -v -u github.com/lytics/metafora/examples/koalemosctl koalemosctl sleep 3 # where "sleep 3" is any command on your $PATH

Since koalemosd is a simple wrapper around OS processes, it does not use the state machine (

statemachine.StatefulHandler
).

Terms

Balancer Go interface consulted by Consumer for determining which tasks can be claimed and which should be released. See balancer.go.
Broker external task and command store like etcd for the Coordinator to use.
Consumer core work runner. Integrates Balancer, Coordinator, and Handlers to get work done.
Coordinator client Go interface to Broker. See coordinator.go.
Handler Go interface for executing tasks.
Task unit of work. Executed by Handlers.

FAQ

Q. Is it ready for production use?

Yes. Metafora with the etcd coordinator has been the production work system at Lytics since January 2014 and runs thousands of tasks concurrently across a cluster of VMs.

Since Metafora is still under heavy development, you probably want to pin the dependencies to a commit hash or tag to keep the API stable. The

master
branch is automatically tested and is safe for use if you can tolerate API changes.

Q. Where is the metaforad daemon?

It doesn't exist. Metafora is library for you to import and use in a service you write. Metafora handles task management but leaves implementation details such as task implementation and daemonization up to the user.

FAQ continued in Documentation...

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.