dnpipes

by mhausenblas

mhausenblas / dnpipes

Distributed Named Pipes

451 Stars 10 Forks Last release: almost 4 years ago (0.1) Apache License 2.0 19 Commits 1 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

dnpipes

Distributed Named Pipes (or:

dnpipes
) are essentially a distributed version of Unix named pipes comparable to, for example, SQS in AWS or the Service Bus in Azure.

dnpipes concept

Conceptually, we're dealing with a bunch of distributed processes (

dpN
above). These distributed processes may be long-running (such as
dp0
or
dp5
) or batch-oriented ones, for example
dp3
or
dp6
. There are a number of situations where you want these distributed processes to communicate, very similar to what IPC enables you to do on a single machine. Now,
dnpipes
are a simple mechanism to facilitate IPC between distributed processes. What follows is an interface specification as well as a reference implementation for
dnpipes
.

Interface specification

Interpret the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", "MAY NOT", and "OPTIONAL" in the context of this repo as defined in RFC 2119.

A

dnpipe
is a distributed ordered queue (FIFO) of messages available to a number of participating distributed processes. A distributed process is a useful abstraction provided by systems such as DC/OS (for example a Marathon app or a Metronome job) or Kubernetes (ReplicaSet or a Job) that give a user the illusion that a service or application she is executing on a bunch of commodity machines (the cluster) behaves like one global entity while it really is a collection of locally executed processes. In DC/OS this locally executed process would be a Mesos task and in Kubernetes a pod.

A

dnpipes
implementation MUST support the following operations:
  • push(TOPIC, MSG)
    … executed by a publisher, this writes the message
    MSG
    to a
    dnpipe
    called
    TOPIC
    .
  • MSG  … executed by a subscriber, this reads a message from a 
    dnpipe
    called
    TOPIC
    .
  • reset(TOPIC)
    … executed by either a publisher or consumer, this removes all messages from a
    dnpipe
    called
    TOPIC
    .

The following MUST be true at any point in time:

  1. After
    push
    is executed by the publisher
    MSG
    MUST be available for
    pull
    to any participant until
    reset
    is triggered and has completed.
  2. A
    pull
    does not remove a message from a
    dnpipe
    , it merely delivers its content to the consumer.
  3. The way how participants discover a
    dnpipe
    is outside of the scope of this specification.

Note concerning the last point: since there are many ways to implement service discovery in a distributed system we do not expect that an agreement can be found here hence we leave it up to the implementation how to go about it. The next sections shows an example using Kafka and DNS to achieve this.

Use cases

A

dnpipe
can be useful in a number of situations including but not limited to the following:

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.