exprotobuf

by bitwalker

bitwalker / exprotobuf

Protocol Buffers in Elixir made easy!

457 Stars 67 Forks Last release: over 5 years ago (0.10.0) Apache License 2.0 182 Commits 40 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

Protocol Buffers for Elixir

exprotobuf works by building module/struct definitions from a Google Protocol Buffer schema. This allows you to work with protocol buffers natively in Elixir, with easy decoding/encoding for transport across the wire.

Build Status Hex.pm Version

Features

  • Load protobuf from file or string
  • Respects the namespace of messages
  • Allows you to specify which modules should be loaded in the definition of records
  • Currently uses gpb for protobuf schema parsing

TODO:

  • Clean up code/tests

Breaking Changes

The 1.0 release removed the feature of handling

import "...";
statements. Please see the imports upgrade guide for details if you were using this feature.

Getting Started

Add exprotobuf as a dependency to your project:

defp deps do
  [{:exprotobuf, "~> x.x.x"}]
end

Then run

mix deps.get
to fetch.

Add exprotobuf to applications list:

def application do
  [applications: [:exprotobuf]]
end

Usage

Usage of exprotobuf boils down to a single

use
statement within one or more modules in your project.

Let's start with the most basic of usages:

Define from a string

defmodule Messages do
  use Protobuf, """
    message Msg {
      message SubMsg {
        required uint32 value = 1;
      }

  enum Version {
    V1 = 1;
    V2 = 2;
  }

  required Version version = 2;
  optional SubMsg sub = 1;
}

""" end

iex> msg = Messages.Msg.new(version: :'V2')
%Messages.Msg{version: :V2, sub: nil}
iex> encoded = Messages.Msg.encode(msg)
<<16, 2>>
iex> Messages.Msg.decode(encoded)
%Messages.Msg{version: :V2, sub: nil}

The above code takes the provided protobuf schema as a string, and generates modules/structs for the types it defines. In this case, there would be a Msg module, containing a SubMsg and Version module. The properties defined for those values are keys in the struct belonging to each. Enums do not generate structs, but a specialized module with two functions:

atom(x)
and
value(x)
. These will get either the name of the enum value, or it's associated value.

Values defined in the schema using the

oneof
construct are represented with tuples:
defmodule Messages do
  use Protobuf, """
    message Msg {
      oneof choice {
        string first = 1;
        int32 second = 2;
      }
    }
  """
end
iex> msg = Messages.Msg.new(choice: {:second, 42})
%Messages.Msg{choice: {:second, 42}}
iex> encoded = Messages.Msg.encode(msg)
<<16, 42>>

Define from a file

defmodule Messages do
  use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__)
end

This is equivalent to the above, if you assume that

messages.proto
contains the same schema as in the string of the first example.

Loading all definitions from a set of files

defmodule Protobufs do
  use Protobuf, from: Path.wildcard(Path.expand("../definitions/**/*.proto", __DIR__))
end
iex> Protobufs.Msg.new(v: :V1)
%Protobufs.Msg{v: :V1}
iex> %Protobufs.OtherMessage{middle_name: "Danger"}
%Protobufs.OtherMessage{middle_name: "Danger"}

This will load all the various definitions in your

.proto
files and allow them to share definitions like enums or messages between them.

Customizing Generated Module Names

In some cases your library of protobuf definitions might already contain some namespaces that you would like to keep. In this case you will probably want to pass the

use_package_names: true
option. Let's say you had a file called
protobufs/example.proto
that contained:
package world;
message Example {
  enum Continent {
    ANTARCTICA = 0;
    EUROPE = 1;
  }

optional Continent continent = 1; optional uint32 id = 2; }

You could load that file (and everything else in the protobufs directory) by doing:

defmodule Definitions do
  use Protobuf, from: Path.wildcard("protobufs/*.proto"), use_package_names: true
end
iex> Definitions.World.Example.new(continent: :EUROPE)
%Definitions.World.Example{continent: :EUROPE}

You might also want to define all of these modules in the top-level namespace. You can do this by passing an explicit

namespace: :"Elixir"
option.
defmodule Definitions do
  use Protobuf, from: Path.wildcard("protobufs/*.proto"),
                use_package_names: true,
                namespace: :"Elixir"
end
iex> World.Example.new(continent: :EUROPE)
%World.Example{continent: :EUROPE}

Now you can use just the package names and message names that your team is already familiar with.

Inject a definition into an existing module

This is useful when you only have a single type, or if you want to pull the module definition into the current module instead of generating a new one.

defmodule Msg do
  use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__), inject: true

def update(msg, key, value), do: Map.put(msg, key, value) end

iex> %Msg{}
%Msg{v: :V1}
iex> Msg.update(%Msg{}, :v, :V2)
%Msg{v: :V2}

As you can see, Msg is no longer created as a nested module, but is injected right at the top level. I find this approach to be a lot cleaner than

use_in
, but may not work in all use cases.

Inject a specific type from a larger subset of types

When you have a large schema, but perhaps only care about a small subset of those types, you can use

:only
:
defmodule Messages do
  use Protobuf, from: Path.expand("../proto/messages.proto", __DIR__),
only: [:TypeA, :TypeB]
end

Assuming that the provided .proto file contains multiple type definitions, the above code would extract only TypeA and TypeB as nested modules. Keep in mind your dependencies, if you select a child type which depends on a parent, or another top-level type, exprotobuf may fail, or your code may fail at runtime.

You may only combine

:only
with
:inject
when
:only
is a single type, or a list containing a single type. This is due to the restriction of one struct per module. Theoretically you should be able to pass
:only
with multiple types, as long all but one of the types is an enum, since enums are just generated as modules, this does not currently work though.

Extend generated modules via
use_in

If you need to add behavior to one of the generated modules,

use_in
will help you. The tricky part is that the struct for the module you
use_in
will not be defined yet, so you can't rely on it in your functions. You can still work with the structs via the normal Maps API, but you lose compile-time guarantees. I would recommend favoring
:inject
over this when possible, as it's a much cleaner solution.
defmodule Messages do
  use Protobuf, "
    message Msg {
      enum Version {
        V1 = 1;
        V2 = 1;
      }
      required Version v = 1;
    }
  "

defmodule MsgHelpers do defmacro using(_opts) do quote do def convert_to_record(msg) do msg |> Map.to_list |> Enum.reduce([], fn {_key, value}, acc -> [value | acc] end) |> Enum.reverse |> list_to_tuple end end end end

use_in "Msg", MsgHelpers end

iex> Messages.Msg.new |> Messages.Msg.convert_to_record
{Messages.Msg, :V1}

Attribution/License

exprotobuf is a fork of the azukiaapp/elixir-protobuf project, both of which are released under Apache 2 License.

Check LICENSE files for more information.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.