A scalable RPC library for Erlang-VM based languages
master):
develop):
To build this project you need to have the following:
Erlang/OTP >= 19.1
git >= 1.7
GNU make >= 3.80
rebar3 >= 3.2
Getting started with
gen_rpcis easy. First, add the appropriate dependency line to your
rebar.config:
{deps, [ {gen_rpc, {git, "https://github.com/priestjim/gen_rpc.git", {branch, "master"}}} ]}.
Or if you're using
hex.pm/
rebar3:
{deps [ {gen_rpc, "~> 2.0"} ]}.
Or if you're using Elixir/Mix:
def project do [ deps: [ {:gen_rpc, "~> 2.0"} ] ]
Then, add
gen_rpcas a dependency application to your
.app.src/
.appfile:
{application, my_app, [ {applications, [kernel, stdlib, gen_rpc]} ]}
Or your
mix.exsfile:
def application do applications: [:gen_rpc] end
Finally, start a couple of nodes to test it out:
([email protected])1> gen_rpc:call('[email protected]', erlang, node, []). '[email protected]'
gen_rpcimplements only the subset of the functions of the
rpclibrary that make sense for the problem it's trying to solve. The library's function interface and return values is 100% compatible with
rpcwith only one addition: Error return values include
{badrpc, Error}for RPC-based errors but also
{badtcp, Error}for TCP-based errors.
For more information on what the functions below do, run
erl -man rpc.
call(NodeOrNodeAndKey, Module, Function, Args)and
call(NodeOrNodeAndKey, Module, Function, Args, Timeout): A blocking synchronous call, in the
gen_serverfashion.
cast(NodeOrNodeAndKey, Module, Function, Args): A non-blocking fire-and-forget call.
async_call(NodeOrNodeAndKey, Module, Function, Args),
yield(Key),
nb_yield(Key)and
nb_yield(Key, Timeout): Promise-based calls. Make a call with
async_calland retrieve the result asynchronously, when you need it with
yieldor
nb_yield.
multicall(Module, Function, Args),
multicall(Nodes, Module, Function, Args),
multicall(Module, Function, Args, Timeout)and
multicall(NodesOrNodesWithKeys, Module, Function, Args, Timeout): Multi-node version of the
callfunction.
abcast(NodesOrNodesWithKeys, Name, Msg)and
abcast(Name, Msg): An asynchronous broadcast function, sending the message
Msgto the named process
Namein all the nodes in
NodesOrNodesWithKeys.
sbcast(NodesOrNodesWithKeys, Name, Msg)and
sbcast(Name, Msg): A synchronous broadcast function, sending the message
Msgto the named process
Namein all the nodes in
NodesOrNodesWithKeys. Returns the nodes in which the named process is alive and the nodes in which it isn't.
eval_everywhere(Module, Function, Args)and
eval_everywhere(NodesOrNodesWithKeys, Module, Function, Args): Multi-node version of the
castfunction.
gen_rpcsupports multiple outgoing connections per node using a key of arbitrary type to differentiate between connections. To leverage this feature, replace
Nodein your calls (single-node and multi-node alike) with
{Node, Key}. The
Keyis hashed using
erlang:phash2/1, attached to the client process name and a new connection is initiated.
Attention: When using functions that call
gen_rpc:nodes/0implicitly (such as
gen_rpc:multicall/3), the channels used to communicate to the nodes are the keyless ones. To leverage the sharded functionality, pre-create your
{Node, Key}lists and pass them as the node list in the multi-node function.
gen_rpcsupports executing RPC calls on remote nodes that are running only specific module versions. To leverage that feature, in place of
Modulein the section above, use
{Module, Version}. If the remote
moduleis not on the version requested a
{badrpc,incompatible]will be returned.
tcp_server_port: The plain TCP port
gen_rpcwill use for incoming connections or
falseif you do not want plain TCP enabled.
tcp_client_port: The plain TCP port
gen_rpcwill use for outgoing connections.
ssl_server_port: The port
gen_rpcwill use for incoming SSL connections or
falseif you do not want SSL enabled.
ssl_client_port: The port
gen_rpcwill use for outgoing SSL connections.
ssl_server_optionsand
ssl_client_options: Settings for the
sslinterface that
gen_rpcwill use to connect to a remote
gen_rpcserver.
default_client_driver: The default driver
gen_rpcis going to use to connect to remote
gen_rpcnodes. It should be either
tcpor
ssl.
client_config_per_node: A map of
Node => {Driver, Port}or
Node => Portthat instructs
gen_rpcon the
Portand/or
Driverto use when connecting to a
Node. If you prefer to use an external discovery service to map
Nodesto
{Driver, Port}tuples, instead of the map, you'll need to define a
{Module, Function}tuple instead with a function that takes the
Nodeas its single argument, consumes the external discovery service and returns a
{Driver, Port}tuple.
rpc_module_control: Set it to
blacklistto define a list of modules that will not be exposed to
gen_rpcor to
whitelistto define the list of modules that will be exposed to
gen_rpc. Set it to
disabledto disable this feature.
rpc_module_list: The list of modules that are going to be blacklisted or whitelisted.
authentication_timeout: Default timeout for the authentication state of an incoming connection in milliseconds. Used to protect against half-open connections in a DoS attack.
connect_timeout: Default timeout for the initial node-to-node connection in milliseconds.
send_timeout: Default timeout for the transmission of a request (
call/
castetc.) from the local node to the remote node in milliseconds.
call_receive_timeout: Default timeout for the reception of a response in a
callin milliseconds.
sbcast_receive_timeout: Default timeout for the reception of a response in an
sbcastin milliseconds.
client_inactivity_timeout: Inactivity period in milliseconds after which a client connection to a node will be closed (and hence have the TCP file descriptor freed).
server_inactivity_timeout: Inactivity period in milliseconds after which a server port will be closed (and hence have the TCP file descriptor freed).
async_call_inactivity_timeout: Inactivity period in milliseconds after which a pending process holding an
async_callreturn value will exit. This is used for process sanitation purposes so please make sure to set it in a sufficiently high number (or
infinity).
socket_keepalive_idle: Seconds idle after the last packet of data sent to start sending keepalive probes (applies to both drivers).
socket_keepalive_interval: Seconds between keepalive probes.
socket_keepalive_count: Probs lost to consider the socket closed
gen_rpcuses hut for logging. This allows the developer to integrate the logging library of their choice by providing the appropriate definition in their
rebar.config. The default logging facility of
hutis SASL.
For more information on how to enable
gen_rpcto use your own logging facility, consult the README.md of
hut.
gen_rpcsupports SSL for inter-node communication. This allows secure communication and execution over insecure channels such as the internet, essentially allowing a trully globally distributed Erlang/Elixir setup.
gen_rpcis very opinionated on how SSL should be configured and the bundled default options include:
A proper PFS-enabled cipher suite
Both server and client-based SSL node CN (Common Name) verification
Secure renegotiation
TLS 1.1/1.2 enforcement
All of these settings can be found in
include/ssl.hrland overriden by redefining the necessary option in
ssl_client_optionsand
ssl_server_options. To actually use SSL support, you'll need to define in both
ssl_client_optionsand
ssl_server_options:
The public and private keys in PEM format, for the node you're running
gen_rpcon, using the usual
certfile,
keyfileoptions.
The public key of the CA that signs the node's key and the public key(s) of CA that
gen_rpcshould trust, included in the file
cacertfilepoints at.
Optionally, a Diffie-Hellman parameter file using the
dhfileoption.
To generate your own self-signed CA and node certificates, numerous articles can be found online such as this.
Usually, the CA that will be signing your client and server SSL certificates will be the same so a nominal
sys.confgthat includes SSL support for
gen_rpcwill look like:
{gen_rpc, [ {ssl_client_options, [ {certfile, "priv/cert.pem"}, {keyfile, "priv/cert.key"}, {cacertfile, "priv/ca.pem"}, {dhfile, "priv/dhparam.pem"} ]}, {ssl_server_options, [ {certfile, "priv/cert.pem"}, {keyfile, "priv/cert.key"}, {cacertfile, "priv/ca.pem"}, {dhfile, "priv/dhparam.pem"} ]} ]}
For multi-site deployments, a performant setup can be provisioned with edge
gen_rpcnodes using SSL over the internet and plain TCP for internal data exchange. In that case, non-edge nodes can have
{ssl_server_port, false}and
{default_client_driver, tcp}and edge nodes can have their plain TCP port firewalled externally and
{default_client_driver, ssl}.
gen_rpccan call an external module to provide driver/port mappings in case you want to use an external discovery service like
etcdfor node configuration management. The module should implement the
gen_rpc_external_sourcebehaviour which takes the
Nodeas an argument and should return either
{Driver, Port}(
Driverbeing
tcpor
ssland
Portbeing the port the remote node's
gen_rpc's driver is listening in) or
{error, Reason}(if the service is unavailable). To set it, change
client_config_per_nodefrom the default of
{internal, #{}}to
{external, ModuleName}where
ModuleNameis the module that implements the
gen_rpc_external_sourcebehaviour.
gen_rpcbundles a
Makefilethat makes development straightforward.
To build
gen_rpcsimply run:
make
To run the full test suite, run:
make test
To run the full test suite, the XRef tool and Dialyzer, run:
make dist
To build the project and drop in a console while developing, run:
make shell-master
or
make shell-slave
If you want to run a "master" and a "slave"
gen_rpcnodes to run tests.
To clean every build artifact and log, run:
make distclean
A full suite of tests has been implemented for
gen_rpc. You can run the CT-based test suite, dialyzer and xref by:
make dist
If you have Docker available on your system, you can run dynamic integration tests with "physically" separated hosts/nodes by running the command:
make integration
This will launch 3 slave containers and 1 master (change that by
NODES=5 make integration) and will run the
integration_SUITECT test suite.
TL;DR:
gen_rpcuses a mailbox-per-node architecture and
gen_tcpprocesses to parallelize data reception from multiple nodes without blocking the VM's distributed port.
The reasons for developing
gen_rpcbecame apparent after a lot of trial and error while trying to scale a distributed Erlang infrastructure using the
rpclibrary initially and subsequently
erlang:spawn/4(remote spawn). Both these solutions suffer from very specific issues under a sufficiently high number of requests.
The
rpclibrary operates by shipping data over the wire via Distributed Erlang's ports into a registered
gen_serveron the other side called
rex(Remote EXecution server), which is running as part of the standard distribution. In high traffic scenarios, this allows the inherent problem of running a single
gen_serverserver to manifest: mailbox flooding. As the number of nodes participating in a data exchange with the node in question increases, so do the messages that
rexhas to deal with, eventually becoming too much for the process to handle (don't forget this is confined to a single thread).
Enter
erlang:spawn/4(remote spawn from now on). Remote spawn dynamically spawns processes on a remote node, skipping the single-mailbox restriction that
rexhas. The are various libraries written to leverage that loophole (such as Rexi), however there's a catch.
Remote spawn was not designed to ship large amounts of data as part of the call's arguments. Hence, if you want to ship a large binary such as a picture or a transaction log (large can also be small if your network is slow) over remote spawn, sooner or later you'll see this message popping up in your logs if you have subscribed to the system monitor through
erlang:system_monitor/2:
{monitor,<4685.187.0>,busy_dist_port,#Port<4685.41652>}
This message essentially means that the VM's distributed port pair was busy while the VM was trying to use it for some other task like Distributed Erlang heartbeat beacons or mnesia synchronization. This of course wrecks havoc in certain timing expectations these subsystems have and the results can be very problematic: the VM might detect a node as disconnected even though everything is perfectly healthy and
mnesiamight misdetect a network partition.
gen_rpcsolves both these problems by sharding data coming from different nodes to different processes (hence different mailboxes) and by using a different
gen_tcpport for different nodes (hence not utilizing the Distributed Erlang ports).
In order to achieve the mailbox-per-node feature,
gen_rpcuses a very specific architecture:
Whenever a client needs to send data to a remote node, it will perform a
whereisto a process named after the remote node.
If the specified
clientprocess does not exist, it will request for a new one through the
dispatcherprocess, which in turn will launch it through the appropriate
clientsupervisor. Since this |
whereis> request from dispatcher sequence > start client| can happen concurrently by many different processes, serializing it behind a
gen_serverallows us to avoid race conditions.
The
dispatcherprocess will launch a new
clientprocess through the client's supervisor.
The new client process will connect to the remote node's
gen_rpc server, submit a request for a new server and wait.
The
gen_rpc serverserver will ask the
acceptorsupervisor to launch a new
acceptorprocess and hands it off the new socket connection.
The
acceptortakes over the new socket and authenticates the
clientwith the current Erlang cookie and any extra protocol-level authentication supported by the selected driver.
The
clientfinally encodes the request (
call,
castetc.) along with some metadata (the caller's PID and a reference) and sends it over the TCP channel. In case of an
async call, the
clientalso launches a process that will be responsible for handing the server's reply to the requester.
The
acceptoron the other side decodes the TCP message received and spawns a new process that will perform the requested function. By spawning a process external to the server, the
acceptorprotects itself from misbehaving function calls.
As soon as the reply from the server is ready (only needed in
async_calland
call), the
acceptorspawned process messages the server with the reply, the
acceptorships it through the TCP channel to the
clientand the client send the message back to the requester. In the case of
async call, the
clientmessages the spawned worker and the worker replies to the caller with the result.
All
gen_tcpprocesses are properly linked so that any TCP failure will cascade and close the TCP channels and any new connection will allocate a new process and port.
An inactivity timeout has been implemented inside the
clientand
serverprocesses to free unused TCP connections after some time, in case that's needed.
gen_rpcis being used in production extensively with over 150.000 incoming calls/sec/node on a 8-core Intel Xeon E5 CPU and Erlang 19.1. The median payload size is 500 KB. No stability or scalability issues have been detected in over a year.
rpcand remote spawn.
This project is published and distributed under the Apache License.
Please see CONTRIBUTING.md