Efficient RPCs for datacenter networks
eRPC is a fast and general-purpose RPC library for datacenter networks. Our NSDI 2019 paper describes the system in detail. Documentation is available online.
Some highlights: * Multiple supported networks: Ethernet, InfiniBand, and RoCE * Low latency: 2.3 microseconds round-trip RPC latency with UDP over Ethernet * Performance for small 32-byte RPCs: ~10M RPCs/sec with one CPU core, 60--80M RPCs/sec with one NIC. * Bandwidth for large RPC: 75 Gbps on one connection (one CPU core at server and client) for 8 MB RPCs * Scalability: 20000 RPC sessions per server * End-to-end congestion control that tolerates 100-way incasts * Nested RPCs, and long-running background RPCs * A port of Raft as an example. Our 3-way replication latency is 5.3 microseconds with traditional UDP over Ethernet.
scripts/packages/for required software packages for your distro. Install exactly one of the following, mutually-incompatible packages:
sudo make install T=x86_64-native-linuxapp-gcc DESTDIR=/usr). Other DPDK versions are not supported.
neRPC processes, eRPC uses kernel UDP ports
{31850, ..., 31850 + n - 1}.These ports should be open on the management network. See
scripts/firewalld/erpc_firewall.shfor systems running
firewalld.
cmake . -DPERF=OFF -DTRANSPORT=infiniband; make -j; sudo ctest.
DPERF=OFFenables debugging, which greatly reduces performance. Set
DPERF=ONfor performance measurements.
infinibandshould be replaced with
rawfor Mellanox Ethernet NICs, or
dpdkfor Intel Ethernet NICs.
scripts/run-tests-dpdk.shinstead of
ctest.
hello_worldapplication:
cd hello_world
common.h
hello_worldusing
make infiniband,
make raw, or
make dpdk.
./serverat the server, and
./clientat the client
doxygen
DTRANSPORT=raw
DTRANSPORT=dpdk
rawtransport is faster for Mellanox NICs, which also support DPDK
-DTRANSPORT=dpdk -DAZURE=on
DTRANSPORT=infiniband. Add
DROCE=onif using RoCE.
driversdirectory
eRPC works well on Azure VMs with accelerated networking.
Configure two Ubuntu 18.04 VMs as below. Use the same resource group and availability zone for both VMs.
eth0) and
lointerfaces.
az network nic create --resource-group --name --vnet-name --subnet default --accelerated-networking true --subscription
eth1, which eRPC will use for DPDK traffic.
Prepare DPDK 19.11.5:
sudo apt install make cmake g++ gcc libnuma-dev rdma-core libibverbs-dev libgflags-dev libgtest-dev numactl
(cd /usr/src/gtest && sudo cmake . && sudo make && sudo mv libg* /usr/lib/)
sudo modprobe ib_uverbs
sudo modprobe mlx4_ib
config/common_baseby changing
CONFIG_RTE_LIBRTE_MLX5_PMDand
CONFIG_RTE_LIBRTE_MLX4_PMDto
yinstead of
n.
sudo make install T=x86_64-native-linuxapp-gcc DESTDIR=/usr
sudo bash -c "echo 2048 > /sys/devices/system/node/node0/hugepages/hugepages-2048kB/nr_hugepages" sudo mkdir /mnt/huge sudo mount -t hugetlbfs nodev /mnt/huge
Build eRPC's library and latency benchmark:
cmake . -DTRANSPORT=dpdk -DAZURE=on make make latency
Create the file
scripts/autorun_process_filelike below. Here, do not use the IP addresses of the accelerated NIC (i.e., not of
eth1).
31850 0 31850 0
Run the eRPC application (the latency benchmark by default):
./scripts/do.sh 0 0
./scripts/do.sh 1 0
appsdirectory contains a suite of benchmarks and examples. The instructions below are for this suite of applications. eRPC can also be simply linked as a library instead (see
hello_world/for an example).
scripts/autorun_app_fileand change its contents to one of the available directory names in
apps/. See
scripts/example_autorun_app_filefor an example. Then generate a Makefile using
cmake . -DPERF=ON -DTRANSPORT=raw/infiniband/dpdk.
apps/contains a config file that must specify all flags defined in
apps/apps_common.h. For example,
num_processesspecifies the total number of eRPC processes in the cluster.
scripts/autorun_process_file. Each line in this file must be
.
scripts/do.shfor each process:
num_processesmachines are needed. Run
scripts/do.sh 0on machine
iin
{0, ..., num_processes - 1}.
num_machines = ceil(num_processes / 2)machines are needed. Run
scripts/do.shon machine i in
{0, ..., num_machines - 1}.
scripts/autorun_process_file, run
scripts/run-all.sh. For some applications, statistics generated in a run can be collected and processed using
scripts/proc-out.sh.
Anuj Kalia ([email protected])
Copyright 2018, Carnegie Mellon UniversityLicensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.