Marathon-lb is a service discovery & load balancing tool for DC/OS
Marathon-lb is a tool for managing HAProxy, by consuming Marathon's app state. HAProxy is a fast, efficient, battle-tested, highly available load balancer with many advanced features which power a number of high-profile websites.
Take a look at the marathon-lb wiki for example usage, templates, and more.
The marathon-lb script
marathon_lb.pyconnects to the marathon API to retrieve all running apps, generates a HAProxy config and reloads HAProxy. By default, marathon-lb binds to the service port of every application and sends incoming requests to the application instances.
Services are exposed on their service port (see Service Discovery & Load Balancing for reference) as defined in their Marathon definition. Furthermore, apps are only exposed on LBs which have the same LB tag (or group) as defined in the Marathon app's labels (using
HAPROXY_GROUP). HAProxy parameters can be tuned by specify labels in your app.
To create a virtual host or hosts the
HAPROXY_{n}_VHOSTlabel needs to be set on the given application. Applications with a vhost set will be exposed on ports 80 and 443, in addition to their service port. Multiple virtual hosts may be specified in
HAPROXY_{n}_VHOSTusing a comma as a delimiter between hostnames.
All applications are also exposed on port 9091, using the
X-Marathon-App-IdHTTP header. See the documentation for
HAPROXY_HTTP_FRONTEND_APPID_HEADin the templates section
You can access the HAProxy statistics via
:9090/haproxy?stats, and you can retrieve the current HAProxy config from the
:9090/_haproxy_getconfigendpoint.
The package is currently available from the universe. To deploy marathon-lb on the public slaves in your DC/OS cluster, simply run:
dcos package install marathon-lb
To configure a custom ssl-certificate, set the dcos cli option
ssl-certto your concatenated cert and private key in .pem format. For more details see the HAProxy documentation.
For further customization, templates can be added by pointing the dcos cli option
template-urlto a tarball containing a directory
templates/. See comments in script on how to name those.
Synopsis:
docker run -e PORTS=$portnumber --net=host mesosphere/marathon-lb sse|poll ...
You must set
PORTSenvironment variable to allow haproxy bind to this port. Syntax:
docker run -e PORTS=9090 mesosphere/marathon-lb sse [other args]
You can pass in your own certificates for the SSL frontend by setting the
HAPROXY_SSL_CERTenvironment variable. If you need more than one certificate you can specify additional ones by setting
HAPROXY_SSL_CERT0-
HAPROXY_SSL_CERT100.
ssemode
In SSE mode, the script connects to the marathon events endpoint to get notified about state changes. This only works with Marathon 0.11.0 or newer versions.
Syntax:
docker run mesosphere/marathon-lb sse [other args]
pollmode
If you can't use the HTTP callbacks, the script can poll the APIs to get the schedulers state periodically.
Syntax:
docker run mesosphere/marathon-lb poll [other args]
To change the poll interval (defaults to 60s), you can set the
POLL_INTERVALenvironment variable.
You can also run the update script directly. To generate an HAProxy configuration from Marathon running at
localhost:8080with the
marathon_lb.pyscript, run:
$ ./marathon_lb.py --marathon http://localhost:8080 --group external --strict-mode --health-check
It is possible to pass
--auth-credentials=option if your Marathon requires authentication:
console $ ./marathon_lb.py --marathon http://localhost:8080 --auth-credentials=admin:password
It is possible to get the auth credentials (user & password) from VAULT if you define the following environment variables before running marathon-lb: VAULTTOKEN, VAULTHOST, VAULTPORT, VAULTPATH where VAULT_PATH is the root path where your user and password are located.
This will refresh
haproxy.cfg, and if there were any changes, then it will automatically reload HAProxy. Only apps with the label
HAPROXY_GROUP=externalwill be exposed on this LB.
marathon_lb.pyhas a lot of additional functionality like sticky sessions, HTTP to HTTPS redirection, SSL offloading, virtual host support and templating capabilities.
To get the full documentation run:
console $ ./marathon_lb.py --help
You can provide your SSL certificate paths to be placed in frontend marathonhttpsin section with
--ssl-certs.
$ ./marathon_lb.py --marathon http://localhost:8080 --group external --ssl-certs /etc/ssl/site1.co,/etc/ssl/site2.co --health-check --strict-mode
If you are using the script directly, you have two options:
/etc/ssl/cert.pemas the certificate path. Put the certificate in this path or edit the file for the correct path.
--ssl-certscommand line argument and config will use these paths.
If you are using the provided
runscript or Docker image, you have three options:
HAPROXY_SSL_CERTenvironment variable. Contents will be written to
/etc/ssl/cert.pem. Config will use this path unless you specified extra certificate paths as in the next option.
--ssl-certscommand line argument. Your config will use these certificate paths.
/etc/ssl/cert.pemand config will use it.
You can skip the configuration file validation (via calling HAProxy service) process if you don't have HAProxy installed. This is especially useful if you are running HAProxy on Docker containers.
$ ./marathon_lb.py --marathon http://localhost:8080 --group external --skip-validation
You can use HAProxy maps to speed up web application (vhosts) to backend lookup. This is very useful for large installations where the traditional vhost to backend rules comparison takes considerable time since it sequentially compares each rule. HAProxy map creates a hash based lookup table so its fast compared to the other approach, this is supported in marathon-lb using
--haproxy-mapflag.
$ ./marathon_lb.py --marathon http://localhost:8080 --group external --haproxy-map
Currently it creates a lookup dictionary only for host header (both HTTP and HTTPS) and X-Marathon-App-Id header. But for path based routing and auth, it uses the usual backend rules comparison.
Marathon-lb exposes a few endpoints on port 9090 (by default). They are:
| Endpoint | Description | |-------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| |
:9090/haproxy?stats| HAProxy stats endpoint. This produces an HTML page which can be viewed in your browser, providing various statistics about the current HAProxy instance. | |
:9090/haproxy?stats;csv| This is a CSV version of the stats above, which can be consumed by other tools. For example, it's used in the
zdd.pyscript. | |
:9090/_haproxy_health_check| HAProxy health check endpoint. Returns
200 OKif HAProxy is healthy. | |
:9090/_haproxy_getconfig| Returns the HAProxy config file as it was when HAProxy was started. Implemented in
getconfig.lua. | |
:9090/_haproxy_getvhostmap| Returns the HAProxy vhost to backend map. This endpoint returns HAProxy map file only when the
--haproxy-mapflag is enabled, it returns an empty string otherwise. Implemented in
getmaps.lua. | |
:9090/_haproxy_getappmap| Returns the HAProxy app ID to backend map. Like
_haproxy_getvhostmap, this requires the
--haproxy-mapflag to be enabled and returns an empty string otherwise. Also implemented in
getmaps.lua. | |
:9090/_haproxy_getpids| Returns the PIDs for all HAProxy instances within the current process namespace. This literally returns
$(pidof haproxy). Implemented in
getpids.lua. This is also used by the
zdd.pyscript to determine if connections have finished draining during a deploy. | |
:9090/_mlb_signal/hup* | Sends a
SIGHUPsignal to the marathon-lb process, causing it to fetch the running apps from Marathon and reload the HAProxy config as though an event was received from Marathon. | |
:9090/_mlb_signal/usr1* | Sends a
SIGUSR1signal to the marathon-lb process, causing it to restart HAProxy with the existing config, without checking Marathon for changes. | |
:9090/metrics| Exposes HAProxy metrics in prometheus format. |
* These endpoints won't function when marathon-lb is in
pollmode as there is no marathon-lb process to be signaled in this mode (marathon-lb exits after each poll).
App labels are specified in the Marathon app definition. These can be used to override HAProxy behaviour. For example, to specify the
externalgroup for an app with a virtual host named
service.mesosphere.com:
{ "id": "http-service", "labels": { "HAPROXY_GROUP":"external", "HAPROXY_0_VHOST":"service.mesosphere.com" } }
Some labels are specified per service port. These are denoted with the
{n}parameter in the label key, where
{n}corresponds to the service port index, beginning at
0.
See the configuration doc for the full list of labels.
Marathon-lb global templates (as listed in the Longhelp) can be overwritten in two ways: -By creating an environment variable in the marathon-lb container -By placing configuration files in the
templates/directory (relative to where the script is run from)
For example, to replace
HAPROXY_HTTPS_FRONTEND_HEADwith this content:
frontend new_frontend_label bind *:443 ssl crt /etc/ssl/cert.pem mode http
Then this environment variable could be added to the Marathon-LB configuration:
"HAPROXY_HTTPS_FRONTEND_HEAD": "\\nfrontend new_frontend_label\\n bind *:443 ssl {sslCerts}\\n mode http"
Alternately, a file called
HAPROXY_HTTPS_FRONTEND_HEADcould be placed in
templates/directory through the use of an artifact URI.
Additionally, some templates can also be overridden per app service port. You may add your own templates to the Docker image, or provide them at startup.
See the configuration doc for the full list of templates.
Some templates may be overridden using app labels, as per the labels section. Strings are interpreted as literal HAProxy configuration parameters, with substitutions respected (as per the templates section). The HAProxy configuration will be validated for correctness before reloading HAProxy after changes. Note: Since the HAProxy config is checked before reloading, if an app's HAProxy labels aren't syntactically correct, HAProxy will not be reloaded and may result in stale config.
Here is an example for a service called
http-servicewhich requires that
http-keep-alivebe disabled:
{ "id": "http-service", "labels":{ "HAPROXY_GROUP":"external", "HAPROXY_0_BACKEND_HTTP_OPTIONS":" option forwardfor\n no option http-keep-alive\n http-request set-header X-Forwarded-Port %[dst_port]\n http-request add-header X-Forwarded-Proto https if { ssl_fc }\n" } }
The full list of per service port templates which can be specified are documented here.
As a shortcut to add haproxy global default options (without overriding the global template) a comma-separated list of options may be specified via the
HAPROXY_GLOBAL_DEFAULT_OPTIONSenvironment variable. The default value when not specified is
redispatch,http-server-close,dontlognull; as an example, to add the
httplogoption (and keep the existing defaults), one should specify
HAPROXY_GLOBAL_DEFAULT_OPTIONS=redispatch,http-server-close,dontlognull,httplog. - Note that this setting has no effect when the `HAPROXYHEAD` template has been overridden._
HAPROXY_{n}_PORTlabel; prefer defining service ports.
--group) for internal and external load balancing. On DC/OS, the default group is
external. A simple
options.jsonfor an internal load balancer would be:
{ "marathon-lb": { "name": "marathon-lb-internal", "haproxy-group": "internal", "bind-http-https": false, "role": "" } }
X-Marathon-App-Idheader. For example, to access an app with the ID
tweeter:
$ curl -vH "X-Marathon-App-Id: /tweeter" marathon-lb.marathon.mesos:9091/ * Trying 10.0.4.74... * Connected to marathon-lb.marathon.mesos (10.0.4.74) port 9091 (#0) > GET / HTTP/1.1 > Host: marathon-lb.marathon.mesos:9091 > User-Agent: curl/7.48.0 > Accept: */* > X-Marathon-App-Id: /tweeter > < HTTP/1.1 200 OK
/_mlb_signalendpoints and the
/_haproxy_getpidsendpoint (and by extension, zero-downtime deployments) may behave unexpectedly if more than one instance of marathon-lb is running in the same PID namespace or if there are other HAProxy processes in the same PID namespace.
HAPROXY_SYSLOGDenvironment variable or
container-syslogdvalue in
options.jsonlike so:
{ "marathon-lb": { "container-syslogd": true } }
zdd.pyis not to be used in a production environment and is purely developed for demonstration purposes.
Marathon-lb is able to perform canary style blue/green deployment with zero downtime. To execute such deployments, you must follow certain patterns when using Marathon.
The deployment method is described in this Marathon document. Marathon-lb provides an implementation of the aforementioned deployment method with the script
zdd.py. To perform a zero downtime deploy using
zdd.py, you must:
HAPROXY_DEPLOYMENT_GROUPand
HAPROXY_DEPLOYMENT_ALT_PORTlabels in your app template
HAPROXY_DEPLOYMENT_GROUP: This label uniquely identifies a pair of apps belonging to a blue/green deployment, and will be used as the app name in the HAProxy configuration
HAPROXY_DEPLOYMENT_ALT_PORT: An alternate service port is required because Marathon requires service ports to be unique across all apps
zdd.pyscript to orchestrate the deploy: the script will make API calls to Marathon, and use the HAProxy stats endpoint to gracefully terminate instances
iptablescommands) due to the issues outlined in the excellent blog post by the Yelp engineering team found here
An example minimal configuration for a test instance of nginx is included here. You might execute a deployment from a CI tool like Jenkins with:
./zdd.py -j 1-nginx.json -m http://master.mesos:8080 -f -l http://marathon-lb.marathon.mesos:9090 --syslog-socket /dev/null
Zero downtime deployments are accomplished through the use of a Lua module, which reports the number of HAProxy processes which are currently running by hitting the stats endpoint at the
/_haproxy_getpids. After a restart, there will be multiple HAProxy PIDs until all remaining connections have gracefully terminated. By waiting for all connections to complete, you may safely and deterministically drain tasks. A caveat of this, however, is that if you have any long-lived connections on the same LB, HAProxy will continue to run and serve those connections until they complete, thereby breaking this technique.
The ZDD script includes the ability to specify a pre-kill hook, which is executed before draining tasks are terminated. This allows you to run your own automated checks against the old and new app before the deploy continues.
Zdd has support to split the traffic between two versions of same app (version 'blue' and version 'green') by having instances of both versions live at the same time. This is supported with the help of the
HAPROXY_DEPLOYMENT_NEW_INSTANCESlabel.
When you run zdd with the
--new-instancesflag, it creates only the specified number of instances of the new app, and deletes the same number of instances from the old app (instead of the normal, create all instances in new and delete all from old approach), to ensure that the number of instances in new app and old app together is equal to
HAPROXY_DEPLOYMENT_TARGET_INSTANCES.
Example: Consider the same nginx app example where there are 10 instances of nginx running image version v1, now we can use zdd to create 2 instances of version v2, and retain 8 instances of V1 so that traffic is split in ratio 80:20 (old:new).
Creating 2 instances with new version automatically deletes 2 instances in existing version. You could do this using the following command:
$ ./zdd.py -j 1-nginx.json -m http://master.mesos:8080 -f -l http://marathon-lb.marathon.mesos:9090 --syslog-socket /dev/null --new-instances 2
This state where you have instances of both old and new versions of same app live at the same time is called hybrid state.
When a deployment group is in hybrid state, it needs to be converted into completely current version or completely previous version before deploying any further versions, this could be done with the help of the
--complete-curand
--complete-prevflags in zdd.
When you run the below command, it converts all instances to new version so that traffic split ratio becomes 0:100 (old:new) and it deletes the old app. This is graceful as it follows usual zdd procedure of waiting for tasks/instances to drain before deleting them.
$ ./zdd.py -j 1-nginx.json -m http://master.mesos:8080 -f -l http://marathon-lb.marathon.mesos:9090 --syslog-socket /dev/null --complete-cur
Similarly you can use
--complete-prevflag to convert all instances to old version (and this is essentially a rollback) so that traffic split ratio becomes 100:0 (old:new) and it deletes the new app.
Currently only one hop of traffic split is supported, so you can specify the number of new instances (directly proportional to traffic split ratio) only when app is having all instances of same version (completely blue or completely green). This implies
--new-instancesflag cannot be specified in hybrid mode to change traffic split ratio (instance ratio) as updating Marathon label (
HAPROXY_DEPLOYMENT_NEW_INSTANCES) currently triggers new deployment in marathon which will not be graceful. Currently for the example mentioned, the traffic split ratio is 100:0 -> 80:20 -> 0:100, where there is only one hop when both versions get traffic simultaneously.
Marathon-lb supports load balancing for applications that use the Mesos IP-per-task feature, whereby each task is assigned unique, accessible, IP addresses. For these tasks services are directly accessible via the configured discovery ports and there is no host port mapping. Note, that due to limitations with Marathon (see mesosphere/marathon#3636) configured service ports are not exposed to marathon-lb for IP-per-task apps.
For these apps, if the service ports are missing from the Marathon app data, marathon-lb will automatically assign port values from a configurable range if you specify it. The range is configured using the
--min-serv-port-ip-per-taskand
--max-serv-port-ip-per-taskoptions. While port assignment is deterministic, the assignment is not guaranteed if you change the current set of deployed apps. In other words, when you deploy a new app, the port assignments may change.
When running with isolated containers, you may need to take care of reaping orphaned child processes. HAProxy typically produces orphan processes because of its two-step reload mechanism. Marathon-LB uses tini for this purpose. When running in a container without PID namespace isolation, setting the
TINI_SUBREAPERenvironment variable is recommended.
PRs are welcome, but here are a few general guidelines:
pip install -r requirements-dev.txt nosetests
bash /path/to/marathon-lb/scripts/install-git-hooks.sh
Running unit and integration tests is automated as
maketargets. Docker is required to use the targets as it will run all tests in containers.
Several environment variables can be set to control the image tags, DCOS version/variant, etc. Check the top of the
Makefilefor more info.
To run the unit tests:
make test-unit
To run the integration tests a DCOS installation will be started via dcos-e2e. The installation of
dcos-e2eand management of the cluster will all be done in docker containers. Since the installers are rather large downloads, it is benificial to specify a value for
DCOS_E2E_INSTALLERS_DIR. By default
DCOS_E2E_INSTALLERS_DIRis inside the
.cachedirectory that will be removed upon
make clean. You must provide a repository for the resultant docker image to be pushed to via the
CONTAINTER_REPOenvironemnt variable. It is assumed that the local docker is already logged in and the image will be pushed prior to launching the cluster.
To run the integration tests on the OSS variant of DCOS:
DCOS_E2E_INSTALLERS_DIR="${HOME}/dcos/installers" \ CONTAINTER_REPO="my_docker_user/my-marathon-lb-repo" make test-integration
To run the integration tests on the ENTERPRISE variant of DCOS:
DCOS_LICENSE_KEY_PATH=${HOME}/license.txt \ DCOS_E2E_VARIANT=enterprise \ DCOS_E2E_INSTALLERS_DIR="${HOME}/dcos/installers"\ CONTAINTER_REPO="my_docker_user/my-marathon-lb-repo" make test-integration
To run both unit and integration tests (add appropriate variables):
CONTAINTER_REPO="my_docker_user/my-marathon-lb-repo" make test
You need to install the curl development package.
# Fedora dnf install libcurl-develUbuntu
apt-get install libcurl-dev
The
pycurlpackage linked against the wrong SSL backend when you installed it.
pip uninstall pycurl export PYCURL_SSL_LIBRARY=nss pip install -r requirements-dev.txt
Swap
nssfor whatever backend it mentions.
Create a Github release. Follow the convention of past releases. You can find something to copy/paste if you hit the "edit" button of a previous release.
The Github release creates a tag, and Dockerhub will build off of that tag.
Make a PR to Universe. The suggested way is to create one commit that only copies the previous dir to a new one, and then a second commit that makes the actual changes. If unsure, check out the previous commits to the marathon-lb directory in Universe.