Dropwizard + Hystrix Module.
Tenacity is Dropwizard+Hystrix.
Dropwizard is a framework for building REST services. Hystrix is a resiliency library from Netflix and Ben Christensen.
Hystrix's goals are to:
Tenacity makes Hystrix dropwizard-friendly and for dropwizard-developers to quickly leverage the benefits of Hystrix.
TenacityCommand).
Tenacity is meant to be used with Breakerbox which adds real-time visualization of metrics and dynamic configuration.
tenacity-core: The building blocks to quickly use Hystrix within the context of Dropwizard.
tenacity-client: Client for consuming the resources that
tenacity-coreadds.
tenacity-testing:
TenacityTestRuleallows for easier unit testing. Resets internal state of Hystrix.
tenacity-jdbi: Pulls in dropwizard-jdbi and provides a DBIExceptionLogger and SQLExceptionLogger to be used with the ExceptionLoggingCommandHook.
Here is a sample
TenacityCommandthat always succeeds:
public class AlwaysSucceed extends TenacityCommand { public AlwaysSucceed() { super(DependencyKey.ALWAYS_SUCCEED); }@Override protected String run() throws Exception { return "value"; }
}
A quick primer on the way to use a
TenacityCommandif you are not familiar with Hystrix's execution model:
AlwaysSucceed command = new AlwaysSucceed(); String result = command.execute();
This executes the command synchronously but through the protection of a
Future.get(configurableTimeout).
Future futureResult = command.queue();
Observable observable = new AlwaysSucceed().observe();
When execution fails, it is possible to gracefully degrade with the use of fallbacks.
Earlier we saw:
public class AlwaysSucceed extends TenacityCommand { public AlwaysSucceed() { super(DependencyKey.ALWAYS_SUCCEED); } ... }
The arguments are:
commandKey: This creates a circuit-breaker, threadpool, and also the identifier that will be used in dashboards. This should be your implementation of the
TenacityPropertyKeyinterface.
It is possible to create multiple circuit-breakers that leverage a single threadpool, but for simplicity we are not allowing that type of configuration.
To leverage within dropwizard first add the following to your
pom.xml:
com.yammer.tenacity tenacity-core 1.1.2
Or you can leverage the tenacity-bom:
com.yammer.tenacity tenacity-bom 1.1.2 pom import com.yammer.tenacity tenacity-core
Enumerate your dependencies. These will eventually be used as global identifiers in dashboards. We have found that it works best for us when you include the service and the external dependency at a minimum. Here is an example of
completie's dependencies. Note we also shave down some characters to save on space, again for UI purposes. In addition, you'll need to have an implementation of a
TenacityPropertyKeyFactorywhich you can see an example of below.
public enum CompletieDependencyKeys implements TenacityPropertyKey { CMPLT_PRNK_USER, CMPLT_PRNK_GROUP, CMPLT_PRNK_SCND_ORDER, CMPLT_PRNK_NETWORK, CMPLT_TOKIE_AUTH, CMPLT_TYRANT_AUTH, CMPLT_WHVLL_PRESENCE }
public class CompletieDependencyKeyFactory implements TenacityPropertyKeyFactory { @Override public TenacityPropertyKey from(String value) { return CompletieDependencyKeys.valueOf(value.toUpperCase()); } }
Create a
TenacityBundleConfigurationFactoryimplementation - you can use the
BaseTenacityBundleConfigurationFactoryas your starting point. This will be used to register your custom tenacity dependencies and custom configurations.
public class CompletieTenacityBundleConfigurationFactory extends BaseTenacityBundleConfigurationFactory {@Override public Map getTenacityConfigurations(CompletieConfiguration configuration) { final ImmutableMap.Builder builder = ImmutableMap.builder();
builder.put(CompletieDependencyKeys.CMPLT_PRNK_USER, configuration.getRanking().getHystrixUserConfig()); builder.put(CompletieDependencyKeys.CMPLT_PRNK_GROUP, configuration.getRanking().getHystrixGroupConfig()); builder.put(CompletieDependencyKeys.CMPLT_PRNK_SCND_ORDER, configuration.getRanking().getHystrixSecondOrderConfig()); builder.put(CompletieDependencyKeys.CMPLT_PRNK_NETWORK, configuration.getRanking().getHystrixNetworkConfig()); builder.put(CompletieDependencyKeys.CMPLT_TOKIE_AUTH, configuration.getAuthentication().getHystrixConfig()); builder.put(CompletieDependencyKeys.CMPLT_WHVLL_PRESENCE, configuration.getPresence().getHystrixConfig()) return builder.build();
} }
Then make sure you add the bundle in your
Application.
Maptype.
@Override public void initialize(Bootstrap bootstrap) { ... bootstrap.addBundle(TenacityBundleBuilder . newBuilder() .configurationFactory(new CompletieTenacityBundleConfigurationFactory()) .build()); ... }
Use
TenacityCommandto select which custom tenacity configuration you want to use.
public class CompletieDependencyOnTokie extends TenacityCommand { public CompletieDependencyOnTokie() { super(CompletieDependencyKeys.CMPLT_TOKIE_AUTH); } ... }
When testing use the
tenacity-testingmodule. This registers appropriate custom publishers/strategies, clears global
Archaiusconfiguration state (Hystrix uses internally to manage configuration), and tweaks threads that calculate metrics which influence circuit breakers to update a more frequent interval. Simply use the
TenacityTestRule.
@Rule public final TenacityTestRule tenacityTestRule = new TenacityTestRule();
com.yammer.tenacity tenacity-testing 1.1.2 test
Last is to actually configure your dependencies once they are wrapped with
TenacityCommand.
Once you have identified your dependencies you need to configure them appropriately. Here is the basic structure of a single
TenacityConfigurationthat may be leverage multiple times through your service configuration:
executionIsolationThreadTimeoutInMillis: 1000 executionIsolationStrategy: THREAD threadpool: threadPoolCoreSize: 10 keepAliveTimeMinutes: 1 maxQueueSize: -1 queueSizeRejectionThreshold: 5 metricsRollingStatisticalWindowInMilliseconds: 10000 metricsRollingStatisticalWindowBuckets: 10 circuitBreaker: requestVolumeThreshold: 20 errorThresholdPercentage: 50 sleepWindowInMillis: 5000 metricsRollingStatisticalWindowInMilliseconds: 10000 metricsRollingStatisticalWindowBuckets: 10 semaphore: maxConcurrentRequests: 10 fallbackMaxConcurrentRequests: 10
The following two are the most important and you can probably get by just fine by defining just these two and leveraging the defaults.
executionIsolationStrategy: Which to use THREAD or SEMAPHORE. Defaults to THREAD. Execution Isolation Strategy.
executionIsolationThreadTimeoutInMillis: How long the entire dependency command should take.
threadPoolCoreSize: Self explanatory.
Here are the rest of the descriptions:
keepAliveTimeMinutes: Thread keepAlive time in the thread pool.
maxQueueSize: -1 uses a
SynchronousQueue. Anything >0 leverages a
BlockingQueueand enables the
queueSizeRejectionThresholdvariable.
queueSizeRejectionThreshold: Disabled when using -1 for
maxQueueSizeotherwise self explanatory.
requestVolumeThreshold: The minimum number of requests that need to be received within the
metricsRollingStatisticalWindowInMillisecondsin order to open a circuit breaker.
errorThresholdPercentage: The percentage of errors needed to trip a circuit breaker. In order for this to take effect the
requestVolumeThresholdmust first be satisfied.
sleepWindowInMillis: How long to keep the circuit breaker open, before trying again.
Here are the semaphore related items:
maxConcurrentRequests: The number of concurrent requests for a given key at any given time.
fallbackMaxConcurrentRequests: The number of concurrent requests for the fallback at any given time.
These are recommended to be left alone unless you know what you're doing:
metricsRollingStatisticalWindowInMilliseconds: How long to keep around metrics for calculating rates.
metricsRollingStatisticalWindowBuckets: How many different metric windows to keep in memory.
Once you are done configuring your Tenacity dependencies. Don't forget to tweak the necessary connect/read timeouts on HTTP clients. We have some suggestions for how you go about this in the Equations section.
One of the great things about Tenacity is the ability to aid in the reduction of mean-time-to-discovery and mean-time-to-recovery for issues. This is available through a separate service Breakerbox.
Breakerbox is a central dashboard and an on-the-fly configuration tool for Tenacity. In addition to the per-tenacity-command configurations shown above this configuration piece let's you define where and how often to check for newer configurations.
breakerbox: urls: http://breakerbox.yourcompany.com:8080/archaius/{service} initialDelay: 0s delay: 60s waitForInitialLoad: 0s
urlsis a list of comma-deliminated list of urls for where to pull tenacity configurations. This will pull override configurations for all dependency keys for requested service.
initialDelayhow long before the first poll for newer configuration executes.
delaythe ongoing schedule to poll for newer configurations.
waitForInitialLoadis the amount of time to block Dropwizard from starting while waiting for
Breakerboxconfigurations.
In order to add integration with Breakerbox you need to implement the following method in your
TenacityBundleConfigurationFactoryimplementation:
@Override public BreakerboxConfiguration getBreakerboxConfiguration(CompletieConfiguration configuration) { return configuration.getBreakerbox(); }
Configurations can happen in a lot of different spots so it's good to just spell it out clearly. The order in this list matters, the earlier items override those that come later.
How to configure your dependent services can be confusing. A good place to start if don't have a predefined SLA is to just look at actual measurements. At Yammer, we set our max operational time for our actions somewhere between p99 and p999 for response times. We do this because we have found it to actually be faster to fail those requests, retry, and optimistically get a p50 response time.
Tenacity
p99 + median + extra
(p99 in seconds) * (m1 rate req/sec) + extra
HTTP client
33% of executionIsolationThreadTimeoutInMillis
110% of executionIsolationThreadTimeoutInMillis
Note: These are just suggestions, feel free to look at Hystrix's configuration documentation, or implement your own.
Tenacity adds resources under
/tenacity:
GET /tenacity/propertykeys: List of strings which are all the registered propertykeys with Tenacity.
GET /tenacity/configuration/{key}: JSON representation of a
TenacityConfigurationfor the supplied {key}.
GET /tenacity/circuitbreakers: Simple JSON representation of all circuitbreakers and their circuitbreaker status.
GET /tenacity/circuitbreakers/{key}: Single circuitbreaker status
PUT /tenacity/circuitbreakers/{key}: Expected "FORCEDCLOSED, FORCEDOPEN, or FORCED_RESET" as the body.
GET /tenacity/metrics.stream: text/event-stream of Hystrix metrics.
By default these are put onto the main application port. If you want to place these instead on the admin port, you can configure this when building the
Tenacitybundle.
TenacityBundleBuilder . newBuilder() ... .usingAdminPort() .build();
An exception mapper exists to serve as an aid for unhandled
HystrixRuntimeExceptions. It is used to convert all the types of unhandled exceptions to be converted to a simple HTTP status code. A common pattern here is to convert the unhandled
HystrixRuntimeExceptions to 429 Too Many Requests:
TenacityBundleBuilder . newBuilder() .configurationFactory(configurationFactory) .mapAllHystrixRuntimeExceptionsTo(429) .build();
If you don't handle logging exceptions explicitly within each
TenacityCommand, you can easily miss problems or at-least find them very hard to debug. Instead you can add the
ExceptionLoggingCommandHookto the
TenacityBundleand register
ExceptionLoggers to handle the logging of different kinds of Exceptions. The
ExecutionLoggingCommandHookacts as a
HystrixCommandExecutionHookand intercepts all Exceptions that occur during the
run()method of your
TenacityCommands. By sequencing
ExceptionLoggers from most specific to most general, the
ExceptionLoggingCommandHookwill be able to find the best
ExceptionLoggerfor the type of Exception.
TenacityBundleBuilder . newBuilder() .configurationFactory(configurationFactory) .mapAllHystrixRuntimeExceptionsTo(429) .commandExecutionHook(new ExceptionLoggingCommandHook( new DBIExceptionLogger(registry), new SQLExceptionLogger(registry), new DefaultExceptionLogger() )) .build();
TenacityJerseyClientand
TenacityJerseyClientBuilderto reduce configuration complexity when using Tenacity and
JerseyClient. At the moment these two have competing timeout configurations that can end up looking like application exceptions when they are simply
TimeoutExceptions being thrown by JerseyClient.
TenacityJerseyClientaims to fix this by adjusting the socket read timeout on a per-request basis on the currently set execution timeout value for resources built from
TenacityJerseyClientand its associated
TenacityPropertyKey. Note: Metrics associated with the
TenacityPropertyKeyare NOT updated whenever the underlying
Clientis used. The
TenacityPropertyKeymetrics are only ever updated when using
TenacityCommandor
TenacityObservableCommandat the moment.
Client client = new JerseyClientBuilder(environment).build("some-external-dependency"); Client tenacityClient = TenacityJerseyClientBuilder .builder(YOUR_TENACITY_PROPERTY_KEY) .usingTimeoutPadding(Duration.milliseconds(50)) //Padding to add in addition to the Tenacity set time. Default is 50ms. //Result: TenacityTimeout + TimeoutPadding = SocketReadTimeout .build(client);//Then use tenacityClient the same way as you'd use client. TenacityClient overrides resource/asyncResource and those in turn are Tenacity*Resources. //They adjust timeouts on every use or on a per-request basis.
There is now the ability to add a
HealthCheckwhich returns
unhealthywhen any circuit is open. This could be because a circuit is forced open or because of environment circumstances. The default is for this not to be turned on. You can enable this
HealthCheckby configuring it on the bundle that is added to your application.
TenacityBundleBuilder .newBuilder() ... .withCircuitBreakerHealthCheck() .build();
Java8 brings functional interfaces and
TenacityCommand/
TenacityObservableCommandoffers support.
TenacityCommand .builder(DependencyKey.GENERAL) .run(() -> 1) .fallback(() -> 2) .execute()
TenacityObservableCommand .builder(DependencyKey.GENERAL) .run(() -> Observable.just(1)) .fallback(() -> Observable.just(2)) .observe()