Saturday, June 21, 2014

Exposing the Circuit Breaker Pattern as a JMX MXBean

Circuit Breaker Pattern

I first read about the Circuit Breaker pattern in Release It as a way to prevent cascading failures. Martin Fowler has also written about them in CircuitBreaker. The basic idea is that you protect all your operations that are likely to fail or timeout (remote operations) with a circuit breaker that monitors the operation. The breaker is initially in a closed state, but after a failure threshold is reached it opens, or trips. When the breaker is open, no calls are made to the remote resource. This prevents you from having too many callers waiting on a remote resource that is not going to respond. Too many callers waiting and hogging resources could have caused the failure to cascade to their part of the system.

After a period of time the breaker can try the operation again and close, or reset, if successful. It would be nice to be able to view the state and manually open or close the breaker though. This is where JMX comes into the picture.

JMX

Java Management Extensions (JMX) is a Java technology for managing and monitoring resources represented by MBeans. MXBeans are a special type of MBean that is usable by any client. Normally JMX is used in the context of monitoring an application server and the JVM it is running in, but applications are free to provide their own and register them with the application server's MBean server.

Circuit Breaker MXBean

I felt that Release It glossed over how to monitor a circuit breaker. However in the Java world JMX seems like an obvious choice, and I had never written my own JMX MBean, so I wanted to see how it would work.

First you need an interface containing the things that will be exposed through JMX. The getters and setters map to attributes and other methods map to operations.



My implementation of the interface is pretty basic and not well tested, but I think it captures the basic functionality you would want though. Notice it contains an execute method that was not in the interface. I did not think it made sense to expose execute through JMX.



Lastly I created a servlet that registers the MXBean and invokes the execute method with fake tasks.



Controlling the MXBean With JConsole

JConsole is a graphical monitoring tool for JMX resources. You can view and modify the MXBean's attributes and trigger the operations that are exposed. These changes are made without needing to restart the application or application server.


Update May 1, 2015

The e-book Migrating to Cloud-Native Application Architectures also covers circuit breakers and has code examples showing how to add them to a Spring Boot application by using the Netflix OSS Hystrix library. Interestingly, Hystrix also uses another fault-tolerance pattern from Release It called bulkheads by operating each circuit breaker in its own thread pool. This library collects metrics like traffic volume, request rate, latency, etc. and emits them as an event stream which can be aggregated and visualized by other Netflix projects. That's obviously better for a cloud application that what I did above.