CanaryAdvisor     

CanaryAdvisor - overview

Canary testing is a novel technique that significantly reduces the risk of down time in a production system by quickly identifying system degradations when a new version is deployed. However, canary testing requires collecting and analyzing vast amounts of data coming from the running application. Analyzing this data manually is a laborous task.

During a canary test, a new version of a microservice (called the “canary”) is deployed alongside the stable running version (called the “baseline”). Then a small portion of the user traffic is diverted to the canary. The behavior of the canary is compared to the behavior of the baseline by monitoring streams of metrics which describe performance and correctness of these versions. Any unexpected degradation in performance, correctness, or resource consumption by the canary results in a canary test failure. In the case of a failure, all the traffic is re-routed back to the baseline, and the canary is aborted, resulting in a minimal disruption to the applicaiton. Otherwise, the canary is considered to be safe for larger-scale deployment and the canary test succeeds. In the case of success, all the traffic is routed to the canary version, and the canary becomes the new baseline version.
Canary testing significantly reduces the risk of introducing failures into the production version of the software. However, canary testing poses a unique set of challenges. Canary testing requires collecting a vast amount of raw information from the running system, and transforming it to the streams of higher-level, informative metrics that are meaningful to compare across application versions. Values of these metrics are often biased due to the stochastic nature of the cloud-based systems, and this bias should not affect the outcome of the canary test. Finally, the failures and performance degradations must be detected as quickly as possible, to reduce the impact on the production system. At the same time, the test results must be accurate.
To address these difficulties we developed a CanaryAdvisor – a tool to automate canary testing of cloud-based applications. CanaryAdvisor works by continuously monitoring baseline and canary versions of a microservice application. It collects values from system performance counters and application logs, and transforms them into metrics that characterize performance, correctness, and resource utilization of the system. CanaryAdvisor compares the values of these metrics collected from the baseline and the canary. To ensure meaningful results in a noisy, non-determistic cloud environment, CanaryAdvisor employs statistical methods to check if metric values significantly differ between the baseline and canary.