Statistical Anomaly Detectors

In this document, we compare three methods: the Standard Deviation Method, the Percentile Based Anomaly Detector, and the Tukey Fences approach. These methods represent different variants of Dynamic Baselining; the Standard Deviation Method serves as our legacy approach, while we benchmark it against two more recent alternatives.

• Standard Deviation (SD) Method

The SD method establishes a threshold for outliers, considering all points above this threshold as anomalous. For a given set of observations X related to a specific metric and time window, we define this threshold as follows:

In essence, we assess points that exceed a specified number of standard deviations added to the mean of the observations. This method has a single tuning parameter (num_sd), which dictates the algorithm's sensitivity; a lower value results in a higher number of detected anomalies, while a higher value does the opposite.

• Percentile Based Anomaly Detector

The SD Method assumes that the timing metric (e.g., response time) aggregated over a time window follows a Gaussian distribution. If the distribution is indeed Gaussian, it is feasible to calibrate confidence levels to the appropriate standard deviation bounds and identify anomalies. This feature is crucial for our use case, as it provides users with control over the desired sensitivity and confidence levels. However, if the Gaussian assumption is violated (e.g., due to a heavy-tailed distribution), calibration errors may occur. Consequently, users may struggle to manage the number of detected anomalies based on their target confidence or sensitivity levels. Ideally, a high confidence and low sensitivity level should yield only a few extreme values, while a low confidence and high sensitivity level should result in a greater number of anomalies.

Under the Gaussian assumption, setting num_sd = 1 corresponds to a confidence level of 68%, meaning approximately 68% of samples would fall within that limit, leaving 32% (100–68) as anomalies. Increasing num_sd to 2 raises the confidence level to 95%, resulting in 5% of samples being flagged as anomalies. The following illustration demonstrates the varying threshold curves for num_sd = 1, 2, and 3, where samples crossing these limits are identified as anomalies:

The red dots in the image signify anomalies flagged by the num_sd = 3 threshold, which theoretically corresponds to a 99.7% confidence level. If the distribution were Gaussian, one would expect far fewer anomalies (red dots). However, as the actual distribution is not Gaussian, a considerable number of anomalies are detected, even at the highest confidence level. The prevalent high number of false positives flagged by the SD algorithm stems from miscalibration due to the non-Gaussian nature of the underlying distribution. This calibration error leads to a lack of control over the number of anomalies based on the desired confidence and sensitivity levels.

To address this issue, we adopted a strategy whereby instead of calculating the mean and standard deviation, the anomaly detector computes rolling percentiles (quantiles) over the same time window for a specified sensitivity level; the resulting envelope from varying confidence levels can then be employed to identify anomalies. This technique is known as the percentile (quantile) anomaly detector.

In Figure 3, the confidence levels for the percentile detector align with the 1, 2, and 3 standard deviation boundaries of the SD detector. The red dots in both plots indicate the highest confidence level. The figure illustrates that the highest percentile substantially reduces false positives identified by the SD approach.

Below, we present a calibration plot:

In Figure 4, the "ideal" category represents the target confidence level. For example, at a low sensitivity level, the confidence level is 68%, leading to 32% anomalies; similarly, a 95% confidence level should ideally result in 5% anomalies. The percentile (quantile) algorithm demonstrates a closer match to the target confidence levels compared to the STD (rolling standard deviation) approach. The total deviation from the ideal for the STD detector is 15%, while the percentile detector achieves a reduction to 6%, resulting in a 60% improvement in calibration.

In addition to calibration (or the ability to control the number of anomalies) according to customer preferences, the percentile detector identifies anomalies of superior quality. The following plot illustrates the distinction between anomalous and non-anomalous samples for both STD and percentile anomaly detectors:

Figure 5 reveals that the percentile algorithm more effectively separates anomalous samples from non-anomalous ones, as evidenced by the reduced overlap in the histogram. The effect size, which measures the separation between the two categories, improves from 0.73 for the STD method to 0.85 for the percentile method, indicating a 16% enhancement in distinguishing anomalies from normal samples.

The percentile algorithm has been further refined and optimized to create a high-performance online solution using the Tukey Fences method.

• Tukey Fences Method

The Tukey Fences algorithm similarly establishes a threshold for outliers, categorizing all points above this threshold as anomalous. For a given set of observations X related to a specific metric and time window, we define this threshold as follows:

Here, P_low and P_high represent the lower and upper percentiles, respectively, while k is a sensitivity parameter akin to the num_sd in the SD method. Essentially, we examine the observations over the time window to identify a low percentile value and a high one; we then multiply the distance between these values by k and add the result to the low percentile value to establish the outlier limit. Many of our clients’ metrics exhibit positively-skewed distributions, suggesting that low percentile values will be close together, whereas high percentile values can escalate quickly. This insight indicates that the algorithm will be more influenced by the choice of k-parameter and P_high than by P_low.