What is Anomaly detection?
Anomaly detection (AD) is finding abnormal/out-of-range values in the data pattern of one or more variables. In the case of one variable, the method is called univariate anomaly detection. In case of correlations between multiple variables, the combination of these abnormal variables should be checked. This is called multivariate anomaly detection.
Both types are only applicable to time-series data. The detection can be threefold: streaming, batch or change points detection. The first one evaluates if the latest data point is an anomaly using the data seen up to that point. Batch detection on the other hand flags out anomalies throughout your entire dataset. If trends should be highlighted, change point detection is used.
When to use Anomaly detection?
Anomaly detection is a powerful method to determine when there might be data quality issues or even worse, a problem in the real world. Below, we've listed some possible and common use cases for Anomaly detection.
Possible use cases
- IoT data: imagine having numerous IoT data from machines, knowing the output variables are correlated, but the pattern/possible combinations are unclear as it is a new machine or settings are adapted, anomaly detection can detect impossible/unusual combinations of variables. Danger zones, such as very high or low values can still be filtered out at first hand by applying a rule of thumb. For instance, streaming detection comes in handy in case of continuously checking the values of production processes.
- Preventative maintenance: By highlighting possible last-point anomalies such as in the previous case, one can instantly see if certain high/low values of variables of machines or other devices could indicate future damage. If these anomalies occur more than average, one can be alerted that parts of machines should be replaced or repaired before breaking.
- If average behaviour is unclear or the context has been adapted, data can be gathered over time and anomaly detection can filter out outliers and thus conclusions can be drawn about the averages.
- Data cleaning: garbage in = garbage out a; thus, if an ML model is trained, the data should reflect normal behaviour. As such, anomaly detection can clean the data by filtering out the unusual values. As such, only normal patterns remain for the conditions at hand. Be aware that multivariate detection is a good option to check if this could be a possible value in combination with other conditions. Sometimes, you do not want to clean out too much data to account for less commonly occurring but still reliable situations that might happen in the future and we still want to predict.
There are lots of more examples in the industry depending on the use case. If there are rules of thumb well known in the industry, one can use these, but anomaly detection can be an added value for unsure situations such as IoT data.
What does the Anomaly detector of Azure Cognitive services do?
The resource can be made in Azure creating an API that can handle requests. The detector's managed identity, however, should have 'read access' to the data lake where the time-series data is stored preferably in a CSV. The detector trains the model on a time frame of the data and acts as a black box when applying it to new data.
This directly answers the added benefit of using this service. It is unnecessary to train a model yourself, the computational effort is also diminished and one can easily adapt the features of the API to account for different results.
To make an informed decision, the severity and score per possible anomaly data point are given in the case of multi-variate AD. The lower the severity, the less important the anomaly is to be filtered out. The score, on the other hand, considers the local and global perspective and can even indicate the relevance in a data segment.
Procedure & examples
The package "azure.ai.anomalydetector" is used for univariate and multivariate anomaly detection. The connection to the anomaly detector can then be made with the API key and the endpoint (= resource HTTP) as follows "client = AnomalyDetectorClient(ENDPOINT, AzureKeyCredential(API_KEY))"
As follows:
# Python code
# This is one of the two keys given in the resource name of anomaly detector in your resource
API_KEY = [KEY]
# This is a http of your resource name
ENDPOINT = [HTTP]
# The connection is made to the Anomaly detector
client = AnomalyDetectorClient(ENDPOINT, AzureKeyCredential(API_KEY))
Univariate AD
A possible dataset could be the average temperature in function of the months. If the anomalies are found, the outcome will look like the graph below:
Multivariate AD
In a multivariate case, there are several variables for instance series 0 until series 5. This is a random case in which the variables are correlated.
Important to notice is that the training can be adapted by changing the changing period, the align mode etc... More info can be found via the following link: Best practices for using the Multivariate Anomaly Detector API
In the case now, the align mode is outer for a oneTable case, the following graphs show which variable combinations should be anomalies.
How does Anomaly Detection compare to great expectations (GE)
There are of course other tools and methods to set up Anomaly Detection. We've evaluated how Azure Anomaly Detection compares to a popular framework 'Great Expectations'.
Great expectations are tests for data validation to refine the data quality in different stages of the data processing. For instance, it calculates how many null values and unique values a variable has and if the values lie in a certain predefined range. It is a way of exploring the data and cleaning the data if the behaviour of the data pattern is known. It treats the variables separate from one another, not taking correlations into account.
You can already tell the difference between anomaly detection and great expectations. If correlations and outlier behavior are of importance, the first method is more the way to go, but if data quality in case of knowledge of the data (knowing expected ranges) is a requirement, great expectations is in my opinion a good method.
Of course, the methods can co-exist and anomaly can be a good add-on to investigate the data pattern and remove outliers.
Conclusion
When analyzing your data, the goal is to find a reliable answer to your business question. The data should represent the average, normal behaviour for the answer to be reliable. Anomaly detection can help with this by identifying abnormal values in your data. As mentioned before, anomaly detection can thus have an added benefit in exploring your time-series data pattern, removing outliers and especially alerting for abnormal behaviour when dealing with IoT data. Anomaly detection can even take the correlations of several variables into account setting it apart from the framework "Great Expectations".
One way of implementing anomaly detection without knowing the ins and outs of the ML model behind it is relying on the anomaly detector of Azure Cognitive Services.
If you're interested in getting started with Anomaly Detection, please contact us.
