Evaluate Performance of Trained Models in TYDE

In our previous blog post, ‘Artificial Intelligence in Tyde’, we introduced the artificial intelligence technique we used, how we train the models and how we use the trained models for anomaly detection.

Now you may be curious about what is the performance of the models. So, in this article, we will take a closer look at the models.

In the asset diagram, we select an asset to check the anomaly score generated by the model and the corresponding sensor data.

Now we are in the user interface for the selected asset, on the top of the user interface, we can see the model’s version and when this model is trained, we find this model is trained at 29 August,

We call data before this day as historical data and the corresponding anomaly situations as historical anomaly situations.

After the model is trained, we use the trained model for real-time anomaly detection, so we call data after this day as real-time data, and the corresponding anomaly situations as real-time anomaly situations.

In the user interface, we can see the different months, days marked in different colors.

If there are no anomaly situations in a month, the month is marked in green, otherwise in red, it’s the same for the days, so we can easily find in which month, which days have anomaly situations.

Let’s start to take a look at one of the historical anomaly situations. We can see there are no anomaly situations in July and August, but June is marked in red color, it means there are some anomaly situations in June, we click June, then we can find there is an anomaly situation at 18 June.

We find it’s an obvious anomaly situation, the anomaly score increased from 0 to 1, and the sensor readings in this period are significantly different from the rest of the time.

What happens? We check the reason for this anomaly situation through a closer look at the sensor readings.

We can see the flow rate through Klappeluke 2, which routes the water around the turbine has a dramatic increase.

We also find that in the duration of the anomaly, water to the aggregate drops to zero.

In addition to the dramatic increase of anomaly scores, we can also find some situations with a slight increase in the anomaly scores.

For example, on 3 May, the anomaly score increased slightly, from 0 to around 0.2.

To have a clearer observation, we can check the long term trend of the sensor readings.

In Tyde, users can further investigate the model’s detailed information using this tool, called orchestrator.

Click this ‘C’, we can open a plot of all the training data.

We can see the data around 3 May is clearly different as compared to most of the other data used for training.

After checking the historical sensor readings and historical anomaly situation, let’s further check the real-time anomaly detection.

This model is trained at 29 Aug, there are no anomaly situations in September, we can see September is green, but there are some anomaly situations in October.

On 11 Oct, the hydropower plant had an anomaly situation, it is because of a short-term shutdown, and we can see the trained model successfully detected this situation.

In the real-time anomaly detect, we can also find the model is reacting for the slight change of sensor data.

For example, for this asset, because of the change of sensor readings, the anomaly score started to increase slightly.

We find the anomaly score is reflecting the variations of the sensor readings, because of the sensor data only changed a little bit, so this period is not marked as an anomaly situation.

In this blog, through check the model’s performance, we find the trained model successfully detected anomaly situations, both for the historical training data and the real-time data.

Additionally, we find the model not only reacting to the significant change of sensor data but also could reflect a slight change of the sensor data.

In the next step, we will implement functionalities that allow users to simulate anomaly situations and evaluate the model’s performance.

Please stay tuned for our next blog.

Share This

Share on facebook
Share on twitter
Share on whatsapp