In this article, let us try and understand the concept of Model Drift
What Is Model Drift?
We live in a world that is constantly evolving. What was once cutting-edge has become outdated, and what is now a unique notion will most certainly be forgotten in a few years. For organizations, it is critical to comprehend change. Organizations that grasped the shift in consumer behaviour outlasted the competition, while those who couldn’t keep up with the rate of change were the most impacted.
Similarly, the environment and data used by a machine learning model to perform a job or make predictions are continuously changing. As a result of these changes, the model’s weights/perceptions may be skewed which may lead to loss of predicting ability on the model. This change in the ML model is referred to as the drift.
Drift is a phrase that describes how the performance of predictive ML models, as well as the relationship between the variables, degrades as the environment changes. It emerges into ML models over time and, if not detected promptly, can harm pipeline or services.
Figure below shows at what stage in the ML lifecycle, Model Drift and Data Drift can be analysed.
These shifts/drift can be of many types:
Data drift
It occurs due to changes in the statistical properties of the independent/predictor variables.
Concept drift
A shift in the relationship between the model inputs and the output is referred to as concept drift.
When macroeconomic circumstances make lending riskier and there is a higher threshold to be qualified for a loan ,this is called the concept drift. In this scenario, a formerly credit worthy income level is no longer credit worthy.
Prediction drift
It refers to a change in the model’s predictions.
For example, if your product was introduced in a more affluent region, you could see a higher number of credit-worthy applicants. The model remains valid, but your company may be unprepared for this circumstance.
Label drift
A shift in the output or label distribution of the model.
In this article, we will be focusing on data drift and concept drift. Furthermore, we will throw some light on why it occurs, its ramifications, how we may identify them, and, finally, how to counteract their impacts.
Data drift formally is referred to as the dataset drift between source distribution (S) and a target distribution (T), is defined as a change in the joint distribution of features (x) and target (y). Mathematically, It is represented as follows.
In layman’s terms, data drift occurs when the input data distribution of the variables differs significantly from the data on which training occurred. As a result, the trained model becomes obsolete, and the model’s quality suffers. When developing a Machine Learning model, the goal is to discover the relationships between the input and the target variable. After creating such a model based on this data, we may input additional data from the same distribution and expect similar results. But this however, rearely occurs in reality.
Changing consumer habits, technological advancements, economic, political, and other uncontrollable variables can all have a significant impact on either:
The input dataset, the target variable or the underlying patterns and relationships between the input and output data.
Why Data Drift occurs?
Before we talk about the reasons for data drift, it’s essential to understand that no machine learning model can last forever.
“Past performance is not necessarily indicative of future results” – J. M. Brain[2]
Model performance might suffer as a result of poor data quality, malfunctioning pipelines or technical bugs. If the cause of the performance drop is not one of these, it would be due to model drift. Model drift is also called decay or staleness. It occurs due to Data drift (changes in data distributions) or Concept drift (change in relationships).
All models degrade with time although the rate of degradation varies from model to model. Few models such as image classification models may be used for years whilst others may need to be tweaked now and then. The model quality metrics like accuracy, mean error rate or any downstream business KPI such as click-through rate are used to assess this deterioration.
When the model’s performance falls below the threshold performance metric, it loses its predictive capacity and is no longer feasible for usage.
What is Concept drift?
Concept drift is the phenomenon that occurs when the statistical characteristics of a class variable also, known as dependent/target variable varies over time.
In other domains, this phenomenon is also called “covariate shift,” “dataset shift,” or “nonstationarity.” For example, changes in customer purchasing behavior over time may be influenced by the strength of the economy.
Four types of concept drift according to severity and speed of changes noisy blips.
Why Concept drift occurs?
While training an ML model, it tries to find a mapping between the independent variables and the dependent variables. The resulting mapping function is then used to predict the new/unseen data points. All this works very well in a static environment. (i.e. no changes evolves in independent nor the dependent variables) Such an environment is great for any ML model. Here the power of predictions never diminishes over time. However, in reality, such an environment is rare.
In reality, a dynamic environment (i.e. any change can occur in independent or the dependent variables at any point of time) is most prominent. Here, not only the statistical properties of the dependent variable change but also its meaning, which makes the old mapping function irrelevant.
Understanding Concept drift with an example
Consider the cybersecurity area, where we created a system that alerts us to possible DOS (Denial-of-Service) attacks. This model considers factors like the number of requests received by the server per minute and the source. At the time of training, the server’s traffic of 1000 requests per minute was deemed excessive, indicating that something was amiss. After a few months, the firm started an advertising campaign, which increased the website’s popularity significantly. The old model would now consider this ordinary traffic to be an attack and would continue to issue alerts. It is an example of how the definition of suspicious conduct has shifted in reality.
As you can see in the above depiction that the boundary of the dependent variable which were had a clear separation after the concept drift this separation between the dependent/target variable blurs. Model tries to predict according to the previous class margins on the new reality data(after concept drift). This is the main cause of the loss of predictive powers.
Key reasons for Drift
Unrepresentative training data
Sample selection bias
When ‘training data’ is acquired, on the biased and faulty approach. So, it doesn’t accurately represent the operating environment in which the model is to be deployed.
Changes in hidden variables
The hidden variable, which cannot be assessed directly and can have an impact on some of the independent variables.
Non-stationary environments
Dynamic environment
An environment in which data and relations change as a result of the domain’s intrinsic nature.
For example, competitor moves that alter pricing/offers or the arrival of new competitors may alter the market dynamic.
Technical issue
A faulty pipeline or changes in one of the feature’s values – either owing to a bug or a purposeful unannounced schema change, or even a change in the upstream data source’s default value.
Domain shift
Changes in the meaning of values or terms are referred to as transformations. Inflation, for example, diminishes the value of money, therefore an item’s price or a person’s income will have different consequences at various times.
How can we detect these drift?
There are two main approaches when it comes to drift detection.
Examining the data i.e feature distributions of training and production data.
Monitoring the performance metrics of the model.
By using ML models
Examining the data
Below is the density plot for one of the independent variables. It shows the distribution of training and production data is not symmetrical. Therefore, the performance we get at the time of model building and performance at production server would likely not be same. Other statistical properties of both the data are matched to find inconsistencies.
If found then, that will be our clue to dig deeper into what and where things have changed. We should ask questions like does the meaning of the target variable has changed? Does the data pipeline is bugged? Were there any major event recently which can cause this change?
Monitoring the model’s performance metrics
Data drift can also be monitored by monitoring the model’s performance metrics like F1 Score, Log Loss, Root Mean Squared Error(RMSE) etc. Once the chosen performance metric drops below the threshold (decided by the user) then it is the sign to check why it happened? One of the reasons could be model drift.
By using ML models
A model is used in this method to determine the similarity of a given point or groups of points to a reference baseline. While utilising ML-based approaches can provide a more accurate picture of drift describing the technique’s foundation might be difficult for forming an intuitive.
How to prevent drift
The main question now is whether this situation can be prevented and what steps can be taken to prevent it from occurring.
Ways to address drift
Online Learning: The learner is continually updated as the model processes one sample at a time during online learning. In reality, the vast majority of real-world applications are dependent on streaming data and online learning is the most often used way of minimizing concept drift.
Remove features: Another method to cope with concept drift is to remove features. Multiple models are created one feature at a time, and features with poor AUC-ROC response are dropped.
Ensemble learning with model weighting: It is a technique in which many models are ensembled and the result is a weighted average of each model output.
Re-train the model on a regular basis which can be triggered at various times, such as when the model performance falls below a certain threshold or when the average confidence score between two windows of data shows significant drift.
References
1. Online Ensemble Learning with Abstaining Classifiers for Drifting and Noisy Data Streams – Scientific Figure on ResearchGate.
2. http://iwann.ugr.es/2011/pdf/InvitedTalk-FHerrera-IWANN11.pdf
3. https://www.researchgate.net/figure/Static-assumption-and-Naivety-toward-adversarial-activity-puts-ML-at-risk_fig3_325235071
4. Bare Demo of IEEEtran.cls for Computer Society Journals (arxiv.org)
5. Quionero-Candela, Joaquin, et al. Dataset shift in machine learning. The MIT Press, 2009.
6. J. M. Brain, ““Past performance is not necessarily indicative of future results” — The proven-in-use argument and the retrospective application of modern standards,” 5th IET International Conference on System Safety 2010, 2010, pp. 1-4, doi: 10.1049/cp.2010.0833.
7. https://towardsdatascience.com/how-to-detect-model-drift-in-mlops-monitoring-7a039c22eaf9
So,this was an overview on the concept of Model Drift.
In our next article, Let’s look at Tools and Technologies for MLOps.
Stay tuned.
Check out the earlier articles in this series to understand how to install MLflow and implementing MLOps using MLflow.
Author
Explore No-code, automated machine learning for analytics teams here.
