Data Drift and Concept Drift: When Your Model Goes Stale
Every model is trained on a snapshot of the world at a particular moment. The world doesn't hold still afterward. "Drift" is the general term for what happens when the real-world data a deployed model sees starts to diverge from what it was trained on — and it's one of the main reasons a model that performed great at launch can quietly get worse over time, with no code changes at all.
Data drift (also called feature or covariate drift) — the distribution of the input features changes, even if the underlying relationship between features and target stays the same. Example: an e-commerce model trained mostly on desktop-browser traffic starts seeing a much higher proportion of mobile traffic. The relationship between behaviour and purchase intent might be unchanged, but the mix of inputs the model sees has shifted, which can still degrade performance if the model hasn't seen enough of that pattern before.
Concept drift — the actual relationship between the inputs and the target changes. Example: a model predicting "will this customer churn" trained before a major product change may simply no longer reflect why customers leave, because the underlying reasons themselves have changed. This is generally the more serious case, since no amount of "more of the same kind of data" fixes it — the pattern the model learned is now genuinely out of date.
How to detect drift:
- Monitor performance metrics directly, when ground truth becomes available (even with a delay) — a drop in accuracy, F1, or another task-specific metric is the most direct signal, but it's often only available after the fact.
- Compare input feature distributions between training data and recent production data — statistical tests (like the Kolmogorov-Smirnov test or population stability index) or simply plotting distributions side by side can flag when they've meaningfully diverged.
- Watch the distribution of the model's own predictions — a sudden shift in how often the model predicts each class, or in the range of its predicted values, is often a useful early warning even without ground truth labels.
- Segmented analysis — check whether drift or performance decline is concentrated in a specific subgroup rather than spread evenly, which changes how you'd respond to it.
What to do once you've detected it: the usual response is some form of retraining on more recent data, but it's worth first understanding why the drift happened — a genuine, lasting shift in the real world calls for a different response than a temporary blip (a one-off event, a data pipeline bug that's mimicking drift) that might resolve on its own or need a different fix entirely.
Why is this important? Drift is one of the main reasons machine learning systems need ongoing maintenance in a way that a lot of traditional software doesn't — the code can be perfectly correct and unchanged while the model quietly becomes wrong, simply because reality moved. Planning for this from the start (via monitoring and a retraining plan) is part of treating a model as a living system rather than a one-time deliverable.
Where to go deeper: Evidently AI's guides on data drift and concept drift cover both concepts with clear definitions, detection methods, and worked examples.