Model Retraining Strategies
Once you accept that models can go stale (see Data and Concept Drift), the natural next question is: when, and how, should you actually retrain? There's no single right answer — the right strategy depends on how fast your data changes, how costly retraining is, and how costly a stale model is.
Common retraining triggers:
- Scheduled retraining — retrain on a fixed cadence (daily, weekly, monthly), regardless of whether drift has been explicitly detected. Simple to implement and reason about, but can waste compute retraining when nothing has meaningfully changed, or retrain too infrequently if the world is moving faster than your schedule assumes.
- Performance-triggered retraining — retrain when monitoring detects that a key metric has dropped below an agreed threshold. More efficient than a fixed schedule, but depends on having a reliable, timely way to measure that performance drop in the first place.
- Drift-triggered retraining — retrain when data or prediction drift crosses a defined threshold, even before a performance drop is confirmed (useful when ground-truth labels are slow to arrive).
- On-demand / manual retraining — a person decides to retrain, usually after noticing a problem or after a known change in the underlying system (a new product launch, a policy change). Common for smaller teams and projects without full automation.
What "retraining" can mean in practice:
- Full retrain from scratch — retrain on the full, updated dataset. Simple and thorough, but can be expensive for large datasets or complex models.
- Incremental / online learning — update the existing model with just the new data, rather than starting over. Cheaper, but not all algorithms support it well, and it can be more prone to the model drifting towards whatever's most recent at the expense of older, still-relevant patterns.
- Sliding window retraining — retrain on only the most recent N months (or N examples) of data rather than the full history, which can help the model stay current when older data is genuinely less relevant — at the cost of potentially losing useful long-term patterns.
Before deploying a retrained model, always re-evaluate it properly — against the same held-out evaluation set (or an updated equivalent) used for the previous version, not just against whatever new data triggered the retrain. This connects directly to A/B Testing and Canary Deployments: a retrained model should generally go through the same careful rollout process as any other new model version, not skip straight to full production traffic just because it's "the same model, just updated."
Why is this important? Without a deliberate retraining strategy, teams tend to drift (no pun intended) into one of two bad defaults: never retraining until something breaks visibly, or retraining reflexively without checking whether the new version is actually better. A clear, agreed strategy avoids both.
Where to go deeper: Google Cloud's MLOps continuous delivery and automation pipelines guide covers retraining triggers as part of a broader automated pipeline design.