Learning Curves and Diagnosing Model Problems
When a model isn't performing well, it's tempting to just start randomly trying things — a bigger model, more data, different features. A learning curve gives you a much faster, evidence-based way to figure out what's actually wrong before you spend time on the wrong fix.
What a learning curve is: a plot of training and validation error (or accuracy) against the amount of training data used (or, in a related version, against training time/epochs). It shows you not just how the model is doing, but how it's trending — which tells you a lot about what would actually help.
How to read one:
- Both training and validation error are high, and close together. The model is underfitting — it isn't even doing well on the data it has. More data probably won't help much on its own; you likely need a more expressive model, better features, or less regularization. See Bias-Variance Tradeoff.
- Training error is low, validation error is noticeably higher, and the gap isn't closing. The model is overfitting. More training data often helps directly here, and so does regularization, a simpler model, or data augmentation.
- Both curves are still improving and haven't plateaued as more data is added. You're likely data-limited — collecting more training data is probably one of the highest-leverage things you can do.
- Both curves have flattened out and validation error is acceptable. You're near the limit of what this model/feature combination can achieve; further gains likely require a different model, better features, or a rethink of the problem itself.
Why this matters more than it seems: without this kind of diagnosis, it's very easy to spend days collecting more data when the real problem was an underfitting model that needed more capacity — or to spend days building a bigger model when the real problem was noisy, insufficient data. A learning curve turns "something's wrong, let's guess" into "here's specifically what's wrong."
Related diagnostics worth knowing:
- Validation curves — the same idea, but plotted against a specific hyperparameter (e.g. tree depth, regularization strength) instead of dataset size, useful for choosing that hyperparameter directly.
- Error analysis — manually look at a sample of the examples the model gets wrong. Patterns often jump out (a particular category is consistently confused, a certain input length breaks the model) that a single aggregate metric would never reveal.
Where to go deeper: scikit-learn's learning curve documentation includes ready-to-use code for plotting both learning and validation curves, with example plots showing each of the patterns described above.