Machine Learning Basics · 15 min · 170 XP
Overfitting and how to spot it
Recognise a model that memorised its training data, and dial its flexibility back.
A model overfits when it learns the training data too well: noise, typos and one-off flukes included. The telltale sign is the gap between scores. Excellent on the training set, noticeably worse on validation. A model that's bad on both is underfitting: too simple, or missing the features that matter.
k-nearest neighbours makes this easy to see. To classify a new point, find the k closest training points and take a majority vote. With k = 1 every training point is its own nearest neighbour, so training accuracy is always 100%, and a single mislabelled point carves out a little region of wrong answers around itself. A larger k outvotes the noise, and past a point it starts to blur real boundaries.
Every model family has a dial like k: tree depth, the number of features, regularisation strength. Choose it on validation data, never training data. With little data, use cross-validation: split into, say, 5 folds, validate on each fold in turn, and average the scores, so one lucky split doesn't decide.
from sklearn.model_selection import cross_val_score
from sklearn.neighbors import KNeighborsClassifier
for k in (1, 3, 5, 9):
scores = cross_val_score(KNeighborsClassifier(n_neighbors=k), X, y, cv=5)
print(k, scores.mean())model train accuracy validation accuracy deep tree 99% 71% shallow tree 84% 82%
Loading your workspace…