Machine Learning Basics · 14 min · 160 XP

Metrics that do not lie

Accuracy, precision and recall, and picking the one that matches the cost of a mistake.

For classification, accuracy (the share of predictions that are right) is the first number everyone reaches for, and often the wrong one. If 5 in 100 orders are fraudulent, a model that says "never fraud" is 95% accurate and catches nothing.

Count the four outcomes instead. This table is called the confusion matrix. For the positive class (fraud):

text
                  predicted fraud   predicted fine
actually fraud    true positive     false negative   ← missed fraud
actually fine     false positive    true negative
                  ↑ blocked a good customer

Precision = TP / (TP + FP): of the orders you flagged, how many were really fraud? Recall = TP / (TP + FN): of the real fraud, how much did you catch? Most models output a score, and you choose the threshold. Raising it trades recall for precision, and lowering it does the opposite.

Choose by the cost of each mistake. If missing a case is expensive (fraud, a disease screen), favour recall. If a false alarm is expensive (sending a real customer's email to spam, blocking a card at the till), favour precision. Say the choice out loud, because it's a product decision as much as a technical one.

With scikit-learn (not run here)
from sklearn.metrics import confusion_matrix, precision_score, recall_score

precision_score(y_true, y_pred)   # labels: 1 = fraud, 0 = fine
recall_score(y_true, y_pred)
confusion_matrix(y_true, y_pred)

Loading your workspace…