Machine Learning Basics · 14 min · 160 XP
Metrics that do not lie
Accuracy, precision and recall, and picking the one that matches the cost of a mistake.
For classification, accuracy (the share of predictions that are right) is the first number everyone reaches for, and often the wrong one. If 5 in 100 orders are fraudulent, a model that says "never fraud" is 95% accurate and catches nothing.
Count the four outcomes instead. This table is called the confusion matrix. For the positive class (fraud):
predicted fraud predicted fine
actually fraud true positive false negative ← missed fraud
actually fine false positive true negative
↑ blocked a good customerPrecision = TP / (TP + FP): of the orders you flagged, how many were really fraud? Recall = TP / (TP + FN): of the real fraud, how much did you catch? Most models output a score, and you choose the threshold. Raising it trades recall for precision, and lowering it does the opposite.
Choose by the cost of each mistake. If missing a case is expensive (fraud, a disease screen), favour recall. If a false alarm is expensive (sending a real customer's email to spam, blocking a card at the till), favour precision. Say the choice out loud, because it's a product decision as much as a technical one.
from sklearn.metrics import confusion_matrix, precision_score, recall_score precision_score(y_true, y_pred) # labels: 1 = fraud, 0 = fine recall_score(y_true, y_pred) confusion_matrix(y_true, y_pred)
Loading your workspace…