Data Analytics Toolkit · 14 min · 170 XP

Comparisons that hold up

Compare rates rather than counts, spot when the mix of groups flips a result, and don't read causes into correlations.

"Region North had 50 refunds and South had 20" says nothing until you know how many orders each had. Compare rates (refunds ÷ orders), not raw counts, whenever the groups differ in size. Show the base alongside the rate, because 50% of 4 orders isn't the same evidence as 50% of 4,000.

Say whether a change is in percentage points or percent. A conversion rate going from 10% to 12% is +2 points, or +20%. Both are true, and they sound very different; writing the wrong one is a common way to exaggerate a result by accident.

Simpson's paradox: a comparison can point one way inside every group and the opposite way overall, when the groups are mixed in different proportions. In an A/B test, variant B can win on mobile and on desktop yet lose overall, simply because B happened to get most of the mobile traffic, which converts worse for everyone. The fix is to compare like with like, within segments, and to check that the groups were comparable in the first place.

The A/B test in this lesson's challenge
variant  device   visitors  conversions   rate
A        mobile        200           10     5%
A        desktop       800           80    10%
B        mobile        800           48     6%
B        desktop       200           22    11%

Overall: A 90 / 1000 = 9%     B 70 / 1000 = 7%

Finally, correlation isn't causation. Customers who use the mobile app spend more, but maybe your biggest spenders are the ones who bother to install the app. Before saying "X causes Y", ask what else could explain both. The reliable way to establish cause is a controlled experiment: randomise who gets X.

Loading your workspace…