Data Analytics Toolkit · 14 min · 170 XP
Comparisons that hold up
Compare rates rather than counts, spot when the mix of groups flips a result, and don't read causes into correlations.
"Region North had 50 refunds and South had 20" says nothing until you know how many orders each had. Compare rates (refunds ÷ orders), not raw counts, whenever the groups differ in size. Show the base alongside the rate, because 50% of 4 orders isn't the same evidence as 50% of 4,000.
Say whether a change is in percentage points or percent. A conversion rate going from 10% to 12% is +2 points, or +20%. Both are true, and they sound very different; writing the wrong one is a common way to exaggerate a result by accident.
Simpson's paradox: a comparison can point one way inside every group and the opposite way overall, when the groups are mixed in different proportions. In an A/B test, variant B can win on mobile and on desktop yet lose overall, simply because B happened to get most of the mobile traffic, which converts worse for everyone. The fix is to compare like with like, within segments, and to check that the groups were comparable in the first place.
variant device visitors conversions rate A mobile 200 10 5% A desktop 800 80 10% B mobile 800 48 6% B desktop 200 22 11% Overall: A 90 / 1000 = 9% B 70 / 1000 = 7%
Finally, correlation isn't causation. Customers who use the mobile app spend more, but maybe your biggest spenders are the ones who bother to install the app. Before saying "X causes Y", ask what else could explain both. The reliable way to establish cause is a controlled experiment: randomise who gets X.
Loading your workspace…