Two ways to measure the cumulative impact of experiments
Mature experimentation programs eventually have to report the cumulative impact of their shipped changes. The request might come as an ROI story for leadership, a revenue update for finance, or a gut check on the quarter’s progress. The tempting shortcut is to sum the observed lift from each winning experiment and report the total. That naive sum almost always overstates the truth because of a statistical artifact called the winner’s curse.