Sciweavers

KDD
2012
ACM

Trustworthy online controlled experiments: five puzzling outcomes explained

11 years 6 months ago
Trustworthy online controlled experiments: five puzzling outcomes explained
Online controlled experiments are often utilized to make datadriven decisions at Amazon, Microsoft, eBay, Facebook, Google, Yahoo, Zynga, and at many other companies. While the theory of a controlled experiment is simple, and dates back to Sir Ronald A. Fisher’s experiments at the Rothamsted Agricultural Experimental Station in England in the 1920s, the deployment and mining of online controlled experiments at scale—thousands of experiments now—has taught us many lessons. These exemplify the proverb that the difference between theory and practice is greater in practice than in theory. We present our learnings as they happened: puzzling outcomes of controlled experiments that we analyzed deeply to understand and explain. Each of these took multiple-person weeks to months to properly analyze and get to the often surprising root cause. The root causes behind these puzzling results are not isolated incidents; these issues generalized to multiple experiments. The heightened awareness...
Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbot
Added 28 Sep 2012
Updated 28 Sep 2012
Type Journal
Year 2012
Where KDD
Authors Ron Kohavi, Alex Deng, Brian Frasca, Roger Longbotham, Toby Walker, Ya Xu
Comments (0)