Sciweavers

Share
KDD
2009
ACM

Seven pitfalls to avoid when running controlled experiments on the web

11 years 11 days ago
Seven pitfalls to avoid when running controlled experiments on the web
Controlled experiments, also called randomized experiments and A/B tests, have had a profound influence on multiple fields, including medicine, agriculture, manufacturing, and advertising. While the theoretical aspects of offline controlled experiments have been well studied and documented, the practical aspects of running them in online settings, such as web sites and services, are still being developed. As the usage of controlled experiments grows in these online settings, it is becoming more important to understand the opportunities and pitfalls one might face when using them in practice. A survey of online controlled experiments and lessons learned were previously documented in Controlled Experiments on the Web: Survey and Practical Guide (Kohavi, et al., 2009). In this follow-on paper, we focus on pitfalls we have seen after running numerous experiments at Microsoft. The pitfalls include a wide range of topics, such as assuming that common statistical formulas used to calculate s...
Thomas Crook, Brian Frasca, Ron Kohavi, Roger Long
Added 25 Nov 2009
Updated 25 Nov 2009
Type Conference
Year 2009
Where KDD
Authors Thomas Crook, Brian Frasca, Ron Kohavi, Roger Longbotham
Comments (0)
books