Groupon_Controlled Experimentation_Panel_The Hive
0 likes621 views
Controlled experimentation (A/B testing) allows companies to systematically study the effects of potential product changes or treatments by randomly assigning users to a control group or treatment group. The document discusses how controlled experiments can validate hypotheses with data, determine if a treatment has a causal effect, and provide examples of how A/B testing can be used for website variants, call-to-actions, and personalized recommendations. It also outlines best practices for running controlled experiments such as ensuring identical distributions between control and treatment groups and carefully monitoring each variant.
Groupon_Controlled Experimentation_Panel_The Hive
- 1. Controlled Experimentation to Guide Product Innovation! ! Rajesh Parekh!
- 2. Controlled Experimentation (A/B Testing)! • Method to study effects of a treatment # • Concept:! - Randomly split users into two groups# Randomly) Divide) A : Control# B: Treatment# - A and B are identical to each other except A)(Control)) B)(Treatment)) for the treatment being evaluated# - Collect performance metrics from the experiment# - Run statistical tests to determine if differences between A and B are purely by chance# Measure)&) Evaluate) Controlled)Experimenta=on)Panel) 2
- 3. Why Run Controlled Experiments?! • Commonly used approach in clinical trials! - What is the effect of a particular drug / treatment?# • Systematically validate hypotheses with data! ! • Concurrently run the treatment and control! - The difference (if any) is# Because of the treatment OR# Due to random chance# • Determine if a treatment is causal in nature! - E.g., Making the search box bigger causes increase in queries / user# Controlled)Experimenta=on)Panel) 3
- 4. Controlled Experimentation: Use Cases! # A"B"Stract"Widget"Company" A"B"Stract"Widget"Company" _________________) _________________) _________________) _________________) _________________) _________________) _________________) BUY)NOW) _________________) BUY"NOW" Website)Variants) Controlled)Experimenta=on)Panel) 4
- 5. Controlled Experimentation: Use Cases! # Free)Trial) Play)Now) Mobile)Call)to)Ac=on) Controlled)Experimenta=on)Panel) 5
- 6. Controlled Experimentation: Use Cases! # Top)deal)highlighted) Email)Template)Design) Controlled)Experimenta=on)Panel) 6
- 7. Controlled Experimentation: Use Cases! # Backend)changes)(e.g.,)Personaliza=on)Algorithm)) Controlled)Experimenta=on)Panel) 7
- 8. Controlled Experimentation: Use Cases! # • Follow-up message for users who previously clicked on an ad# • Incentive campaign to re- engage lapsed users# • Think of this as placing filters / guards on a randomly chosen user population# Custom)Defined)User)Segments) Controlled)Experimenta=on)Panel) 8
- 9. Key Components of an Experimentation Platform! Hashing function! Metrics – suite of KPI! Group)0) ! ! Revenue" ! ! F()))))))))))))) Group)1) ! ! Time"Spent" Abandonment" Click>Through"Rate" ! Group)NU1) ! ! ! Session"Length" Purchase"Rate" Logging! Dashboard! ! ! ! ! • Detailed)logging)of)all)user)interac=ons) • Metric)improvements)and)Sta=s=cal) Significance)in)a)central)place) Controlled)Experimenta=on)Panel) 9
- 10. Ensure Identical Control and Treatment! Gender"" • Custom Segments# Male) Female) • Frequency Distribution# CONTROL" TREAMENT" Region"Size" Small) Medium) Large) CONROL" TREATMENT" Prior"Exposure" • Large Difference in Prior Exposure Rate violates δ%" assumptions# No) Yes) CONROL" TREATMENT" Controlled)Experimenta=on)Panel) 10
- 11. A/A Tests! • Run an experiment with two identical variants# • Helps to determine if:# - Users are being split uniformly at random# - Correct data is being logged# - Variance between identical populations of users is acceptable# • Challenge:! - Few purchases of high value deals render statistically significant difference between treatment and control# SPAIN"TRIP" $1,999" Controlled)Experimenta=on)Panel) 11
- 12. Monitor Each Variant! • Place yourself in each variant to validate the experience# ! • Wrong sort order!! ! ! Carefully)inspect)each)variant) Controlled)Experimenta=on)Panel) 12
- 13. Objective Function! # # Conversion" Revenue" P(conversion)) E(rev))=)P(conversion))*)price) ) ) • Favors)lower) • More)expensive)deals)can) price)deals) dominate) Need)to)balance)mul=ple,)oZen)conflic=ng)objec=ves)) Controlled)Experimenta=on)Panel) 13
- 14. Measure Overall Impact! • Test focuses on# - A particular area of the website# - A sub-population of users# • Measure! - Improvement on the sub-segment AND# - Entire population!# Measure)overall)impact)to)guard)against)cannibaliza=on) Controlled)Experimenta=on)Panel) 14
- 15. Panel Discussion: Questions! Controlled)Experimenta=on)Panel) 15
- 16. Acknowledgements# Thanks to many talented individuals at Groupon I am privileged to work with!# • Data Science# • Engineering# • Marketing / Market Research# Controlled)Experimenta=on)Panel) 16
- 17. Rajesh Parekh! Groupon! [email protected]! ! Controlled)Experimenta=on)Panel) 17