You conduct an A/B test comparing the current version of an e-commerce site with an alternate upselling
Fantastic news! We've Found the answer you've been seeking!
Question:
- You conduct an A/B test comparing the current version of an e-commerce site with an alternate "upselling" version tat adds additional product recommendations to the checkout process, with visitors randomly assigned to one of the two conditions. This is a membership-based site, so you are confident that each person was consistently in their assigned condition throughout the experiment. You see that average spend is higher in the treatment condition than in the control condition & it's a practically meaningful amount, but the result is not statistically significant when tested in a regression analysis.
- You re-run your initial regression, adding the amount spent by the customer in the prior year on the site as a control variable. The effect is now significant. Why might the significance test result have changed?
- You re-run your initial regression but use log of spend (instead of spending in dollars) as the dependent variable. You find a significant negative effect in this analysis. What is the reason that this difference in results could have happened?
- What is a next step you could take to better understand this difference in results? (This could be an additional analysis with the same data or a follow-up experiment).
Related Book For
Posted Date: