Suppose you are the head of the marketing department for an on-demand video streaming service. A registered user pays a monthly subscription fee to access the platform’s entire library. Your research team has recently developed a new personal recommender system that purportedly works well for users with limited viewing history on the platform. You are excited about the recommender system as many of new users on the platform leave soon after registration, often for a competing platform such as Netflix, citing the difficulty of finding titles they like. The new recommender system could potentially improve the customer retention rate, which measures the probability of a customer staying with an active subscription in the next period and is a key performance indicator for your company.

a. Describe how you could use A/B testing to evaluate the new recommender system. Describe your research design, including whom to recruit for your experiment, how you would administer the test, and what metrics you would monitor to evaluate the new recommender system. State the null and alternative hypotheses for your test.

b. Suppose the current monthly retention rate is 80%. If you wish to detect a 5% increase (in absolute terms) in the retention rate at a 5% significance level and 90% power, assuming a balanced 1 design, how many participants would you need for your experiment? Show your calculations or your R codes to arrive at your conclusion.

c. Suppose your test has indicated a 7% improvement in the retention rate. List and explain at least one factor that could lead to a discrepancy between the observed effect in your test and the actual effect once you deploy the new recommender to the entire platform.

Answer rating: 100% (QA)

Solution a To evaluate the new recommender system we could use AB testing by randomly assigning users to two groups the control group which continues ... View the full answer