We have an idea that we think can increase activity and trading. Mornings are often one of the most volatile times in the day for a market as market makers complete scheduled trades and as people react to after-hours trading and news. We suspect many of our high-value customers miss opportunities during market opening and want to make customers more aware of market hours.

To make users more aware, we ran an experiment where we sent users a push notification every morning at market opening reminding them that the market was opening. Specifically, we sent notifications to active current users (users who installed at least 4 days prior to the experiment).

We limit the experiment to 1,000,000 members in the treatment to reduce risk.

After running the experiment for 14 days, we see the following results in our experiment system:

	Metric	Impact	P-Value
1	D1_TradingRev/User	+0.12%	0.1723
2	D1_OtherRev/User	+0.20%	0.2992
3	D1_Revenue/User	+0.32%	0.0475
4	Daily Sessions/User	+1.98%	0.0022
5	D14_NetPromoterScore	-0.22%	0.2021
6	D1_Retention	+0.03%	0.0495
7	D7_Retention	+0.01%	0.1023
8	D14_Retention	-0.02%	0.0819
9	D1_TotalTimeSpent/ActiveUser	+0.32%	0.1456
10	D7_TotalTimeSpent/ActiveUser	+0.64%	0.0921
11	D14_TotalTimeSpent/ActiveUser	+0.92%	0.0433
12	D1_TimeSpentPerDay/ActiveUser	+0.91%	0.0644

D1_TradingRev/User is the amount of revenue the company makes from a user's trading the during the first day after a user is exposed to the push notifications feature.

D1_OtherRev/User is the amount of non-trade revenue the company makes from a user during the first day after a user is exposed to the push notifications feature.

D1_Revenue/User is the total revenue the company makes from a user during the first day after a user is exposed to the push notifications feature.

DailySessions/User is the average number of sessions per day after the user is exposed to the push notifications feature.

D14_NetPromoterScore is the average net promoter score from a survey randomly released to users 14 days after their exposure.

D1_Retention is the % of users who are active within 0 and 24 hours of being exposed to the push notifications feature.

D7_Retention is the % of users who are active within 144 and 168 hours of being exposed to the push notifications feature.

D14_Retention is the % of users who are active within 312 and 336 hours of being exposed to the push notifications feature.

D1_TotalTimeSpent/ActiveUser is the average amount of time spent in minutes within 0 and 24 hours of being exposed to the push notifications feature by users we consider active in the app.

D7_TotalTimeSpent/ActiveUser is the average amount of time spent in minutes within 144 and 168 hours of being exposed to the push notifications feature by users we consider active in the app.

D14_TotalTimeSpent/ActiveUser is the average amount of time spent in minutes within 312 and 336 hours of being exposed to the push notifications feature by users we consider active in the app.

D1_TotalTimeSpent/User is the average amount of time spent in minutes within 0 and 24 hours of being exposed to the push notifications feature by all users in the app.

D7_TotalTimeSpent/User is the average amount of time spent in minutes within 144 and 168 hours of being exposed to the push notifications feature by all users in the app.

D14_TotalTimeSpent/User is the average amount of time spent in minutes within 312 and 336 hours of being exposed to the push notifications feature by all users in the app.

Please answer the following questions:

1 Which metrics do you interpret as significant? Please describe your criteria. (2 points)

2 What decision do you think the team should make? Why? (1 point)