Module 11 Assignment: Power Analysis

Part 1

Your colleague is upset because they have just done the analyses for an experiment on a worksite health promotion project and found no difference in the outcome for the people who received the intervention and the control group. When you ask your colleague what level of power the study had, they don't know.

Your assignment is to explain to your colleague the importance of power in designing a research study and how the lack of power in the study may be why no effect was found. develop this argument for your colleague.

Make sure to include:

1. A definition of statistical power.
2. The three things that influence power levels (with definitions).
3. The relationship of statistical power to which type of error (type I or type II?).
4. What an acceptable power level should be.
5. An explanation to your colleague why they may have designed an effective intervention for the worksite and still did not find a significant effect.

Part 2

In the second part of this assignment, you will conduct a power analysis using an online tool to determine how many people a study should include in the treatment and control groups to be able to detect whether a worksite health promotion program is effective at helping employees to lose weight. The tool you will use is a power and sample size calculator that can be found at this link Inference for Means: Comparing Two Independent Samples.

First, let's take a look at an example with a different study:

Example for Part 2

Let's try out the tool on an example. Let's say I am interested in a study examining sleep in adolescents. Let's say group 1 gets a new nighttime routine behavioral intervention which I think will increase sleep for adolescents and group 2 is my control group (adolescents who don't get the intervention). My hypothesis would be that adolescents who get the intervention sleep more hours (but don't oversleep!) compared to adolescents who don't get it. Because I want to generalize from my sample to the population, I am proposing that the population value (mu) of average sleep time for adolescents with the new nighttime routine would be higher than adolescents without the nighttime routine (i.e., they would sleep longer because the nighttime routine is effective).

• mu1 = 8 hours of sleep (mean of population 1, the treatment group)
• mu2 = 7 hours of sleep (mean of population 2, control group)

For sigma, I will say that on average there is about 4 hours of variability in the amount of the adolescents' sleep. As seen above, I am estimating that the population of adolescents who get the intervention sleep 8 hours on average and the population of adolescents (i.e., not just adolescents in my study) without the new intervention only sleep 7 hours on averagewith both groups varying an average of 4 hours of sleep from the groups' means.

• Sigma (the measure of how much variability there is in the groups. A smaller sigma means there is less overlap between the two groups.) = 4 hours

Leaving the analysis as a two-sided test (not making assumptions about which group will sleep more), the = .05, and the desired power (1 - ) = .80, enter the above values into the online tool to determine how many adolescents we need in each group: Inference for Means: Comparing Two Independent Samples

When we enter these values and click calculate, we learn that we need 252 adolescents in both groups. Next, to get a sense of how changes in estimates of mu, sigma, and power affect how many participants we need for the study, we will enter different estimates. For example, maybe the treatment group (mu1) sleeps on average 10 hours a night with the new intervention rather than 8.

Leaving the analysis as a two-sided test (not making assumptions about which group will sleep more), the = .05, and the desired power (1 - ) = .80, enter the new values into the online tool (the treatment group now gets 10 hours of sleep) to determine how many adolescents we need in each group: Inference for Means: Comparing Two Independent Samples

With this larger effect size (3 hours of difference in sleep rather than only 1 hour of difference in sleep), we see that we now only need 28 adolescents in each group to detect a statistically significant difference in the study.

Note: As this is an exercise, we are not concerned with how accurate your estimates are of the population mu or sigma, the goal is for you to see how a power analysis can be conducted and how different expectations of outcomes for your study, can make a big difference in how many participants you need. In the real world, you would use estimates from past research to derive values for mu and sigma.

Your Turn:

A worksite health promotion program wants to know how many employees they need to enroll in a study to determine if their program results in a statistically significant difference in weight loss between the treatment group (employees with obesity who participate in the program) and the control group (employees with obesity who will be on a wait list and will not get to participate in the program in the first year).

We will estimate the average weight loss in the first year for the treatment group to be 20 lbs. (mu1 (1) = 20)

And the average weight loss in the first year for the control group to 10 lbs. (mu2 (2) = 10)

And the standard difference or deviation (sigma / ) (a measure of variability for both groups) to be 10.5 (sigma = 10.5)

Leaving the analysis as a two-sided test (not making assumptions about which group will lose more weight), the = .05, and the desired power (1 - ) = .80, enter the above values into the online tool to determine how many employees we need in each group: Inference for Means: Comparing Two Independent Samples

6. How many employees does the worksite health promotion program needs to enroll in each group?

Next, let's enter a different value for 2, and re-calculate the sample size:

We will estimate the average weight loss in the first year for the treatment group to be 20 lbs. (mu1 (1) = 20)

And the average weight loss in the first year for the control group to be 15 lbs. (mu2 (2) = 15)

And the standard difference or deviation (sigma / ) (a measure of variability for both groups) to be 10.5 (sigma = 10.5)

7. How many employees does the worksite health promotion program needs to enroll in each group?

8. Did the needed sample size increase or decrease? Why? (Please provide a response that a colleague who has never studied power analyses could understand)

Next, let's try a different value for sigma () and leave all other values as is.

We will estimate the average weight loss in the first year for the treatment group to be 20 lbs. (mu1 (1) = 20)

And the average weight loss in the first year for the control group to be 15 lbs. (mu2 (2) = 15)

And the standard difference or deviation (sigma / ) (a measure of variability for both groups) to be 15.5 (sigma = 15.5)

9. How many employees does the worksite health promotion program needs to enroll in each group?

10. Did the needed sample size increase or decrease? Why? (Please provide a response that a colleague who has never studied power analyses could understand)

Finally, let's leave all values the same but change the (alpha) from .05 to .01.

We will estimate the average weight loss in the first year for the treatment group to be 20 lbs. (mu1 (1) = 20)

And the average weight loss in the first year for the control group to be 15 lbs. (mu2 (2) = 15)

And the standard difference or deviation (sigma / ) (a measure of variability for both groups) to be 15.5 (sigma = 15.5)

We will change the level of statistical significance () from .05 (the default) to .01

11. How many employees does the worksite health promotion program needs to enroll in each group?

12. Did the needed sample size increase or decrease? Why? (Please provide a response that a colleague who has never studied power analyses could understand)

***********************************************************************

Question from a previous student:

From the last homework, regarding the website we used to determine sample size: it didn't account for things that would affect sample size, like heterogeneous population, generalizing to the population, correlative vs. descriptive study, or the statistic I would be using. Was this a rudimentary calculation / baseline (minimum) number?

Good question! The factors you bring up are things a researcher should consider when designing their study in some cases even before doing the power analysis. Some tips: I would do power analysis when I am concerned about generalizing to the population (if this is not a concern for my study, I probably would not do one). The sigma estimate that you entered (on variability) was a way to address the heterogeneity of the population (If very little variability = lower sigma - and you should know now, does that make it easier or harder to find an effect? See example below). Finally, that website was set up for you to compare 2 means which is a very rudimentary statistical choice (e.g., can be done with a simple t-test analysis) so it determined the needed sample size based on that analysis. Its sample size calculations should not be used for more sophisticated analyses (more group comparisons mean more sample size required). And whether it is a correlative or descriptive study would influence how you design your study and the statistics you decide to use (and again, that website is most applicable for a simple t-test since it was just comparing 2 means).

Varying levels of variability example

Compare these 3 populations, A, B, C in the chart that follows by examining their frequency polygons (the graph of their distributions) of the 3 populations. The graph is of the samples you drew in your study to represent these 3 populations.

*A and B have the same mu/mean (=9). Your sample to represent population C has a higher mean (the middle point of the graph for C looks to be about 19).

*A is much wider than B, so A has more variability in its population than B.

Thinking back to the sleep study example, let's say population C is the treatment group who received the medication and for this example we have not just 1 but 2 control groups (represented by A and B). Group C is indeed sleeping more hours than the control groups A and B. C is sleeping 19 more hours per week on average but groups A and B only slept about 9 hours more per week on average. The treatment group is doing better (getting more sleep) than the 2 control groups. But...

It will be much easier to statistically "spot" the difference between groups B and C, than the difference between groups A and C. Why? Look back at the chart. See how A and C overlap more than B and C? There is more variability (as measured by sigma) in A and C and since they overlap it will be harder to "separate" them to see that yes C is sleeping more than A. This is because some subjects in group C didn't sleep any longer than some subjects in group A.

So now look at groups B and C. Since there is little overlap due to smaller variability, you can more easily see with your eyes (and identify with statistics) that those 2 groups/populations are not the same. You will be much more likely to spot the significant effect of the medication on sleep.

A larger sigma = larger variability, which is a kind of "noise" in your study making it harder to separate differences between the groups (to see the effect thru the variability/noise). This is often why very specific groups are used in studies (e.g., a lot of inclusion/exclusion criteria) to make the groups very homogenous (low variability), means better ability to spot a "true effect," should one exist.