Practice Problems: Statistical Significance in Impact Results Table

Edited excerpt from executive summary ofReport:

This report examines the impact of supportive housing on housing stability and shelter use; interactions with the criminal justice system, including arrests and returns to jail; and... People eligible for the supportive housing program were randomly assigned to one of two groups; individuals in the treatment group were offered supportive housing services, while individuals in the control group received usual care services in the community.

Below is an edited excerpt of a results table from the report.

Estimated Impacts on the Number of Arrests in Each Category

	(1)	(2)	(3)	(4)
Outcome: Arrest type	Mean for Treatment Group (Supportive Housing)	Mean for Control Group (Usual Services)	Difference (Estimated Impact)	Statistical significance
Drugs/narcotics violations	0.71	1.33		***
Aggravated Assault	0.13	0.14
Larceny	0.25	0.45		**

Notes: * p < 0.10 | ** p < 0.05. | *** p < 0.01

Arrests are measured as the number of arrests of a study subject.

1. For each outcome, calculate and fill in the difference column (3). Hint: To estimate an impact, what mean should you subtract from the other?

2. State each estimated impact in words. That would take the form of "the impact of [independent variable/program/X] on [dependent variable/outcome/Y] is [number] [units]."

3. Using the statistical significance stars (column 4) and the notes explaining the meaning of the stars, state the level of statistical significance (or lack of statistical significance) for each estimated impact. Identify the impacts statistically significant at the 5% level.

4. Assess the practical significance of each statistically significant impact.