4. Estimation

4.1. Treatment effects

Given the random assignment, we estimate the effects of the pure

math and combination program with the following model:

yis = + 1Combois +2Mathis + Xis + s +is, (1)

where yis is the outcome of interest of student, i, in randomization block

(center by language), s. The outcomes are the overall mean math score

as well as the scores for the different assessment components (i.e. rote

counting, sorting, etc.). We standardize all scores to have a mean of zero

and a standard deviation of one. Xis is a vector of student and parent

characteristics. This vector includes student age, parent age, household

income, hours worked, fall DRDP scores, mean parent responses to

Economics of Education Review 88 (2022) 102262

parenting questions, indicators for student gender, race/ethnicity,

parent education, and parent participation in texting program in previous

years, and indicators for missing information.7 These covariates

are listed in the randomization checks found below in Tables 3a and 3b.

s is a vector of randomization block fixed effects and is is a student level

error term. Combois and Mathis indicate that the parent received the

combination or pure math text messaging program, respectively. The

omitted category is the control group. The estimates of 1 and 2 can be

interpreted as the causal average treatment effects of the combination

and pure math treatment in comparison to the control group. In all regressions,

we cluster standard errors on the randomization block level.

Appendix Table B.1 presents how the sample, centers, and randomization

blocks are distributed across treatment arms and by child gender.

We also probe for heterogeneity of results by gender and skill level.8

To estimate effects by gender, we fit Eq. (1) separately on the subgroup

of boys and girls. Within each subgroup we estimate the effect of the

combination and pure math programs relative to the control group. We

then test whether the estimated effect of each program is equal for boys

and girls. Unfortunately, children were not assessed in math at the

beginning of the school year, and therefore we cannot assess effect

heterogeneity with respect to baseline skills in math.9 However, we can

investigate how the effects for girls and boys are concentrated along the

outcome distribution.

To that end, we estimate quantile regressions for girls and boys

separately. Quantile regressions estimate the effect of the program for

each quantile of the outcome distribution specified. We use models with

and without the baseline covariates in Eq. (1). One challenge in

comparing quantile effects between girls and boys is that the distributions

of math assessment scores differ for girls and boys. A given quantile

of the girls' distribution may not correspond to the same quantile of the

boys' distribution. To address this challenge, we follow Bitler et al.

(2014) and Strittmatter (2019) and calculate translated quantile effects.

Translated quantile effects assign the quantile effects for girls and boys

to the same absolute scale of a reference distribution. In this vein, we

first calculate the effect at the original quantile for girls and boys. We

then find the quantile in the reference distribution that corresponds to

the math score at the original quantile and record the quantile effect at

the reference quantile. We chose the math score distribution of the

control group as the reference distribution.

4.2. Randomization checks

Causal identification of the program effects relies on successful

randomization. That is, the two treatment groups and the control group

do not systematically differ in observed and unobserved characteristics

other than being assigned to one of our text messaging programs. To

partially test this assumption, we assess covariate balance across treatment

arms with the following fixed effects regression model:

Xis = + 1Combois + 2Mathis + s +is (2)

7 Binary covariates with missing information were set to zero and non-binary

covariates with missing information were imputed with the district mean.

Additionally, we included dummy variables to account for this imputation.

8 This experiment was pre-registered at Open Science Framework, which

included both our main analysis and heterogeneity analyses by gender, race/

ethnicity, and skill level. The full pre-registered analysis plan can be accessed at

https://osf.io/w6qhr/.

9 Though the DRDP can be used as a measure of baseline math skills, the

shortcomings of this assessment do not make it an ideal measure. The assessment

asks teachers to rate students on 43 skills via observation, not one-one-

assessments. As discussed in the Potential Teacher Role in Gender Differences

section, teacher perception of skills and third-party assessment of skills do not

always align. Further, correlations between domains of the DRDP indicate that

teachers do not effectively distinguish between child development domains.

Correlations available upon request.

6C. Doss et al.

Table 4

Attrition Balance.

(1) (2)

Combination Pure Math

Not Assessed 0.007 -0.032

(0.024) (0.023)

Notes: All models include randomization block fixed effects and a covariates

detailed in Tables 3a and 3b. Standard errors are clustered at the randomization

block level. N =1,842. * indicates p <0.05.

Table 5

Effect of Combination and Pure Math Program on Overall Math Achievement.

Mean Math Score

Combination Pure Math p-Value (Pure Math vs

Combination)

N

All Students 0.000 -0.034 0.569 1336

(0.058) (0.056)

Girls 0.156 + 0.015 0.16 661

(0.083) (0.099)

Boys -0.115 -0.025 0.324 675

(0.105) (0.094)

.0101 .07179

p-value (Girls

vs. Boys)

Notes: Mean math score is the average of the standardized subscores. The

average is standardized to have mean zero and standard deviation one. All

regression models include randomization block fixed effects and a full set of

covariates. Standard errors are clustered on the randomization block level. +

indicates p<0.1.

The coefficients of interest are 1 and 2, which represent an estimate

whether the covariate of interest, Xis, is statistically significant between

the control group and the combination program group and pure math

program group, respectively. If randomization was successful most coefficients

should be quantitatively small and statistically insignificant.

Tables 3a and 3b show that the groups that received the combination

and math programs do not significantly differ from the control group in

any of the observed student and parent characteristics. Merely one F-test

of the joint significance of both programs (out of 32 tests) is significant

at the five percent level. Appendix Tables B.2 through B.5 present covariate

balance by gender and once again show that the rate of statistically

significant balance is what one would expect by chance. As such,

these results provide evidence that randomization was successful and

that there are no meaningful differences in observed characteristics.

However, our preferred model includes these covariates to increase

precision.

4.3. Attrition

Causal identification could also be jeopardized if there was differential

attrition among the three arms of the experiment. Of the 1,842

children recruited into the experiment and randomized to groups, we

were unable to assess 452 children due to child absences,10 33 children

because the teacher indicated that they did not want assessors to test

that particular child (e.g., if they were special education), 14 children

because the assessor ran out of time before assessing all children or

forgot a child, and seven children because of their behavior (e.g., child

could not sit still) or because the assessor did not speak the child's

language.

It is unlikely that the text messaging program led students to be

absent during the assessment and thus influenced who is part of our

10 Unfortunately, our data does not allow us to distinguish whether students

had left the district or were simply absent on the date of the assessment.

Economics of Education Review 88 (2022) 102262

estimation sample. However, if that had been the case and these additional

students differed on average from students in the control group,

the estimated effects of Eq. (1) would be biased. Therefore, to assess

selective attrition, we estimate the following fixed effects model:

Ais = + 1Combois + 2Mathis + Xis + s +is, (3)

where Ais is an indicator for attrition; it takes the value one when a

student was not assessed and is therefore not included in the estimation

sample and zero otherwise. The coefficients of interest are once again 1

and 2 which now represent an estimate whether the probability of

attrition from the sample is statistically significant between the control

group and the combination program group and pure math program

group, respectively. If the program did not affect who is in our final

sample then, once again, each coefficient should be quantitatively small

and statistically insignificant. Table 4 shows these coefficients. Neither

the combination nor the pure math program led to significantly more

sample attrition than the control group. Appendix Table B.6 shows no

significant differential attrition by program type in separate boys and

girls subsamples

What was studied?What is the hypothesis? Who was the study done on?(Who are the participants?) How was the study done? Methods? Procedure? What was found? What were the results? What are the implications/limitations of the study? Discussions? Weakness of the study