Comprehensive Doctoral Review Examination Presented in Partial Fulfillment of the Requirements for the Degree Doctor of Philosophy By Sally A. Camacho English as a Second Language School of Education Northcentral University July 3rd, 2016 2 Question 3: Research For question 3, you will need to demonstrate the ability to employ research methods used within your specialization. You are expected to critique research methodologies used by scholarpractitioners and compose responses identifying best practices in educational research. Select five empirical articles from peer-reviewed journals you consider critical to your understanding of your specialization and: 1. Describe each study, including: The research problem, questions, or hypotheses The research purpose The type of design and elements of the design (e.g., sample, data analysis, operationalization of constructs) Threats to validity and if and how addressed The findings and implications Note if the article makes a compelling case for the meaning and significance of the findings 2. Select one of the five articles above. Using the article, assess how you might conduct the research study differently. Address: The research questions The research purpose Type of design and elements of the design (e.g., sample, the type of data you need to collect and how you will collect it, data analysis) The strengths and weaknesses of your envisioned design and methods Quantitative: threats to validity and how your design will address them Quantitative: the constructs you will measure and what you will do in order to determine how to operationalize them (you do not need to identify specific measures) Qualitative: your means of ensuring the quality of your findings Justification for why your chosen design and methods are more appropriate for your research question than alternatives you have considered Your methods of data analysis How the data you collect will enable you to answer your research question and contribute to theory Learning Outcomes 3. Apply relevant theory and research from the specialization coursework to real life situations to solve specific problems and discusses implications. 3 Part I Research Methods used within the Education of English Language Learners and the Challenges they face with High-Stakes Tests The methods considered were based on the education of English Language Learners (ELL) and the challenges they face with high-stakes tests. The five articles explore the changes in the United States educational, standards, accountability measures and regulations during testing. The new educational initiatives challenge content area teachers teaching ELL students to meet the goals set for content assessment during the end of the year standard testing. The ELL population continues to grow and continues to confront disadvantages in linguistic and cultural diversity. Teachers are charged with the difficult task of demonstrating content mastery of all students including non-ELL students and ELL students. School districts must find a way to address standards while supporting the needs of ELL without watering down content. The articles provide much insight on what has been considered for the improvement in the decision making process of accommodations, preparation for testing, and testing. Selected Empirical, Peer Reviewed Articles Article One: Do decision rules matter? A descriptive study of English language proficiency assessment classifications for English language learners and native English speakers in fifth grade The research study's problem is the lack of guiding principles used for classifying and or reclassifying ELL students on language proficiency assessments. The English Language Proficiency (ELP) performance standard is used for accountability allowing for the federal government to report on student academic achievement, but if the students' are not classified correctly, the report will not be accurate. The research questions focus on to what degree the decision rules vary in the total amount of students who may be classified as non-proficient. The second question focuses on to what degree decision rules classify the high-achieving students as 4 non-proficient and the last question centers around to what degree decision rules vary in the total amount of students who may be classified as unqualified for redesignation. The purpose of this research study is to operationalize and test the ELP performance standards and the decision making of the students' classification and eligibility. The procedures began with the administration of ELL students with the English language proficiency assessment (ELPA). A mandated assessment is given annually to ELL students. Another group of non-ELL students were also tested for research purposes only during the scheduled testing window. The design of this study was descriptive focusing on operationalizing and testing English Language Proficiency (ELP) performance standards for student-level decision making. The study collected data from State A Department of Education in the course of the annual testing of formerly identified ELL students, spring of 2010. The sample was chosen from two clusters, K12 ELL students, and K-12 non-ELL students. From the study, one grade level pulled out for analysis, fifth grade. The measurement chosen was the State's English Language Proficiency Assessment (ELPA) that targets listening, speaking, reading and writing and is clustered by grade levels. The second measurement used was the Standards-Based Achievement Assessment (SBAA) which is computer-administered, non-adaptive, and measures academic content: Language Arts, reading, language usage, Math, and Science. The data analysis was based on four decision rules created for non-proficient classifications of ELL students. The four decision rules were as follows; conjunctive geared to all indicators at the performance standards, compensatory geared to below performance standards, mixed allowing for two models to be combined, and complementary which takes into consideration one or two possible performance indicators. For the classification of highachievers to be examined, a different set of rules was set. Researchers hypothesized that the 5 conjunctive model would allow for more non-proficient classifications than of the mixed model. Then, to examine the four decision rules on the students who unqualified for redesignation, researchers would calculate the number and percentage of students were meeting the goals set of ELPA. The research study argues that unless test developers or states under peer-review are in agreement with the U. S. Department of Education and the decision rules and model were chosen then there would be challenges to validity. This particular study meets the criterion for convergent validity because of standards-based achievement performances of the ELL and nonELL students. The findings for the decision rules of non-existent high-achievers proved to be too difficult for the present monolingual students in this study leaving questions about what constitutes non-proficient as opposed to proficient because of the false positive in the conjunctive decision rule. The significance of the study is for future research to ensure a more valid and reliable process with the measurement and assessment systems that would allow for more efficient and equitable testing. Article Two: Classroom Assessment and English Language Learners: Teachers' Accommodations Implementation on Routine Math and Science Tests The article was chosen based on the challenges of ELL students with high-stakes tests. The research explores the problem with the surge of new educational accountability measures and initiatives. The measures brought concerns to content area teachers of ELL students because of the evaluation of the content material taught in classrooms and on high-stakes tests. The problem that teachers face with assessing knowledge but not changing the intended construct of the test is of primary significance. The problem with the policy for how math and science classroom-based assessment practices and accommodations are established are investigated in 6 this mixed study. The recent study strives to answer the following questions: What are the reported accommodations implemented by ELL elementary school teachers on classroom math and science tests? Do teachers report changes in accommodations implementation for ELLs at the different proficiency levels? Do teachers report differently for ELLs with special needs than for other ELLs when they implement accommodations? The methods used will take into consideration the testing accommodations, testing environments, and the criteria used for scoring, to present the effects of the construct-irrelevant variance once the ELL students' linguistic barriers are removed given the same assessment constructs of non-ELL classmates. The participants chosen were from Pennsylvania public schools because of their high rate of ELL student population. K-6 elementary teachers across 10 Pennsylvanian school districts took a survey. Then, ten fourth grade teachers were chosen for classroom observations and another group completed one-on-one interviews. The data collection and analysis was based on the online surveys, semi-structured individual interviews with the ten fourth grade teachers, and the classroom observations. For the quantitative portion of the design, the online adaptive survey of 51 items was analyzed to learn more about the teachers' assessment practices with the ELL students in the areas of math and science. The themes considered were grading and teacher accommodations of ELLs resulting in how teachers feel about classroom math and science assessment. For the qualitative portion of the design a 45 min. Semi-Structured individual interviews of about 15 questions were led by ten fourth grade teachers. The 15 questions were divided into four sessions. The analysis was based on the assessment practices, teacher knowledge, teacher understanding of test and accommodations, and the principles behind the decisions made for ELL students. The data was coded to find the degree to which professional development could help teacher beliefs, student ability to learn, academic language, 7 the role of language development, culture as a part of instruction, and teacher self-efficacy, and teacher modifications in teaching practices. The interviews, coding, and analysis were digitally recorded and transcribed as soon as they were conducted. The findings of the research questions are as follows; for the question, if ELL students are required to take math and science tests, most surveyed teachers always reported giving ELL students in regular math and science classrooms the same test without taking into consideration their proficiency levels. Other teachers, tested beginning proficiency level ELL students test less frequently than the intermediate level ELL students, and another set of teachers exempted beginning proficiency level ELL students from math and science test. For the question, if changes are made to tests or test administrations to assist the ELL students, the answers were, many teachers reported to improve tests depending on ELL students proficiency level. When the question was asked how much additional time is given for ELL students in math and science test, most teachers always reported providing additional time to ELL students. Teacher assistance was also defined by the research resulting in 30% of the teachers provide support to ELL students during tests. Accommodations led to no teachers providing interpreters or translators during classroom-based assessments, bilingual dictionary use was limited or never reported, and or test written in two languages never. The overall findings indicate that ELL students with special needs classroom accommodations implementation is the custom but needs to be the custom for ELL students in general. The implications involve more teacher professional development on assessments, greater teacher awareness of language proficiency levels and the lack of accommodation implementations in classrooms. The significance of the findings indicate that the assumptions states made on classroom accommodations must be re-examined to facilitate ELL students on their performance on high-stakes tests. The study does not mention any threats even 8 though, in the field of language assessment, validity is of concern when it comes to forms of measurement, score interpretation, and when subsequent decisions have to be made based on the assessment. The intent of the test construct was not changed though some of the linguistic barriers were removed to assess the same constructs as the non-ELL students. Article Three: Reading, Writing, and Learning English in an American High School Classroom The research problem in this microenthographic study examines the struggles teachers face with state's standardized curriculum to support multilingual students within an academic school year. Teachers' perception is that standardized curriculum only focuses on standardized tests and not on the individual needs of the multilingual students. Multilingual students in middle schools focus on genres that are directly aligned with the standardized test and are normally shorter passages than what is expected in high school. The research questions targeted in this study focuses on how do high school English language arts classrooms with multilingual learners in reading and writing instruction use the text-based standards-driven curriculum. The second question focuses on teachers' interpretation of the standards-driven curriculum requirements and if multilingual students are supported or limited to opportunities for learning academic reading and writing. The purpose of the study is to examine effects of the educational policies on reading and writing instruction for 9th-grade multilingual learners and how the intended policies determine the ideologies and interpretations of teacher practices. The data included 26 days of field notes and observations through an audio-recorded interview with the teacher, printed data sources from the primary textbook, Holt English 9 curriculum, pacing guide, other resource and supplemental materials used along with the school district's teacher handbook for English 9th grade and benchmark assignments. 9 The data was analyzed by using the micro-ethnographic discourse analysis (MEDA) in three sessions. The analysis alternated between close examination of spoken and written discourse by teachers and students in the classroom and the discourse created through larger events that make up a particular lesson. MEDA identifies multiple levels of analysis. On the other hand, another analysis was conducted through the transcribing of audio recordings and indexing key events in the classroom. Frequency tables demonstrated the much use of the different textbooks, some use, and little use of the textbooks. Threats to validity were not directly stated but because it was a microethnographic study and only one participant was observed the objectivity could be threatened. On the other hand, it is a study that can be replicated on a larger scale, so the assumption is that the threats are minimal. The findings revealed that policy requirements forced multilingual students in reading and writing 9th-grade instruction to focus primarily on a task that prepared students for highstakes tests. The implications of this study are whether or not students learned enough to meet the benchmark assignments to demonstrate then proficiency on the year-end goals of the standardized assessment. The assumption is that skilled teachers may determine how to teach curricular programs to multilingual students. The recommendation for future research is to analyze the new standards and curricular materials and how teachers are using, adapting, and differentiating instruction for multilingual students. Article Four: Formative, Informative, and Summative Assessment: The Relationship Among Curriculum-Based Measurement of Reading, Classroom Engagement, and Reading Performance In Quebec, classroom teachers are required to administer and correct high school exit exams. The high school exit exams are developed by the New Ministry of Education, Leisure and Sports (MELS) program. Classroom teachers are also put in the position of participating in the 10 development and validation process of aligning classroom-based assessments along with a writing exam. The problem is classroom teachers, and the large-scale of test developers confront external standards and requirement issues of validity. For teachers to develop classroom-based assessments, teachers must know their students and focus the assessment on what has been taught in the classroom. Working with a large-scale test development and with the classroom context to consider, the factors caused uncontrolled variables to affect the outcomes of the assessments. The purpose of this research study is to align context and pedagogy to high-stakes English as a Second Language testing. The questions considered refer to how the rater training affect the teachers' use of the marking rubric and how the teachers participation in the rater training perceive its benefits to teaching, learning, and assessing students in the classroom. In Quebec, 50% of the total grade is taken from the exit exams, and the other 50% is taken from the classroom-based assessments. The study employed a convergent mixed method design to examine how secondary teachers' rate training facilitated the alignment of the MELS-expected standards. The data was collected through four phases. The first phase was marking and recording students' scores. The second phase was sharing students' papers. In phase three, teacher rater training took place at the same time the quantitative and qualitative data was collected. During phase four, the rater training on scoring behaviors of both testing periods took center stage. The results for the first question revealed rater behavior across the year 2010 and 2011 led to changes in the rubric. Descriptive statistics and correlations were used by looking at means and standard deviations for the correlations between the variables of the engagement-student report, the engagement-teacher report, reading competence, and reading performance. A Chi-square analysis and Factor Analysis 11 M-Plus was conducted to evaluate if the missing data differed among demographic characteristics. The results for the second question indicate that the classroom teachers training had a positive impact in their classrooms and their understanding of writing exams and rubric expectations improved. The findings offered a clear perspective on the importance of communication and cooperation among test developers, teacher raters and classroom teachers in understanding the expectations of the new writing assessments and rubrics. The significance of the study was to help improve the rigor, alignment, and qualities of the rubric descriptors. The underlying principles allowed for test validity, benefiting teachers and students in teaching and learning. Article Five: The Impact of Communication and Collaboration Between Test Developers and Teachers on a High-Stakes ESL Exam: Aligning External Assessment and Classroom Practices The research on formative assessment and student reading achievement has been investigated for much time now, with the difference that this study examines the problem of curriculum-based measurement and reading (CBM-R) associated with social, behavioral, and or emotional aspects because it has not been fully explored. The purpose is to build a relationship between CBM-R, student engagement, reading competence, and performance in the general education classrooms of third to fifth-grade students. The research questions focus on which formative assessments of reading facilitates learning through engagement and which tools can assist in the decision making of appropriate instruction and intervention. Teachers and students were selected from six schools in high-risk communities, and four were Title I schools. The students were given parental consent forms to participate. The data analysis plan only included information for the students' whose teachers contributed to the 12 survey from the fall and spring data. The instrumentation chosen was teacher report of student engagement, student report of engagement, reading competence using CBM-R measurements, and reading performance from state criterion referenced tests. The data was analyzed using several descriptive statistics to examine central tendencies, variability, and distributions of the study variables. A range of multivariate analyses was conducted to examine the effects of topic choice on writing task scores, the Rasch analyses were performed using the FACETS program, and the MELS was used to evaluate the ratings and the content mainly involving the coding and theme development. The results indicate that students with lower reading competence who were engaged during reading assessments performed better on the summative assessments. On the contrary, students with higher reading ability were not influenced by the engagement. The findings reveal motivation as an essential component of student reading achievement in CBM-R assessments. Its limitations, however are, based on the threats of validity found in the technical qualities of the CBM-R measurement equivalence related to the different subgroups, left unanswered questions. The implications and further research support a growing need for the literature of CBM-R for school use and how it relates to extra-academic factors for student achievement. The study's significance is the continued investigation of whether or how engagement can be adjusted or influenced as an indicator of recent or distal performance. 13 Part II Article Five: The Impact of Communication and Collaboration Between Test Developers and Teachers on a High-Stakes ESL Exam: Aligning External Assessment and Classroom Practices Research Redefined The redefined research study considered is based on the need of English Language Learners in St. Croix, United States Virgin Islands (USVI) taking and meeting the standards on high-stakes tests. The impact of communication and collaboration among all test developers and teachers alike may assist with the alignment of assessments. Revised Research Questions The study would ask the following research questions concerning the need to link context pedagogy in lesson planning and design in preparation for ESL students taking high-stakes tests. Question 1. What literacy assignments did the ESL and content area teachers use in the classroom, and how were they linked to the standard-based assessments? Question 2. What type of literacy test preparation materials did the ESL and content area teachers choose, adjust, or construct for classroom use in order to prepare ESL students for the test? Question 3. What were the benefits of using the literacy test preparation materials to the teaching, learning, and assessments process of the ESL and content area classrooms? Revised Research Purpose The revised purpose focused around secondary school ESL and content area teachers' alignment of classroom-based literacy assessments and materials and high-stakes tests. The study is as a result of concerns with ESL students not meeting the standards set forth by high-stakes tests. The purpose is to examine how teacher training on the alignment of classroom-based assessments and materials and high-stakes tests. Revised Research Design 14 The USVI public schools, where this revised research study would possibly to take place, has an increase of Hispanics/ESL population. Many of the immigrants are coming from the Dominican Republic. There is a need to robust the ESL program and materials for the ESL students to meet the standards set forth by the high-stakes tests at the end of the year. The schools included in this revised study are the public St. Croix schools with a high ESL population, schools with free and reduce lunch, and schools that did not attain reached yearly progress standard goals. Surveys would be given to school administrators to determine eligibility. The data will be analyzed using descriptive statistics to examine the constructs of external assessment development contexts, ELL test-takers, and classroom context through the measures of spread. They will be measured through questions, survey, interview, observations and pre and post testing data. To determine how to operationalize the constructs, I will define variables into measurable factors and develop specific procedures. The research design chosen is a mixed method design because it will best answer the research questions. The quantitative and qualitative data synthesis would take place at the end of the data analysis process after all testing and training have been completed. For the qualitative portion of the study, the St. Croix, school district would serve as a critical site for data collection. In order to ensure the quality of the findings, the qualitative and quantitative data will be conducted simultaneously keeping with the strengths of each method and having in mind that the quantitative data (observations) might lack inconsistency while the qualitative data may develop the content. The students involved are 7th and 8th grade ESL middle school students who take English Language Arts (ELA) portion of the end of the year high-stakes test. The students would have had to be at least a full year in a United States Public School and receiving ESL services. The teachers who would meet the criteria to participate: 1) would have taught and tested by using 15 the current measurement application 2) would have ESL students in their classrooms that they service. 3) would be willing to work closely with the researcher identifying materials, lesson planning, and participating in a training aligning classroom-based assessments and materials to the end-of-the-year assessment. The teachers will be a part of the development and validation process of the classroom content, assessment, and curriculum alignment. Teacher training will be held by district personnel, ELL coordinator, and other available school and district personnel. Focus groups will be established and workshops on curriculum alignment and assessment. The teachers will participate in classroom observations. The administrators and teachers will be involved in interviews and administrators and teachers will take part in training for the purpose of aligning school resources to the assessments. The qualitative data will be analyzed through several descriptive statistics. The transcripts of the semi-structured interview of the 7th and 8th grade ESL and content area teachers, ???eld notes, observations and teacher training results will allow for coding and for the process of thematic content analysis to occur. The quantitative data will be analyzed by focusing on the online survey and the five areas addressed. The data will be organized into tables to identify the patterns, mean, standard deviation, key points of the study, and statistical commentary. Strengths/Weaknesses The strengths are that the training may empower teachers to guide their students learning because they can align the classroom assessments and materials to the end of the year assessments. The strengths of the design are based on the ability to explore a range of questions related to the research. The corroboration of quantitative data and qualitative data allows for triangulation permitting for a stronger research because each method's limitations annul each other and allow the researcher to decide on the needs and focus. The administration would feel a 16 sense of what the teachers' needs are to meet the students' needs and develop training around the critical components for teaching, learning, and assessments. Administrators and teachers will be able to align curriculum with the requirements of the assessments enhancing the ESL program which in turn will validate the assessment process in the classroom. The weaknesses are that some of the details might not have been fully explored leading to problems in the analysis of the results. Threats to Validity The threats to validity would be considered only if there was a logical argument for the forms of measurement scores and interpretations (Clark-Gareca, 2016). Classroom teachers are faced with challenges because while trying to assess the academic knowledge of ELL students; they must make sure they do not change the test construct making sure ELL students follow the test but receive the much-needed accommodations (Clark-Gareca, 2016). Justification The research design and method was considered because it best supports the research questions. Having a mixed method design allows for the combination of qualitative and quantitative data. The method permits for the qualitative data to clarify the process that the quantitative findings may have identified throughout the study. The method also helps justify the claims that secondary school ESL and content area teachers' who are trained in the alignment of classroom-based literacy assessments and materials and high-stakes tests may (a) improve their instructional practices, (b) be more efficient in choosing instructional materials that are aligned with the assessments, and (c) may be able to identify earlier student needs and focus on individualized interventions. Data Collection 17 The data collection would consist of a quantitative portion, an online survey developed to inquire about: 1) teachers assessment practices with ELL students 2) teacher assessment accommodation practices with ELL students 3) teacher educational background in reference to working with ELL students 4) school and teacher professional development and course work with ELL methodologies as a focus 5) school and teacher focus for meeting the standards on high-stakes tests with ELL students in mind. Then, for the qualitative portion of the study, a semi-structured interview would be conducted with 7th and 8th grade ELL and content area teachers who currently serve ELL students in their classrooms. The interviews would be focused on teachers' assessment practices. It would target teachers' understanding of assessments and accommodations and their decision-making the process of what and how to teach ELL students in preparation for a high-stakes test and teacher trainings. The data will enable me to answer the research questions by gaining meaning from the themes and narratives through coding, categorizing, and concepts. Conclusion In conclusion, this redesigned study's contribution to theory would be for the schools and districts to examine their assumptions of testing guidelines and accommodations especially for the ELL population and the classroom context to assist with the preparation of high-stakes test and the end of the year performance. This redesigned research study focused on the interest of ELL students facing challenges with the high-stakes test. Districts, schools, administrators and all those involved are concerned with ELL students not meeting the standards and goals outlined in the tests. The redesigned study targets the concerns by allowing for secondary school ESL and content area teachers to align classroom-based literacy assessments and materials and highstakes tests to improve instructional practices and testing outcomes. 18 References Carroll, P. E., & Bailey, A. L. (2016). Do decision rules matter? A descriptive study of english language proficiency assessment classifications for English-language learners and native English speakers in fifth grade. Language Testing, 33(1), 23-52. doi:10.1177/0265532215576380 Clark-Gareca, B. (2016). Classroom assessment and english language learners: Teachers' accommodations implementation on routine math and science tests. Teaching and Teacher Education, 54, 139-148. doi:10.1016/j.tate.2015.11.003 Gilliland, B. (2015). Reading, writing, and learning english in an american high school classroom. Reading in a Foreign Language, 27(2), 272-293. Glasswell, K., Singh, P., & McNaughton, S. (2016). Partners in design: Co-inquiry for quality teaching in disadvantaged schools. Australian Journal Of Language & Literacy, 39(1), 20-29. Marchand, G. C., & Furrer, C. J. (2014). Formative, informative, and summative assessment: The relationship among curriculum-based measurement of reading, classroom engagement, and reading performance. Psychology in the Schools, 51(7), 659-676. doi:10.1002/pits.21779 Tan, M., & Turner, C. E. (2015). The impact of communication and collaboration between test developers and teachers on a high-stakes esl Exam: Aligning external assessment and classroom practices. Language Assessment Quarterly, 12(1), 29-49