Study conducted by: CML Insight
Released: February 9, 2024
Introduction
In today’s world, where literacy serves as a cornerstone for academic success, Kids Read Now is at the forefront, pioneering initiatives to propel young minds toward a future rich in knowledge and achievement. This whitepaper shows the results of the comprehensive, multi-year data analysis of students enrolled in the Kids Read Now program, a non-profit organization on a mission to eliminate the summer reading slide and foster a love for reading in K-5 students.
Background
Kids Read Now serves school districts in 32 states, igniting a transformative approach to both summer and year-round reading programs. At the heart of this endeavor lies a commitment to empower students through student choice and parental involvement, which have been proven to yield significant improvements in reading skills, positioning Kids Read Now as a low-cost effective alternative to intensive tutoring to recover Pandemic learning losses.
The present study builds upon a body of research conducted on student outcomes from participation in the Kids Read Now summer reading program. This analysis is uniquely valuable in two ways. First, its scope. Where previous studies have looked at outcomes within a single program year, this study has expanded its lens to include 3 years’ worth of program effects. Not only was this intended to form a more solid foundation for findings of student outcomes, it also sought to examine 3 distinct periods of student learning: before, during, and after the COVID-19 pandemic and associated school interruptions.
The second distinguishing feature of this analysis was its methodology. The study was conducted by independent analytics firm CML Insight using their state-of-the-art Causal AI tool, which has shown immense potential in performing retrospective studies to measure and identify effective educational interventions. The CML Insight platform stands out for its ability to identify and analyze which interventions will improve outcomes and to what degree by harnessing AI and machine learning for cause-and-effect analysis. It not only performs propensity matching across large groups but also serves as a causal & predictive AI tool.
| School Year | Classification |
|---|---|
| 2018-2019 | Pre-Pandemic |
| 2020-2021 | During Pandemic |
| 2021-2022 | Post-Pandemic |
Methodology
Data and Matching
In the initial phase of the study, KRN-provided student data served as the primary dataset, capturing reading score data for nearly 7,000 Ohio and Michigan students whose schools participated in the summer reading program, demographic information, and program engagement data.
In the initial phase of the study, KRN-provided student data served as the primary dataset, capturing reading score data for nearly 7,000 Ohio and Michigan students whose schools participated in the summer reading program, school level demographic information, and program engagement data. 3 years of data were provided: 2018-19, 2020-21, and 2021-22, serving as pre-, during, and post-COVID samples respectively. To ensure accurate integration of further datasets, the Stanford Education Data Archive (SEDA) was used to obtain National Center for Education Statistics (NCES) Identification numbers for participating schools.
Additionally, various data sources from Stanford Data Education Archive, EducationData.Urban.Org, Ohio’s report card Website, and Michigan’s school performance data from mischooldata.org were integrated to acquire information on school covariates, offering valuable contextual variables for our analysis. State-level school data provided various performance columns categorized by proficiency bands, allowing us to conduct a comprehensive assessment of school-level outcomes.
Integration involved matching NCES IDs obtained from SEDA with state-level data, facilitating a seamless connection between individual students and corresponding state-level outcomes. Extensive data stitching was performed to consolidate information from multiple sources into a unified dataset.
Modeling and Analysis
In the modeling and analysis phase, KRN schools were systematically matched to non-participating control schools at the school level. Pertinent school covariates from the Education Data Portals were selected to control for potential confounding factors, including poverty indicator using free or reduced price lunch enrollment, student/teacher ratio and test type. Propensity score models were constructed to estimate the likelihood of KRN participation, along with prognostic scores for school’s reading performance, enabling the creation of matched control groups in the propensity-prognostic score space for doubly-robust matching.
Hard matching was implemented to pair KRN-participating schools with non-participating control schools at the school level, followed by soft matching using prognostic and propensity scores, which were computed based on key school characteristics highly predictive of outcome variables. Two-dimensional matching was employed, considering multiple dimensions of similarity, and drill-down segments were conducted to create segment-level impact results whenever possible. To enhance the robustness of the causal effect estimates, matched samples were validated using machine learning bootstrap techniques.
Results
The analysis showed a statistically significant 0.2 SD increase in third-grade percentile reading scores for school test results the following spring after the summer reading intervention among participating schools compared to propensity-matched control schools in the 2018-19 academic year. This translates to an approximate 7% percentile ranking increase1. This effect was significantly higher in high-poverty schools, with an approximate 11.4% percentile ranking gain, highlighting the program’s essential role in reducing educational gaps along economic lines. These results align with prior studies of KRN student outcomes pre-COVID2.
Results from the latter cohorts were more variable. For 2020-21 no statistically significant effect was found, while 2021-22 results saw a nearly significant (p = 0.06), albeit reduced positive effect of participation in the program. These results align with larger social patterns of major disruption in young students’ learning as a result of the COVID-19 pandemic, a disruption which has not been limited to one academic year. And though not statistically significant, year-over-year changes in the results for the during and post-COVID cohorts shows dramatic improvement and a return to previous years’ well-documented reading score gains.
Discussion
Incorporating the current study into the years of research documenting the program’s effectiveness, Kids Read Now presents a unique solution to summer learning loss that is in the process of returning to historical levels of positive student outcomes post-COVID.
This is the 7th study that has independently evaluated the Kids Read Now literacy program by researchers from three different Universities and two nationally renowned research firms, including The University of Dayton, Arizona State University, University of Wisconsin, Soothsayer Analytics, and now CML Insight. All seven studies showed similar results: significant and substantial reading score improvements with effect sizes of .12-.19, which, depending on grade level, convert to 2-4 months over summer for an average 7% improvement, with the best results in high-poverty schools.
While there is no single intervention capable of overcoming all the obstacles K – 5 students face in their early literacy journeys, the Kids Read Now summer reading program fills a unique niche in its cost-effectiveness in targeting systemically disadvantaged students. Compared to traditional summer schools, which cost $1,500 to $3,000 per student yet only serve 10-20% of students in need of services, KRN’s program serves all students in grades K-5 at a cost around $55 per student. Further, the KRN program is often funded entirely with ESSER or Title funding and requires minimal involvement from the school to administer. The low-cost and turnkey nature of the program, along with its proven efficacy, support Kids Read Now as an effective option for schools to address Pandemic learning losses and the ubiquitous summer reading slide.
About Kids Read Now
Kids Read Now is a nonprofit with a singular mission: to accelerate reading gains for K-5 students. Kids Read Now has partnered with school districts in 40 states to implement and manage summer and year-round reading programs that incorporate student choice and parental involvement, which has been shown to improve reading skills significantly.
About CML Insight
CML Insight is pioneering causal analytics for education and multiple industries by identifying cause and effect and measuring the impact of interventions. A SaaS application platform that goes beyond predictions and quickly provides transparent causal insights to suggest specific actions to improve organizational outcomes and KPIs directly.
Works Cited
- Paul T. von Hippel, “Multiple by 37: A Surprisingly Accurate Rule of Thumb for Converting Effect Sizes from Standard Deviations to Percentile Points,” Annenberg Brown University, EdWorkingPaper No. 23-829, September 2023, https://doi.org/10.26300/xk0b-ft25
- Geoffrey D. Borman, Hyunwoo Yang & Xin Xie (2021) A Quasi-Experimental Study of the Impacts of the Kids Read Now Summer Reading Program, Journal of Education for Students Placed at Risk (JESPAR), 26:4, 316-336, DOI: 10.1080/10824669.2020.1855985