Effective as the Data is Long: RBBB achieves ESSA certification
And what in the world does that mean?
Written by RBBB Advocate Wendy White featuring Dawn Gerundo, Melissa Kovacs, and Sophie Allen-Etchart
ESSA is an acronym for many things: East of Scotland Squash Association, the Entomological Society of Southern Africa, and the Empire State Subcontractors Association—just to name a few. But for Read Better Be Better, ESSA stands for the 2015 Every Student Succeeds Act that guides K-12 education policy in the United States.
In Arizona, the Arizona Department of Education’s Move On When Reading policy requires school districts utilize core reading programs and reading intervention programs that meet ESSA evidence level requirements as part of their district-wide literacy plans to improve student reading proficiency. This year, RBBB submitted its program to the Arizona Department of Education to be evaluated for ESSA certification.
The journey to this stage was six years in the making; evidence-building takes time, as well as support, to conduct the necessary evaluations. From the outset, RBBB was fortunate to receive funding from the Valley of the Sun United Way for ongoing program assessments.
The partnership started in 2016 when Dawn Gerundo, Valley of the Sun United Way Community Development and Engagement Director, first met RBBB’s CEO and Founder Sophie Allen-Etchart to discuss areas of collaboration between the two organizations. United Way and RBBB are both committed to increasing 3rd-grade reading proficiency as well as improving youth engagement in education and employment opportunities.
At that meeting, Dawn recognized the potential of RBBB’s fledgling literacy program that pairs 3rd graders with middle school partners to improve reading comprehension. She believed that RBBB had a strong program that benefitted not just the Readers but the Leaders as well.
“By then, RBBB had completed its pilot project at Whittier Elementary School and was operating after-school sessions at several schools in the Phoenix and Avondale school districts,” Dawn says. “I was familiar with the positive evidence around cross-age peer tutoring formats like RBBB’s program. So, I suggested that Sophie gather impact data on both sets of kids—improvements in reading for the 3rd graders and growth in the resiliency and leadership skills for the middle schoolers.”
The initial grant from the Valley of the Sun United Way originated through funding from Red Nose Day USA that supports educational intervention programs with the goal of giving every child a chance to succeed. The funding went toward RBBB’s first independent evaluation by Melissa Kovacs of FirstEval. The evaluation used data collected from RBBB’s Fall 2015 programming through the Read On Avondale project at Lattie Coor and Michael Anderson elementary schools in the Avondale Elementary School District as well as Edison, Phoenix, and Whittier elementary schools in the Phoenix Elementary School District.
FirstEval analyzed data from educational instruments, such as standardized reading assessment tests, along with pre- and post-program teacher and student surveys conducted by RBBB. While the sample size was small, FirstEval found, “The RBBB program has notable effects on its participants’ literacy skills … Also importantly, 3rd-grade participants show significant improvement in reading according to the Dibels DAZE, Dibels ORF, and AIMSweb.”
Additionally, the reading proficiency of the middle school students improved as well. Their participation in the program “has significant effects on their own literacy skills as evidenced by the Galileo assessment, and strong effects on their feelings of social and personal responsibility,” the evaluation summary concluded.
“This research allowed Sophie to approach other school districts and funders in order to expand the program’s reach to more students,” Dawn says. “She let the data tell the story of RBBB to prove her case about the effectiveness of the program.”
During the next three years, RBBB’s program grew to 19 additional school sites in four new districts plus several Boys & Girls Club branches, with funding from numerous partners including a collaborative grant with United Way and the Helios Education Foundation. A portion of the grant was dedicated to ongoing data collection and analysis. At the beginning and end of each semester, students and teachers were surveyed and data from standardized reading tests were gathered and then analyzed, a practice that continues to this day.
In 2018, another independent evaluation was conducted by Melissa Kovacs of FirstEval, which found that RBBB “truly aims to measure its program participant progress through many angles. Overall, the RBBB program is significantly improving reading for its 3rd-grade participants, as evidenced by numerous assessment instruments.”
The 3rd graders who participated in RBBB consistently scored higher than their non-participating peers on standardized reading assessments as well as on teacher surveys that measured students’ focus, the enjoyment of reading, and reading comprehension.
“Once the founding board had sufficient evidence to prove a promising practice, our sole intent was to expand the reach to as many students as possible, while maintaining quality and continuing to evaluate at every step,” Sophie explained. “This second evaluation proved we had done just that—it was such a proud moment … and quite frankly a relief! You know how hard you have worked to commit to excellence and mitigate those risks of exponential growth, but for those of us with a healthy dose of self-doubt, you never know for sure you have succeeded until you see it firsthand—first (and most importantly) in the classrooms and then in the numbers.”
Since 2018, RBBB has grown to serve 75 schools in 11 districts and, because of the pandemic, has also begun collaborations with community-based organization partners to reach even more children to prevent reading regression. United Way has helped with this effort as well by providing funding during Summer 2020 to create Literacy Kits so that a simplified version of RBBB’s curriculum could be implemented anywhere and by anyone. RBBB has distributed over 10,000 kits to more than 100 organizations and schools since last year.
Additionally, Valley of the Sun United Way’s grant also helped transition RBBB’s sessions from schools to a home-based version that paired a 2nd through 4th-grade student with an older family member in the home for the Fall 2020 and Spring 2021 semesters. More than 500 students participated in RBBB’s At Home program during 2020 and 2021.
“RBBB was open and willing to take risks and explore different directions during the pandemic,” Dawn explains. “The nimbleness of the program speaks to the level of quality and evidence for what the organization says it’s doing.”
When it came time for the every-three-year independent evaluation in 2021, only data prior to the pandemic was analyzed. However, the next scheduled evaluation will include data from during and after the pandemic. This distinction will allow RBBB to provide important insights as to how the pandemic has impacted program participants.
The 2021 evaluation found that, “The first report, provided in 2016, showed RBBB effectiveness but had small sample sizes, as the program was still small. The 2018 report and this report benefit from larger sample sizes as more and more schools have joined the program. The 2018 report, and this one, also show RBBB effectiveness but are bolstered by increasing sample sizes as more and more schools are added to the program. This strengthens the findings that the program is having an impact on 3rd-grade readers.”
RBBB’s three independent evaluations, as well as its curriculum that aligns with the 11 Arizona state standards for 3rd grade English Language Arts, were provided to the Arizona Department of Education’s Move On When Reading team that reviewed the efficacy and research for the program.
In May 2021, RBBB was notified that it had been approved as an ESSA-certified and Tier-4 evidence-based program.
Melissa Kovacs, RBBB’s longtime evaluator, applauds this achievement as a Tier-4 evidence-based program. “RBBB has shown year after year, over multiple sites, that student participants have larger reading gains than non-participant students. This consistent evidence of program effectiveness speaks to the strength of RBBB’s impact on reading.”
“While this is just the beginning of our journey to evidence RBBB’s programming, it is a strong start. Now we will continue to follow the guidelines from Arizona Department of Education to move towards higher-tiers of evidence—but we are so proud of how far we have come in such a relatively short time,” Sophie shared. “And that’s a celebration that we want to share with our community who have supported us at every step.”
