When schools buy new reading programs, they look to EdReports. But some of its reviews don’t line up with science.
Even though EdReports is only a decade old, it has quickly become a powerful force in the educational publishing industry. Many schools rely on its reviews when they decide which reading programs to buy. But the nonprofit organization has given high marks to programs that use strategies debunked by cognitive science. And it hasn't endorsed other programs despite studies showing they work.
Elementary schools are racing to replace their reading curricula as part of a national movement to implement teaching methods based in cognitive science. But figuring out which materials measure up is complicated and time-consuming. So, state education departments have been turning to EdReports, a nonprofit organization that calls itself a “Consumer Reports” for education.
EdReports says that it has “always” checked for the science of reading in its reviews of instructional materials. But an investigation by APM Reports has found that its track record in that area is uneven. In the decade since it launched, EdReports has given high marks to several programs that use discredited techniques to teach reading. At the same time, it hasn’t given its seal of approval to other programs that have demonstrated their effectiveness through rigorous studies.
The movement to implement the science of reading was sparked in part by the APM Reports podcast Sold a Story, launched in 2022, and a series of related audio documentaries that preceded it. The reporting focused attention on how a faulty idea about learning to read became ingrained in many elementary schools.
In the years since the podcast premiered, at least 25 states passed new laws about reading instruction, many of them aimed at tightening control over curriculum. In the process, they increased EdReports’ already substantial clout.
Kentucky’s law forced schools to ditch a research-tested program in favor of those with positive reviews from EdReports. Ohio was on the verge of doing that too, until education officials there decided to offer publishers without an EdReports review a new path to state approval.
But even prior to the recent wave of legislation, EdReports had become one of the most powerful gatekeepers influencing the instructional materials schools buy.
“Before EdReports, there was not a lot out there, not much to go on,” the organization’s CEO, Eric Hirsch, said in an interview. “Our goal is to get more information out, to empower districts and states to make right decisions on this really, really important purchase.”
During the interview, Hirsch downplayed the organization’s role in the curriculum market, arguing its reviews are just a starting point. But education policymakers appear to be treating them as much more than that.
Rhode Island and Delaware require schools to use curricula with good reviews from the organization. Other states, including Hawaii and Colorado, recommend that schools consult its reviews before buying new curricula.
If a program gets a positive review from EdReports, “you can take that to the bank,” Carey Wright, Maryland’s state superintendent, said at a state school board meeting last year.
Products with positive reviews on EdReports have seen rapidly increasing market share, and publishers regularly revise their offerings to boost their scores.
But experts in reading instruction say EdReports’ methodology is flawed because it doesn’t give weight to real-world studies testing how effective programs are. Instead, EdReports has largely focused on whether programs adhere to a set of standards called the Common Core. In a system that critics say was too easy to game, publishers had an incentive to pack in more material to check more boxes. But historically, EdReports has rarely penalized programs for including things that research suggests should be left out.
David Liben, a literacy expert who advised EdReports in its early days and trained its first teams of reviewers, has since become a critic of the way it applies the Common Core standards. “Success is dependent upon how [programs] align with standards,” Liben explained, “as opposed to how they align with the science of reading.”
Liben said he’s consulted with publishers who end up “shoehorning standards in where they don’t fit ... losing why we want to read this text, what we can learn about the world and what we can learn about ourselves.”
Liben and his wife, Meredith, were part of the group that developed the Common Core standards for literacy. The Libens have developed an alternative curriculum review system for the Knowledge Matters Campaign, which is partly funded by publishers.
Meredith Liben says EdReports has been “successful beyond everybody’s wildest dreams,” when it comes to influencing publishers to incorporate the standards into their materials. “But what has been so problematic is how deeply flawed they were in that execution.”
The Libens say EdReports has confused what the standards are meant to do: Common Core outlined what a student should learn in each grade, but it didn’t say anything about how to teach that in each lesson. It wasn’t meant to be a checklist for publishers, they argue, but EdReports made it into one.
EdReports recently acknowledged criticism of its methodology and, this past November, released a new scoring rubric in response. Hirsch, the CEO, said the revisions might not have come soon enough to satisfy critics, “but it’s something we take seriously and want to spend time doing well and wisely.” Courtney Allison, the organization’s chief academic officer, said the updates were prompted by the “ongoing conversations” about the science of reading.
But the new rubric only applies going forward. The organization has no plan to revise its past work, leaving an extensive library of reviews on its website scored on an outdated measure.
The changes may also come too late for many states, which are pressing forward with mandates requiring schools to adopt literacy programs that have received EdReports’ approval. And the textbooks districts are buying now could determine how reading is taught for much of the next decade.
All-green ratings become the goal
In 2011, some of the biggest names in education gathered at a 200-acre Southern California estate just outside of Palm Springs. Their goal: to modernize American education.
Sunnylands was once a winter residence for the late philanthropists Walter and Leonore Annenberg. Blocked from public view behind a pink wall and a line of eucalyptus and tamarisk trees, their estate was now billed as a place “where national and international leaders convene.” President Barack Obama liked to entertain foreign dignitaries there.
The education experts met in Sunnylands’ 25,000-square-foot midcentury-modern mansion. A Rodin sculpture greeted them in the atrium, and sunlight streamed through walls of glass.
Bill Gates and staff from his foundation were in attendance. So were Sal Khan, founder of the popular Khan Academy online tutoring program, and Carol Dweck, the famed Stanford psychologist who coined the term “growth mindset.”
They were all there to discuss how to implement the Common Core state standards that Gates had helped to fund. The standards define what students should know and be able to do by the end of each grade. They were designed to bring consistency to the state-by-state patchwork that resulted from America’s decentralized education system.
The standards enjoyed bipartisan support. Conservatives saw them as a tool for test-based accountability; liberals hoped they would bring more equality to the education system. By 2012, all but five states had adopted the Common Core standards for reading and math.
But the luminaries gathered at Sunnylands soon realized they didn’t know which curricula were following those standards. They decided they needed an organization to review classroom materials against the standards, starting with math.
With millions of dollars in support from philanthropists, including the Gates Foundation, EdReports was formed in 2014 to solve that problem. Hirsch, EdReports’ CEO, said at the time that he hoped publishers would one day use his organization’s input to improve their products.
The organization scored materials using a color-coded system. Green was the top rating, followed by yellow and finally red at the bottom. Getting to those final scores is a painstaking, months-long process. Reviewers pore over textbooks, looking for examples of Common Core standards and evaluating usability. The final reports can run more than 100 pages for each grade level.
In 2015, EdReports released its first reviews of math curricula. It flunked most of the programs it looked at, finding only one that fully aligned with standards.
Within just two years, EdReports was being used in 40 states, and it was proving influential in districts’ curriculum choices.
“For all the decades I was in education, no one paid attention to curriculum,” said Susan Pimentel, one of the Common Core’s lead authors. “EdReports came on the scene, thought about what the instructional shifts were in English language arts and math and then said, ‘Now we're going to take a look at that.' They put curriculum in a spotlight.”
The Council of Chief State Schools Officers, the association of state superintendents that played a crucial role in building support for the Common Core standards, encouraged states to use the reviews, and the Gates Foundation put up $10 million to help districts buy curricula that, among other requirements, were highly rated by EdReports.
An “all-green” rating became the goal for textbook publishers. EdReports boasts that more than 40 publishers have adjusted their products in response to its feedback. Within the industry, a top rating from EdReports can be a sales boon. In 2020, Houghton Mifflin Harcourt CEO Jack Lynch predicted his company’s reading program would continue to be a market leader after being “validated by third-party EdReports.”
Districts are increasingly buying English language arts materials that EdReports has rated as high-quality. The percentage of teachers using such materials has more than doubled since 2019, according to data compiled by RAND. EdReports says it has documented nearly a quarter of school districts nationwide using its reviews, representing more than 18 million students.
Awarding points for ineffective strategies
In 2020, Kari Kurto, a longtime advocate for students with dyslexia, started working as a literacy specialist for the Rhode Island Department of Education. The agency was a national leader in the push for high-quality instructional materials. It contracted with EdReports to develop a list of programs every district would be required to use to teach reading and math.
Kurto was excited to see what these high-quality options looked like. Her first week on the job, she flipped through publishers’ sample materials in her cubicle. She called it a “a jaw-dropping moment.”
One of the textbooks Kurto paged through taught students to use context to guess unknown words. That approach, sometimes called “cueing” or “three-cueing,” minimizes the importance of sounding out letters and runs contrary to scientific research on how skilled reading develops. Studies have repeatedly shown that cueing is ineffective and even counterproductive.
“Yet they were on this all-green list that said, ‘Go ahead and adopt these programs. This is what the Rhode Island Department of Education stands behind,’” Kurto said.
In some reviews, EdReports has penalized programs that used the flawed techniques. It gave out the lowest scores it’s ever recorded to two literacy programs that have been most closely associated with teaching those ineffective reading strategies: Fountas & Pinnell Classroom and Units of Study for Teaching Reading, both published by Heinemann.
But EdReports gave good ratings to other programs that included similar strategies.
Benchmark Advance, published in 2021, instructs first-grade teachers to “model how to use sounds, picture clues, and the context of the sentence” as they read “words that are not high-frequency or decodable.” ARC Core, published in 2017, instructs students to “read the pictures,” and it tells teachers that students who misread the word “paws” as “feet,” “legs,” “toes” or “claws” have nevertheless given “acceptable answers.” And EL Education, distributed by both Imagine Learning and Open Up Resources, also said students who had difficultly sounding out letters could “look at the picture(s)” and hear “what would sound right in the sentence.”
EdReports gave ARC Core and EL Education all-green ratings. It gave Benchmark Advance a yellow rating overall, but the program received a perfect score for how it taught foundational reading skills.
Neither American Reading Company nor Benchmark Education Company responded to emails seeking comment. Paul Curtis, EL Education’s managing director of communications, said the cueing tools were intended to be “an optional routine for teachers more familiar with the three-cueing system,” but they were replaced in 2022 with a routine that stressed phonics, spelling patterns and syllable types. “We’ve made a concerted effort to remove the older version from customer-facing materials,” Curtis added.
The positive reviews from EdReports appear not to have been an oversight. The evidence guide EdReports used to support its initial scoring rubric said teachers should use the cueing system “when the phonics patterns do not work or to confirm a word choice.” And the document elsewhere cites the work of the New Zealand researcher Marie Clay, who developed the cueing theory, and scholars Gay Su Pinnell and Irene Fountas, who popularized it in American schools.
Back in Rhode Island, Kurto typed out her concerns in a memo and sent it to her superiors. But by then, she said, it was too late: Some of the larger districts had already picked out programs with the ineffective strategies. She remembered feeling a sense of panic.
“I tried to see if, within that narrow window of time, if there was anything I could possibly do to influence some of the decisions,” Kurto said. “But as a specialist that was two weeks new to the job, my voice was not the first one that they turned to.”
Two years later, Kurto switched jobs. Now at The Reading League, an advocacy group for aligning instruction with reading research, she’s developed her own curriculum review tool — one that she says is meant to supplement EdReports, not compete with it.
EdReports reviews were primarily designed to ensure that every standard was covered. A program could achieve a higher score by adding more material. Hefty anthologies, cluttered with add-ons, were acing the reviews.
Kurto’s tool used a different methodology, focusing on what should be left out. Especially when it comes to curriculum, Kurto says, less is often more. So her rubric docks points for “red flags” in addition to looking for signs of evidence-based instruction.
“These packages are huge,” Kurto said, warning that overstuffed programs could have components that “are going to potentially waste that very, very important instructional time.”
Like The Reading League’s system, EdReports’ updated rubric now includes non-negotiables. It automatically fails programs that contain any evidence of three-cueing practices. “It became clear that it would be a service to states and districts if we also drew a clear line there,” said Allison, the organization’s chief academic officer.
The organization has also revised how it judges usability to prevent publishers from gaming the system by overstuffing their programs. Going forward it will penalize textbooks for containing more than can be taught in a school year.
A ‘barrier’ to research-tested programs
At the same time EdReports was green-lighting programs containing problematic elements, it was denying its seal of approval to others backed by powerful evidence demonstrating their effectiveness.
One notable example was Bookworms. Developed by reading researchers and published in 2018, Bookworms is one of the few reading curricula with multiple peer-reviewed studies showing it significantly raises student achievement.
A study published in late 2023 found that, after adopting Bookworms, students scored 16% higher by the end of fifth grade, the equivalent of about five months in class. The effects were even greater for students who were already behind, especially those in special education.
But EdReports has repeatedly rated the program as only partially meeting expectations. The first review found the curriculum deficient because it didn’t include enough formal writing instruction. One of Bookworms’ authors, Sharon Walpole, a professor at the University of Delaware, responded by adding all-new writing lessons. But Bookworms still couldn’t get a coveted all-green rating.
The second review, done with the same rubric, docked points in areas where Bookworms had excelled in the earlier review. EdReports now took issue with the curriculum’s basic structure. The reviewers wanted to see more “text sets,” several selections on the same topic. But Bookworms favors full-length books. That pedagogical difference led to another poor review.
Walpole revised Bookworms once more, hoping to pass on a third go-round. She reorganized the books into clearer content units that would more closely match EdReports’ rubric. Walpole’s nonprofit publisher, Open Up Resources, spent $455,000 “to support [an] all green rating on EdReports,” tax filings show.
But in 2022, EdReports gave it the worst score yet. The reviewers now took issue with the way the program taught phonics, because those lessons were optional for students who were already skilled readers — something the previous two reviews hadn’t flagged.
Walpole, who declined an interview request, expressed frustration in a 2022 talk at a literacy conference. She called EdReports’ system “unstable” and a “barrier” for programs like hers.
Bookworms’ inability to get an all-green rating points to an underlying problem with EdReports’ model, said Timothy Shanahan, a retired professor from the University of Illinois Chicago who reviewed research for a landmark analysis by the National Reading Panel in 2000.
Shanahan sees evaluating whether a program is aligned with standards as a proxy of sorts: It is something to look at if there’s no research directly showing the program works. The “prediction is that the more aligned they are, the more effective they’ll be. That’s the underlying idea. And that makes great sense — until you actually get one of those programs that actually has research on it,” Shanahan said.
But Bookworms’ negative review breaks that logic. “I’m kind of shocked that they don’t just give it a pass,” Shanahan added. “Bookworms didn’t get all green, and yet it was effective. Maybe you don’t have to have all green to be a really good program.”
Tom Loveless, a Brookings Institution fellow, echoed that point in an Education Next column in 2020. “A curriculum-review process that gives greater weight to adherence to standards than to impact on learning is not identifying high-quality curricula; it is identifying conforming curricula,” he wrote.
Hirsch said he would not comment on specific program reviews. He said EdReports asks publishers to submit evidence of efficacy in a supplement that’s published alongside its reviews. But he noted that “there is a dearth of research available around efficacy for a lot of materials.”
Despite the studies validating it, Bookworms is rarely used in American classrooms. Only four states recommend using it for early reading instruction. And according to RAND survey data, only 1% of teachers use it.
‘The wrong approach’
In 2022, Kentucky legislators passed the Read to Succeed Act in an attempt “to support evidence-based early literacy instruction throughout the commonwealth.” Regulations soon followed, making a good review on EdReports a prerequisite for districts to use a curriculum. That put another program with powerful evidence behind it, Success for All, at risk.
Success for All isn’t exactly a curriculum, but rather a “comprehensive school reform model” that covers every academic subject and even includes a system for addressing absenteeism.
Large-scale studies have repeatedly shown Success for All to be one of the most effective models for getting kids reading. It’s been proven effective in randomized controlled trials, seen as the gold standard in research. That’s the same kind of study used to determine whether pharmaceuticals work, but it’s rarely used in education because of the expense and the difficulty of getting schools to participate.
Nancy Madden and her late husband, Robert Slavin, developed Success for All in the 1990s, two decades before the Common Core was created. As a result, it wasn’t built with those standards in mind, and Madden never thought it made sense to judge her program against them.
“It’s the wrong approach,” Madden said. “We need to judge: What’s the outcome? We need to look at: What is the evidence of effectiveness?”
EdReports for years declined to review Success for All, citing its unusual program design. “Given this structure, reviewing the curriculum alone would not have provided a complete picture of the program’s core services, and we decided not to move forward with a review,” EdReports spokesperson Janna Chan said in an email. She added that EdReports planned to “revisit this conversation this year” about potentially reviewing Success for All.
But as EdReports’ clout has grown, Success for All has suffered.
In Kentucky, two countywide districts, Bell and Knox, are dropping Success for All because the state has required programs to have all-green ratings from EdReports.
Julie Wible, Success for All’s CEO, told Kentucky Department of Education officials in a 2024 email that she was “distressed” by the state’s new regulations. “The programs … recommended by Kentucky have received positive curriculum reviews, but have not demonstrated positive outcomes in outcome research,” Wible wrote.
An official replied that any core program deemed to be all-green on EdReports would comply with the law. “I have been unable to find SFA on EdReports,” the official wrote. “I hope this clears up any misunderstandings.”
Meanwhile, Ohio belatedly added Success for All to its list of approved programs in January, following pushback from school districts concerned that a new state law would force them to jettison the program. The state had not initially approved Success for All, in part because it lacked the EdReports imprimatur.
“It was like relief,” said Melinda Young, superintendent of Steubenville City Schools, describing the moment she learned the state had reconsidered. “Yes. Yes. Relief.”
Steubenville, located on the border with West Virginia, has used the program for 25 years. It routinely produces some of the best third grade reading scores in the country. Those results are especially remarkable because the state considers almost all its students “economically disadvantaged.”
Leaders at Grove Paterson Academy, a magnet school in Toledo with a waitlist of families vying to get in, were also nervous they would have to get rid of Success for All.
“It’s everything,” Principal Herneika Johnson said. “We can’t work without it. If your heart is not beating, your body is shutting down. Because we have all the structures in place, we have the reading curriculum in place, everything else comes out of that. It’s the heart of our school.”
But across Ohio, hundreds of districts already decided on a new curriculum during the yearlong wait on Success for All’s approval. Ohio has disbursed $51 million for districts to spend on new programs. At least two charter schools, Richard Allen Academy in Dayton and Columbus Bilingual, dropped Success for All, and Madden said no Ohio districts reached out about adopting it.
A new scoring system, going forward
Last year Hirsch, who has run EdReports since its founding, wrote an internal memo acknowledging criticism of the organization’s reviews and its increasingly powerful role.
Hirsch said that EdReports’ critics didn’t understand that the organization “has always reviewed instructional materials for the science of reading.” He said a “subset” of the English language arts field “has chosen a strategy that is adversarial instead of collaborative.” He added, “Bad press will come with success.”
Publicly, Hirsch said a lot had changed since EdReports developed its first review criteria a decade ago, including a global pandemic that kept students out of school and the advent of artificial intelligence, both of which upended how teachers assign and grade work.
“We know both what we review for and how we conduct our reviews must reflect these changes,” he said. “We’re drawing upon and learning from our efforts over nine years to continue to be responsive to the education landscape.”
EdReports has already revamped its website to address concerns about how its reviews failed to account for some reading research. A “Science of Reading” snapshot now highlights ratings just for foundational reading skills. An enhanced rubric is also being rolled out.
But even as EdReports acknowledges it needs to change its methodology, the organization has said it will not reevaluate the reviews of 86 English language arts programs that it has already published.
“We just aren’t able to go back right away and review everything ever, given the volume and our size as a nonprofit,” Allison said. With over $10 million in revenue in 2023, EdReports had 60 employees on staff, according to its most recent public tax filing.
Hirsch said he hoped decision-makers would look beyond the top-line ratings for the nuance in EdReports’ reviews. “We are the first to say, ‘Green does not necessarily mean go,’” he said.
Many educators hope that EdReports’ new scoring system will address what they consider its past deficiencies.
“Really, what they’re doing is very new. The standards aren’t that old yet, and this is the first organization that’s tried to do something,” said Chris Hayes, a Nevada reading coach who once advised EdReports on how to assess phonics programs. “I hope that the pushback just makes them better.”
Hayes said EdReports has added a valuable perspective — even if it’s just one perspective. She said any administrators who are ready to spend potentially millions on curriculum should be looking at a range of reviews. That decision-making process is not unlike what the Common Core expects of students, she added.
“You’re supposed to read. You’re supposed to find evidence. You’re supposed to compare texts and come up with your own opinion,” Hayes said. “You’re not supposed to blindly follow any of it. That’s what we’re trying to teach the kids to do. The adults maybe should do that, too.”
Additional reporting by Emily Hanford, Olivia Chilkoti, Kate Martin and Carmela Guaglianone.
