Transcript of Sold a Story E12: The Evidence

In 1969, a young woman named Nancy Madden graduated from high school in Minnesota. And went off to Portland, Oregon. To go to Reed College.  

Nancy Madden: Reed's a very odd place. I mean it's where you go to be very intellectual and very disruptive.  

She was a child of the ‘60s. Protested the Vietnam War. Marched for Civil Rights. And what Nancy was most interested in disrupting was education. She wanted to figure out how to make schools better — especially for poor, Black children.  

Madden: The disparities in opportunity for children were just so obvious at that time.  

In college, Nancy met a guy — a fellow student named Bob Slavin.  

Madden: Our first date was to go on a walk to sort of talk about — how do we improve education? What can we do?  

Nancy had tutored kids at poor schools in Minneapolis when she was in high school. And Bob had worked with kids in Washington, D.C. 

Madden: There was an orphanage that he was a bus driver for when he was 16 years old. And, ah, you know, we both as high school kids knew that there was just so much inequality of opportunity for kids that ... didn't need to be.    

They wanted to do something about it. They wanted to make poor schools better.  

But there was doubt at the time about whether improving schools could help poor children. About whether the quality of a school really mattered. Because of a big report that had been released a few years earlier. A report that had shaken the field of education. 

(Music) 

In the 1960s, the federal government commissioned a sociologist named James Coleman to do a big study of educational opportunity in America. He gathered all kinds of information from thousands of schools. Data on academic achievement, student demographics, teacher training, curriculum. It was one of the largest educational studies ever at the time.   

And what Coleman found was that a student’s academic achievement depended a lot on their family’s socioeconomic status. In fact, the family a child grew up in seemed to matter more than the school a child went to. This undercut the argument that improving schools could improve outcomes for kids. But Nancy and Bob — and other people, too — were convinced there was more to the story.   

(Music ends) 

Because if you dug into the details of Coleman’s report, what you could see in the data was that some schools were having more of an impact than others.   

(Music) 

Some schools were making a difference for kids. And Bob and Nancy wanted to figure out — what were those schools doing? What made them effective?   

I’m Emily Hanford and this is Sold a Story, a podcast from APM Reports.  

In this episode, I’m going to tell you the story of a program that Bob Slavin and Nancy Madden created. A program that’s proving schools can make a difference. It’s the program Steubenville started using 25 years ago. A program that’s been extensively studied and is backed by substantial evidence. In fact, it’s become kind of a poster child for an evidence-based program. But what does it mean for something to be “evidence-based”? It’s a critical question right now because states are making lists. Lists of approved programs — programs they say are backed by the science of reading. But approving programs, making lists — we’ve tried that before in this country, and it kind of backfired. And it might be backfiring again. 

(Music ends) 

Nancy Madden and Bob Slavin got married after college. They moved to Baltimore. They both got Ph.D.s. And by the 1980s, they were working together at a research center at Johns Hopkins University. Studying educational practices. They were looking for things that worked. Things that schools could do to be effective, to make a difference in kids’ lives. And one day they were at Johns Hopkins eating lunch. And a former member of the Baltimore City school board joined them at their table. Nancy says they struck up a conversation.   

Madden: And so we’re talking about, how would you change the schools?  

Things were not good in the Baltimore schools.  

Madden: At the time, Baltimore city schools were failing half of their high school students. I mean, they were just dropping out. And he said, this is wrong, this is not good enough.   

And he issued a challenge.  

Madden: Here you are, Johns Hopkins University, you know, you’re so smart. What would you do?  

Nancy and Bob had been studying what works in education. The school board member wanted to know — what would they actually do if it was their job to fix a school system. Here’s what they told him.  

Madden: You may be worrying about high school, but if those kids aren't reading by third grade, then you've lost them. You have to do that early part really well.  

So, the former school board member said: I’ll find the money. You go do it. Create a program that will help us fix the schools. Bob and Nancy started with preschool.   

Madden: So we started with full time pre-K for the language focus, ‘cause the first thing you have to do is get the language base laid before you even start thinking about letters and sounds and that sort of thing.  

They developed a kindergarten program and a first grade program.  

Madden: We were revolutionary at the time. We put a phonics-based beginning reading program in place from Day 1, very much opposed to the zeitgeist at the time.  

They weren’t interested in what was popular. They were interested in what works.   

(Music)  

And not just what works when it came to reading instruction. They were trying to implement effective practices to address all the things a school needs to do to be successful. And as you heard in the previous episode, it takes more than good instruction to be a successful school.  

Madden: Kids weren't coming to school. So attendance had to be addressed. Parents weren't engaged, so you had to engage parents. Had to have tutoring because kids who start to fall behind, if you don't get them right back in so that they can take advantage of the core instruction, then you're just going to keep losing them.  

Nancy and Bob created what’s known as a “whole school reform” program. It wasn’t just a reading program. It was a program to improve an entire school.   

(Music ends) 

They called their program Success for All.   

And a big focus of Success for All is prevention. Preventing kids from failing in the first place. Bob liked to use a metaphor when he talked about that.  

Bob Slavin: One way to understand how Success for All came about and what it’s trying to achieve ....  

I wasn’t able to interview Bob; he died four years ago. This is from a video.   

Slavin: Consider an old story about a little town that decided to build a gorgeous playground on some land that it had. The problem, however, was that this beautiful land was at the edge of a cliff. And it occurred to the town fathers that there was a danger that children might fall off the cliff. So the local playground board had huge debates. Should we build a fence at the top of the cliff? Or should we put an ambulance at the bottom?   

(Music)  

Slavin: I think, if you think about what that story is telling you, you’ll realize that to put an ambulance at the bottom of the cliff is the way we do so much ordinarily in schools. Part of the idea of Success for All is to try to make sure that children don’t fall off the cliff in the first place. That they’re successful, particularly in reading, from the beginning of the time in school.   

You met someone else in this podcast who was trying to do exactly the same thing. Someone who was trying to prevent reading difficulties. Trying to prevent kids from falling off the cliff.   

Marie Clay: And my idea when I started my special research here in New Zealand was — could, could you see the process of learning to read going wrong?  

Marie Clay. The woman who created the Reading Recovery program.  

(Music ends) 

Marie Clay had the same goal as Bob Slavin. But her mistake was misunderstanding how kids learn to read. A mistake that can be traced back, at least in part, to the way she did her research. She observed children and came up with a theory — an idea — about how they learn. She built her program based on that idea. But then a lot of cognitive science research came along that showed her idea was wrong.   

Bob Slavin and Nancy Madden took a different approach. They didn’t start with an idea about how kids learn to read. They started with a collection of practices, backed by rigorous research. Practices like phonics instruction that had already been studied and shown to work.  

Madden: We didn’t invent a lot of these concepts. We assembled them — you know — tried to look for what is best practice. 

Nancy and Bob were betting that if you put a bunch of effective educational practices together, the result would be a successful school. A school that would make a difference for kids.  

But Bob and Nancy didn’t know if their program would work. It was kind of like they were making a soup. They knew the ingredients were good. But what about the soup? Would the soup taste good? Would the recipe actually work?  

They wanted to know the answer to that question more than anyone. So, as soon as Bob and Nancy created their program, they were studying their program.  

(Music) 

Their first study compared five schools in Baltimore that had chosen to do Success for All with five similar schools that had stuck with business as usual. What they found was that the kids in the Success for All schools did better on several key measures. They were absent less, less likely to repeat a year of school. And by the end of third grade, they were about eight months — almost an entire school year — ahead in reading.   

Bob and Nancy did more studies. 

(Music ends) 

Madden: We were researchers. We did everything as a study.   

And it wasn’t just Bob and Nancy studying their own program. By the ‘90s, independent researchers were also taking an interest in Success for All.   

Geoffrey Borman: Yes, hi, um, my name is Geoffrey Borman.  

Geoffrey Borman was a newly minted University of Chicago Ph.D. in 1997 when he got a job as a research scientist at Johns Hopkins. Education research was his thing. Here’s why:  

Borman:  It may be silly, but I'm still a strong believer in the idea of the American Dream and — that through quality education — that anyone in our society can find success.  

When Geoffrey Borman got to Hopkins, he decided he wanted to do his own study of Success for All. He wanted to know if kids who had been in Success for All elementary schools were still doing better as they got older.  

Borman: If you know the history of education, we struggle so much to find interventions, programs, policies that have lasting impacts on kids. And so it was a really important question to me: Did Success for All have a long-term effect on kids?  

He looked at how the students in those original five Success for All schools in Baltimore were doing by the end of eighth grade.   

Borman: It was quite a rosy picture. A lot stronger impacts than I really imagined initially. 

His study showed that, by eighth grade, the Success for All kids were still ahead in reading. They were still less likely to be held back. And they spent less time in special education. That’s a big deal. Special education is expensive. So is holding a child back — think about it: You have to pay for an additional year of schooling. In his study, Geoffrey Borman compared the costs of doing Success for All to not doing Success for All.   

Borman: Although Success for All was a rather costly program it still was so effective in preventing a lot of these other very costly interventions that, in the long run, it actually cost just the same but had all these wonderful impacts on kids.  

The main impact was the kids were better readers.   

(Music) 

Success for All is now one of the most extensively studied education programs out there. There are dozens of studies that are notable not just because they found positive effects. The studies are notable because they’re high-quality studies. The U.S. Department of Education evaluates programs based on whether the research to support the program was rigorous and well designed. Success for All meets the highest standard for an evidence-based program.   

And that’s what Steubenville was looking for 25 years ago.  

(Music ends) 

More on that, after a break.  

** BREAK **  

(Music)  

(Music ends) 

The schools in Steubenville, Ohio, had a problem. It was the late 1990s and the Ohio legislature had just passed a new law. A law that said students were going to have to pass the state’s fourth grade reading test to move on to fifth grade. And it looked like as many as a third of fourth graders in Steubenville might have to be held back because they weren’t reading well enough. The school district had a reading curriculum at the time. But it was basically just a big textbook. A textbook with lots of stuff in it. Lessons, stories, worksheets.    

Melinda Young: A teacher wasn’t required to actually go through the textbook in an organized fashion. Like, they could pick and choose.  

This is Melinda Young, who was an elementary school principal. And is now the district superintendent.   

Young: It was really left up to the teachers on how they taught reading. You know, they closed their door and they did their thing.  
Emily Hanford: It was a little bit of a “choose you own adventure.”  
Young: Yes, yes.  

They were looking for something different.  

Richard Ranallo: We wanted something that worked.  

Richard Ranallo was an assistant superintendent. He remembers going to meetings with state officials who were advising districts on how they could improve reading achievement. It was at one of those meetings that he learned about Success for All.  

Ranallo: And I brought this, and I ran it off yesterday, ‘cause I knew you were coming .... 

He brought a piece of paper to our interview. It compared several programs available at the time. Rated them on how much research evidence there was. And Success for All ....   

Ranallo: It looked like it was the best. And so we thought, “Well, we want to be the best.”  

But before a district can adopt Success for All, teachers have to vote on it. Success for All requires a teacher vote.  

Madden: I mean, this is real change.  

This is Nancy Madden.  

Madden: If you want real change to occur, you don’t need to be fighting against, you know, sabotage going on.  

Young: The vote was 100%.  

This is Melinda Young again, the Steubenville superintendent.  

Young: And it was a secret vote, so it was a true vote. It wasn’t like all of the administrators were in the room and you know, saying, OK, what’s your vote? What’s your vote?  

But just because all of the teachers voted yes on Success for All doesn’t mean they were all enthusiastic about doing it.  

Lynnett Gorman: There were veteran teachers in the district — without a doubt, they were very resistant.   

Lynnett Gorman, who was a new teacher at the time, says there was a big sticking point for many of the veteran teachers: They didn’t like the scripts.   

(Music) 

Success for All is what’s known as a “scripted” program. That means it’s basically the opposite of “choose your own adventure.” Success for All provides detailed instructions for how to teach each lesson — including specific things for teachers to say and do. This has always been a controversial aspect of the program. A teacher in Nevada told The Wall Street Journal in 1999 that Bob Slavin was — quote — “killing creative teachers.” But remember, Success for All was a collection of effective practices, ingredients that studies showed would likely result in a good soup. But you had to follow the recipe. It wasn’t going to work if everyone was choosing their own adventure.   

(Music ends) 

Teachers in Steubenville remember hours and hours of training, learning how to use the program.  

Christine Ballato: And I just remember tables full of pies. (Laughs)  

Christine Ballato says the district fed teachers well.  

Ballato: Fruit pies, cream pies. And we had like big lunches. And, yeah, we were fed really good.  

When Steubenville started using Success for All, there were close to 2,000 other schools using it. Interest in the program was growing. And it seemed like interest was about to grow a lot more.  

George W. Bush: We will launch a new initiative called Reading First.   

You heard this earlier in the podcast. George W. Bush on the campaign trail, promising a big federal effort to overhaul reading instruction.   

Bush: But we will only support effective programs. Effective reading strategies.   

The president’s advisors wanted to steer the nation’s schools away from flawed, ineffective practices like the cueing strategies you’ve heard about in this podcast. They wanted schools to do things that were effective — like phonics instruction. The goal was to get schools to use programs and practices that were grounded in “scientifically based reading research.” Congress put that term in the law.  

Christopher Peak: Scientifically based reading research — that means you have the kind of studies like Bob Slavin did for Success for All.  

This is my co-reporter, Christopher Peak. He says the law defined “scientifically based reading research” in a specific way for a reason.  

Peak: Almost every program says they have evidence behind it. It's very easy to just say, well, we have this white paper, it's in the back of the book, or it's in the back of the sales materials. It’s a lot harder to actually meet that definition of being “scientifically based.”  

According to the law, a “scientifically based” reading program was one that had been studied using empirical methods. That means experiments and data, not just ideas or theories. And the studies had to be good. Well-designed and rigorous. 

Peak: What you find very quickly when you require extensive levels of research and evidence in order to be able to use something, is that there are very few programs that actually meet that bar.  

In fact, most people familiar with the reading research seemed to agree at the time that there were probably only two reading programs that had been tested and proven with scientific research.  

Peak:  One was Success for All. This other one's called Direct Instruction. Both of those programs had been tested, and they were shown to work repeatedly. Other programs weren't able to meet that same bar.   

Success for All did have the studies. So you would think Reading First would have been a boon for them. A real windfall. But to Nancy Madden’s surprise, it wasn’t.  

Madden: Yeah, it was horrendous.   

Here’s how the law ran into reality and ended in a bit of a train wreck.  

(Music)  

A goal of the law was to get schools to use programs backed by scientifically based research. But here’s the thing — the federal government isn’t supposed to tell schools what programs to use.   

Peak: The federal government is not allowed to endorse, recommend, mandate any curriculum.   

This is reporter Christopher Peak again.   

Peak: And that goes back to the founding of the Department of Education. It's literally written into the statute.  

It’s a provision to protect local control, which is a fundamental principle in American education. So there was a problem, a contradiction. And as the law got put into practice, things got messy.   

One thing that happened — lots of reading programs were saying they were aligned with scientific research, even if they didn’t have studies that showed they worked.   

The other thing that happened — some Reading First officials and consultants were authors of reading programs. And when they were giving presentations to state officials, Nancy Madden says they were promoting their own programs.   

(Music ends) 

Madden: Their PowerPoints had examples. And we were not on their list of examples.  

Nancy says schools that were using Success for All started dropping it.  

Madden: Our schools were told — you have to stop using Success for All. It’s not on the Reading First list. You cannot use it anymore.  

To be clear, there was no official Reading First list. Not from the federal government anyway. But some states were making lists. And when a state left Success for All off its list, the message seemed to be — Success for All isn’t backed by research. It’s not an evidence-based program. And that had a big impact on Success for All.  

Madden: We had a staff of 500 people and we fired half of them. It was just awful.  

And Bob Slavin ….  

Peak: He wanted to do something about it.  

This is Christopher again.  

Peak: So he actually filed a complaint with the Office of the Inspector General.   

The Office of the Inspector General of the U.S. Department of Education — an internal watchdog that investigated alleged wrongdoing in federal education programs.   

Peak:  So he files a complaint that says — hey, inspector general, you need to check out this Reading First program. I think they're violating federal law.   

In his complaint, Bob Slavin wrote, “The Reading First legislation itself is sound, well-intentioned.” 

But Slavin said Reading First had strayed from its intended purpose. That it was not promoting “scientifically based reading research.” That it had become, instead, a giant giveaway to publishers who were making millions of dollars on programs that hadn’t been tested or proven.   

(Music) 

And Bob Slavin wasn’t the only one who felt shut out by Reading First. So did Marie Clay, the creator of the Reading Recovery program. The Reading Recovery program ended up losing schools because of Reading First, too. And just a few months after Bob Slavin filed his complaint with the inspector general, Reading Recovery filed a complaint. Their complaint said, “We join a growing number of educators and scholars who are calling for an investigation.”   

And as you heard earlier in this podcast, there was an investigation. And congressional hearings. And a damning report from the inspector general.   

NBC News Anchor: The program known as Reading First is mismanaged, the auditors say, and full of conflicts of interest.  

The inspector general’s report said some Reading First officials and consultants with professional ties to reading programs were promoting those programs. And the report said some states felt pressured to use those programs in order to get funding. Reading First was ensnared in a scandal over programs.  

Peak: And Bob Slavin called for it to end.   

This is Christopher Peak again.  

Peak: By the very end of it, he was saying, cut funding for this program, he was so upset with it.  

Congress killed the funding. And Reading First collapsed.   

(Music)   

Bob Slavin had spent decades studying effective educational practices and trying to get schools to use them. He had shown that schools can make a difference for kids, if they followed the evidence. But he helped bring down the government’s big effort to try to get schools to do that.   

I’ve talked to several people over the years who expressed bewilderment — even anger — about Bob’s role in helping to bring down Reading First. Because they thought Reading First — even with its flaws — might have been the nation’s best hope for improving reading instruction at scale, for helping millions of kids.  

But Bob didn’t think Reading First was going to do that. Because schools were using programs that hadn’t been proven. And in some cases, they were dropping a proven program — his program — in favor of something else.    

(Music ends) 

Borman: That was a dark era for Bob.   

This is Geoffrey Borman, the researcher you heard earlier.  

Borman: He was so frustrated by the policy landscape here in the U.S. that he left. He left the U.S. altogether.  

Bob and Nancy went to England.  

Madden: He got an offer from the University of York to start an evidence-based policy program in the U.K. So, we did go to the U.K. half-time.  

Nancy says in England there was more support for their work.  

Madden: The government was interested in evidence and focused on really getting everybody to use the evidence that was being produced.    

And she says England was a nice change of scenery, too.  

Madden: We used to buy our meat at the butchery that had been there since the 1300s. That's the part I liked.  

Nancy and Bob continued working with schools and districts in the United States that had stuck with Success for All. Like Steubenville. Steubenville didn’t drop the program. Success for All was working for them.  

Young: SFA just fit us.  

This is Melinda Young, the former principal who is now the superintendent.  

Young: It just was something that we could become. Because we feel right now that it is what we are. Like, we are an SFA district. That’s — when anybody asks why do we have success, we start with SFA.  

But Success for All is not a popular program. About 800 schools use it now — that’s fewer than half as many as 25 years ago. And the requirement in federal law for programs to be based on scientific research? Christopher says Congress dropped that in 2015.   

Peak: And they said, we want schools to be able to use a much wider variety of programs. And they changed it. They said, let's loosen the requirements. We don't need all that scientifically based reading research anymore.  

According to current federal law, a program can be considered evidence-based without any studies at all. All you need is a rationale, an idea about why your program should work.  

Peak: So as long as you make a good argument that your program derives some of its lessons from research, that counts as evidence-based.  

(Music) 

Reading First was trying to get schools to follow the science of reading by using evidence-based programs. But the whole thing ended up blowing up over controversy about programs. By the end of Reading First, programs seemed kind of like the problem, not the solution. And Christopher says the entire educational publishing industry just looked bad.   

Peak: It felt very icky by the end of Reading First.  

And a lot of schools and districts started turning away from programs. They went back to more of a “choose your own adventure” way of teaching reading. And as you know from listening to this podcast, there was a small company in New Hampshire that could help teachers with that. A company that seemed different from the traditional publishing industry: Heinemann.   

(Music ends) 

What Heinemann and its star authors were offering wasn’t really a “program.”   

Peak: This was — here's a guide, and here's some books and fun things you might do to structure your class. So, it really was the antithesis of that scripted program like Success for All.  

Heinemann flourished in the wake of Reading First’s collapse in part because what they were offering didn’t come from scientists. It came from other educators.     

But now people are talking about the science of reading again. And states are making lists. Lists of approved programs. Because of Sold a Story.  

(Music) 

Julie Wible: This idea of science of reading coming to schools across the country, we were thrilled.  

Melissa Weber-Mayrer: We had a very short window to get things in place.  

Young: As naïve as I guess I was, I really just never gave it a second thought.  

Brenda Muse: SFA was not on the list.  

Steubenville Teacher: Why get rid of something that is proven to work?  

The state lists — and why Success for All isn’t on a lot of those lists — next time on Sold a Story.  

(Music) 

We have more about this podcast on our website, including a video that tells the story behind Sold a Story. You can find links in the show notes.  

If you want to help other people find this show, one of the best things you can do is leave a review on your favorite podcast app.  

Sold a Story is an APM Reports podcast produced by me, Emily Hanford, with reporter Christopher Peak. Curtis Gilbert is our editor. Chris Julin does mixing and sound design. Our fact checker is Betsy Towner Levine.   

Andy Kruse is our digital editor. Our theme music is by Wonderly. Final mastering of this episode was by Derek Ramirez.  

The Sold a Story reporting and production team includes Kate Martin, Olivia Chilkoti, Carmela Guaglianone, Emily Haavik and Emily Corwin. Additional help on this episode from Caspar Von Au. Special thanks to Margaret Goldberg.  

Tom Scheck is the deputy managing editor of APM Reports. Our executive editor is Jane Helmke.  

Leadership support for Sold a Story comes from Hollyhock Foundation and Oak Foundation.

Support also comes from Ibis Group, Esther A. & Joseph Klingenstein Fund, Kenneth Rainin Foundation, and the listeners of American Public Media.