TheParagon-Education

The project began in 2009, when stories of underperformance in LA schools prompted the reporters to start looking into how teachers were being held to account by the school district.

“What we realised,” Felch says, “was that there was absolutely no measure of performance. For decades in LAUSD, teachers have essentially been given drive-by evaluations - very quick visits from a principal sitting in the classroom, checking off. Nationally it was the same picture. Teachers all around the country were receiving no feedback on their performance.”

Progress v Achievement
While value-added (and contextual value-added) scores may have fallen out of favour in the UK, they have started to become more popular in the US over the past three years, and the approach piqued Felch’s interest.

“Schools were being called failing schools only because they had poor children,” he says. “Value-added was an effort to correct that by bringing in socio-economics, and bringing in growth rather than achievement level.”
By looking at how much progress a pupil makes over a set period of time rather than raw attainment, the theory goes, schools with low socio- economic catchment areas can be judged fairly alongside their neighbours in more affluent areas. The argument, staff at the Los Angeles Times soon realised, could be extended to teachers: they could mine the data to extract the impact of individual teachers in terms of how much value they added to pupils’ education.

After six months of haggling with the school district, the newspaper finally got hold of the figures it wanted by using freedom of information legislation. “No one had ever asked for the data before,” Felch says. “No one had even thought to ask. Even internally here, people were telling us, ‘You’ll never get that. They’ll never give you the data. Even if they do, you’ll never be able to analyse them.’”

But analyse the data they did. The Los Angeles Times hired Richard Buddin, an education policy expert at RAND Corporation, to do the number crunching, before checking his work with several other academics and its own in-house data experts.

But while critics of the project were quick to damn Felch and his colleagues as journalists out to take a cheap shot at the teaching profession, he insists that their motives were genuine.

“In the United States, our whole education system is a self-fulfilling social prophecy,” he explains. “Because of our accountability structure with testing, poor kids do poorly, rich kids do great on tests. That makes us think that the schools that these rich kids go to are great schools.

“[The system] is built on this ridiculous fallacy. Yet parents, teachers, the state, resources, all of it is geared towards this fallacy. We saw value-added (scores) as a way to cut through the socio-economics that are skewing the whole picture, and really shine a light among students, teachers and schools that’s not just a reflection of socio-economics.

“[This is done] by comparing students with their own prior behaviour. So if a student comes from an inner-city family - dad’s not in the picture, mom’s on drugs - the assumption of value-added is that that (scenario) is a relative constant in this kid’s life.”

Felch argues that by looking purely at the relative progress made by pupils in successive exams, it is possible to strip out extraneous social factors, meaning that pupils - and teachers - can be compared on a like- with-like basis.

Using this measure of pupils’ relative progress, the Los Angeles Times rated the city’s elementary teachers on their effectiveness in teaching English and maths in terms of how much value they added - ie, whether pupils progressed more rapidly than would have been expected, based on their prior performance. Each teacher was classed as “least effective”, “less effective”, “average”, “more effective” or “most effective”.

As well as posting details about the calculations on its website, the Los Angeles Times also gave teachers the chance to raise their concerns.

“A lot of teachers felt maligned by the data,” Felch says. “One of the things I’m proud of is that we took those complaints seriously and allowed teachers to point out mistakes in the data and things that were unfair.”

He estimates that about 80 per cent of complaints were from teachers who simply thought they deserved a better rating; the remaining 20 per cent were legitimate grievances about errors in the data. As a result, some teachers’ scores were removed.

Teachers hit back
The comments that teachers posted next to their ratings offer an insight into the massive impact the project had on teachers’ lives. Some teachers offer reasons for their low scores, such as retirement, maternity leave or the fact that they didn’t actually teach the classes concerned. Others take the opportunity to express their pain and anger.

Angelica Barraza, a third-grade teacher at Hooper Avenue Elementary, writes: “I’ve seen the disheartening effect of your scoring system on excellent teachers that I have had the privilege of working alongside… One teacher in particular comes to mind. He’s the type of teacher who is first in and works through recess and lunch. A good teacher who was made to feel that his efforts as an educator were meaningless based only on test scores. ‘What more can I do?’ he asked as he reviewed the ratings himself, trying to figure out what led to his poor showing.”