Substitute the word ‘schools’ for ‘businesses’ and ‘companies’, and the same Halo Effect can be seen in the vast majority of judgements many people make about schools. More worryingly, in the case of schools, the ‘performance data’ isn’t up to the job assigned to it. The grades are guesswork, and the Halo Effect turns out to be a golden - but fundamentally wrong - glow which surrounds castles built of sand.
Understanding The Halo Effect
Rosenzweig summaries the Halo Effect as follows:
“So many of the things that we – managers, journalists, professors, and consultants – commonly think contribute to company performance are often attributions based on performance.“ p64
He demonstrates this clearly using examples of companies which have benefited from the Halo Effect. In the introduction to the 2014 revised edition of the book, he explores the case of Marks and Spencer:
‘In 2005, Marks & Spencer () ranked () 127th out of 220 companies in a survey of most admired British companies conducted by Management Today. By 2007, Marks & Spencer was in first place, lauded as Britain’s Most Admired Company. In not one of Management Today’s nine categories did it finish out of the top three.
Could it be that Marks & Spencer raked near the top in every one of nine different categories? It seems too good to be true – and sadly, it is. Marks and Spencer’s success in the Management Today survey is a good example of the central phenomenon described in these pages. When it posted strong sales and profits, Marks & Spencer enjoyed a Halo whose aura extended to just about everything it did. Impressed by its strong performance, observers inferred that it must be innovative, that it must have great management, that its marketing must be brilliant, and so on.
Not surprisingly, Marks & Spencer’s performance faltered the next year, it quickly fell out of the top tier on almost every one of these same measures. Without objective measures and valid data, the survey turns out to be little more than a feel-good story masquerading as scientific research.” pxiv/xv
In English education, there are many similar examples of Halo Effects. One of my favourites is the ‘London Effect’, in which various people have tried to account for the rise in GCSE results in London’s Secondary schools in the past fifteen years or so. There is no doubting that GCSE results have been getting better, as they have across the whole of England. Chris Cook has documented the effect of this rise.
'Lessons from London schools for attainment gaps and social mobility')
The general view in education has been that the rise in GCSE results was due to the London Challenge which, according to Ofsted in 2010, ‘continued to improve outcomes for pupils in London’s primary and secondary schools at a faster rate than nationally. Excellent system leadership and pan-London networks of schools allow effective partnerships to be established between schools, enabling needs to be tackled quickly and progress to be accelerated.’ The TES reported in 2011 that, ‘Inspectors, unions, heads and a variety of impressive statistics all seem to agree that the hugely influential London-wide programme has succeeded.’ Forbes magazine reported similar views about Tower Hamlets, an Inner London borough, saying, ‘much of what we know about school improvement is the result of work at school level. What makes the Tower Hamlets experience particularly useful is that it demonstrates the possibility of success not just in one school but an area level. As a result, the Tower Hamlets template offers hope not just at a local scale, but for all children.’
Earlier this year, various different reports looked more closely into the reasons for London’s ‘transformation’. The Institute of Fiscal Studies had a good look, and concluded that the London Effect was actually due to higher prior attainment at primary school level, not anything which secondary schools had themselves done.
This surprised many people, who were sure that the increase in ‘performance’ at GCSE was surely due to something which the schools themselves had done, wasn’t it? It couldn’t be a classic Halo Effect attribution, surely? The IFS’s suggestion was that somehow primary schools must have done something to cause Key Stage 2 results to increase. They noted that the National Strategies for Literacy and Numeracy were introduced in Primary Schools at around the right time, and suggested – purely based on the Halo Effect, as far as I can see – that this was probably the cause of the rise in results.
The Centre for London’s Lessons from London Schools interviewed many people involved in London schools who - unsurprisingly for those who understand The Halo Effect - felt that what they had believed to be positive contributions to ‘performance’ had indeed made significant contributions to ‘performance’… It’s ‘Halos all the way down’, as Rosenzweig notes dryly.
Certainly, the results in London Schools are incredibly different to schools outside the city. The difference between the GCSE results of disadvantaged pupils in inner London and those in the rest of England are startling. In 2013 almost 50% of disadvantaged children in Inner London were awarded 5 A*-C GCSEs compared to just over 35% of disadvantaged children in England as a whole. This means that almost 50% more children hit achieved this measure, so something was going on. But as to what that was, with no objective method of assessing the underlying cause, is anyone’s guess...
I have a theory, and you might have one too, but in reality, we simply don’t know. But that hasn’t stopped people looking at ‘performance’ and building huge castles of sand based on Halos... and not much else.
The reverse Halo can hit you too
The Halo Effect works in reverse, naturally, and companies which falter are somehow held to have failed in some way. Rosenzweig explores work done by Barry Staw of University of Illinois. Shaw conducted an experiment in which groups were told – completely at random - they had either performed well or performed poorly on a task. In fact, both had performed exactly the same.
“When told they had performed well, people described their group as having been highly cohesive, with better communication, more openness to change, and superior motivation. When told they performed poorly, they recalled a lack of cohesion, poor communication, and low motivation. Staw concluded that people attribute one set of characteristics to groups they believe to be effective, and a very different set of characteristics to groups they believe are ineffective. That’s the Halo Effect in action.” p54
Schools outside London have borne the brunt of this in the ongoing fall out of the ‘London Effect’, with luminaries such as Sir Michael Wilshaw suggesting that, ‘education chiefs in the worst councils should be stripped of their powers to run school improvement programmes, with their responsibilities handed over to private companies, chains of academies or high-performing local authorities from other areas.’ If London Schools have better ‘performance’, then surely the schools outside London must be somehow ‘less good’ – maybe due to ‘a lack of cohesion, poor communication, and low motivation’. This is, of course, simply a reverse Halo based on attributing negative traits which are believed to contribute to poor ‘performance’ to ‘less good’ schools.
But we wrote a report about successful schools…
A further problem Rosenzweig identifies is endemic in education: Looking for schools which are successful and trying to replicate what is believed to have caused that success. Ofsted is notorious for this kind of thinking, and has produced endless reports based on this delusion, which Rosenzweig calls the ‘Delusion of Connecting the Winning Dots’ because, ‘if all we compare are successful companies, we can connect the dots any way we want but we will never get an accurate picture.’
As he says, “Ask managers why their companies are successful, and we’re likely to get attributions of the sort we’ve seen over and over. There’s no need to fake the analysis because the data are suspect from the start”
If you study a sample made up entirely of outstanding companies, then you create “a sample selection based on the dependent variable – that is, based on outcomes. By only looking at companies that perform well, we can never hope to show what makes them different from companies that perform less well.” p92
Its highly likely, in my view, given Ofsted’s overbearing influence on English schools, that schools which perform ‘less well’ were trying to do the same things that ‘good’ schools were doing, but underlying factors will have affected their ‘data’. Either way, by only ever looking at ‘successful’ schools, the winning dots are connected in ways which ignore the wider picture.
But we have research which shows what makes schools successful…
Ofsted is pretty much a case study in the ‘Delusion of Rigorous Research’. As Rosenzweig makes clear, ‘we can do our best to select samples of high performers and low performers, but if the data are coloured by the Halo Effect, we’ll never know what drives high or low performance; instead we’ll merely find out how high and low performers are described. If your data sources are corrupted by the Halo Effect, it doesn’t matter how much you’ve gathered. You can stack Halos all the way to heaven, but you’ve still only got Halos”. p100/101
As I and others have documented already, Ofsted reports are based on data, and if the data is no good, then the whole pack of cards collapses on close inspection. The way in which Ofsted describes a school simply reflects the Halo which has been cast over the school by dubious data.
But some schools are brilliant aren’t they?
Rosenzweig casts some interesting light on this, too. His ‘Delusion of Lasting Success’ shows that it simply isn’t possible to be top of the heap year after year in a competitive arena.
He found that performance at two thirds of companies in one survey and 50% in another declined in the years following the surveys. “Lasting business success, it turns out, is largely a delusion.” And yet, some schools are consistently rated outstanding by Ofsted and parents. How can this be? Because the underlying factors which drive both the subjective views of Ofsted and parents are fairly constant – highly motivated, high achieving children – and when we attribute performance to what we see in ‘successful’ schools we miss the underlying factors. There is a reason why a school such as Hills Road Sixth Form College sends so many children to Oxbridge, and it clearly isn’t simply what the school does with the pupils in its care.
Halos all the way down, and castles made of sand
The Halo Effect is required reading for those who have an interest in English education. It lays bare the fundamental flaw at the heart of Ofsted, that of attributing positive traits to schools with ‘good results’ and negative traits to those with ‘poor results’ even though the data is flawed and significantly skewed by underlying factors which children bring to school.
And the next time you read an Ofsted report, look at the judgement for achievement first. And read the rest of the report knowing that the Halo Effect of that judgement spreads across just about everything else written in the report. It’s Halos all the way down, and castles built on sand.
The Halo Effect: How Managers let themselves be deceived by Phil Rosenzweig