Is there an algorithm to explain U-turns?

If you’re a cynic you might think that the government’s decision to do something about its catastrophic cock-up over algorithms in public examinations was largely driven by Boris Johnson’s desire to start his holiday. However, the back-pedalling will still be welcomed by candidates and schools alike. The interesting question is how we have got into this mess and how both government and Ofqual got it so wrong. It will be fun over the next few weeks to see how each of them blames the other!

It will be news to some people that the government regularly intervenes in the business of examination standards. Since the first Schools Curriculum and Assessment Authority (SCAA) was set up just over thirty years ago it has always had the right to inspect examination standards before results are issued and to comment. These comments are not public but there’s no doubt that they impact and that Ofqual and its predecessors perceive them as instructions.

It is also the case that government has a terrible fear of grade creep which can be summarised as young people doing better in examinations than they ought to. The fear is not of school improvement but of castigation by members of Parliament, vociferous right wing commentators, the right-wing media and the elderly. Bizarrely, this overlooks the fact that those who comment are generally the ones who succeeded in school examinations and went on to positions of power. They are unlikely to see the process which got them there as fundamentally flawed.

In some ways, it is a baffling position to take up. Fear of grade creep got rid of some really good improvements in teaching and learning like General Vocational Qualifications, modular A-levels, practical skills tests and coursework. The final stage, overseen by Michael Gove, was the return to single terminal examination papers taken at the end of the course. He is the least likely to apologise but his decision left no evidence base for 2020 assessments.

Anyway, going back to interventions, the way that Ofqual manages grading gives some hint as to why its behaviour this year has appeared so weird to those outside the process. Each year, and for every subject, the examination boards submit tentative figures to Ofqual along with their proposed grade boundaries. Ofqual aggregates these into what is essentially a national order of merit so that it can see, in theory at least, if one examination board is being lenient with its clients or another one being severe and if the national standard being contemplated is sufficiently in line with previous years to be explainable. That is an odd expression because some people would argue that the huge investment in education in the past few years, the expectation that teachers are better qualified and trained, more on-the-job training and professional qualifications for leaders should be reflected in a rise in standards. It’s a balancing act but, in general, Conservative governments like to be tough on standards and then to be able to berate the Labour opposition for being soft when in power. The Murdoch press and the Daily Mail like this as well!

If Ofqual advises the government that everything is hunky-dory and the maintenance of standards is guaranteed, and the government agrees with this, then the examination boards get the nod to finalise the results. In some subjects they may be told to make minor changes – typically downwards. Although the examining boards have made decisions about grade boundaries on the basis of standards by inspecting genuine scripts there is sufficient leeway within the process for this to happen without the awarding meetings being recalled.

This year therefore, and in line with practice, the examination boards sent Ofqual a series of teacher assessments probably expressed in terms of marks and percentages as in any other year. The Government implied in public and, probably also, private statements that it did not expect a free for all in awards and that standards should be the same this year as in any other. So, when Ofqual discovered that nationally this was not the case the natural tendency was to reach for the algorithm!

As everyone knows, there is nothing fancy about the Ofqual algorithm and the old statement that if you put rubbish in you get rubbish out seems to have held true. What the algorithm tried to do was to compare standards in schools in subjects year by year and compare the 2020 pupil cohort taking examinations with those of other years based on key stage assessments. That sounds a bit fancy but it’s like taking your 11+ score and seeing if it predicts your geography A-level grade! To test the algorithm they ran it on the data from 2019 and discovered it would have been 60% accurate. It seems not to have occurred to them that it would have been 40% inaccurate which is about what has happened this year!

I think some of the above explains the sheer blindness at Ofqual (to some extent shared with the examination boards) in failing to spot the iceberg which was looming up in front of them. If they had only thought about the human level they might have done better because it is worth underlining here that predictions are bread-and-butter in schools. If you have a twelve year old in their second year in secondary school it is quite likely that their report tells you what grade is predicted for them at GCSE four years down the line. So, the A-level student who says they were predicted three A grades is talking about an achievement trail which goes back at least three years, not something conjured out of the air. This year’s teacher assessment known as the Centre Assessed Grade builds on these sort of forecasts. It’s a reliable measure.

We just need to divert to scotch all this nonsense about teachers being overly generous. A teacher assessment indicates what the student will do against exactly the same criteria as the examination but not under examination conditions, stress, nerves and whatever. As an indicator of potential rather than performance it is probably a better guide than an examination. You can have a political position on this but, yes, teacher assessments do deliver higher outcomes.

In the end, Ofqual failed to think things through and then it fudged. It didn’t tell government because it was doing what the government wanted and one suspects that the agency thought it would get away with it. It’s a mess now but not that bad. Young people have had a horrific 2020. Imagine being locked down with your parents when you are eighteen for a whole Spring and Summer if you can! It’s not a happy thought. It’ll be very interesting as well to see whether the 2020 GCSE cohort does better or worse at A Level in 2022. My suspicion is that by the time Ofqual have run a few new algorithms the standard will be almost exactly the same…

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s