Reforms Based on PISA Tests Alone Won't Fix GCSE Standards

PISA test documents (Photo credit: Wikipedia)

by Jo-Anne Baird, University of Oxford

With the creeping rise of exam results over the past few decades, many have questioned whether standards are really as high as they were in the past.

More worrying still is whether pupils in the UK can maintain pace in educational standards compared with other countries.

After all, we operate in a global knowledge economy, in which the competition is tough in terms of supply of workers for top-end companies.

And so it is in this climate that Michael Gove has announced changes to GCSEs that will toughen the standards to reflect that of high-performing countries.

International tests like the Programme for International Student Assessment (PISA), run by the Organisation for Economic Cooperation and Development (OECD) are used to compare the performance of students. PISA mainly tests reading, mathematics and science, but it has recently also tested problem solving.

In fact, the International Association for the Evaluation of Educational Achievement (IEA) run other tests in mathematics, science and literacy, but Gove rarely quotes their findings because we have done rather better on those tests than on the PISA tests and this does not suit his political narrative.

Ofqual, the examinations regulator, has indicated that the new, tougher GCSEs will have a numbered scale instead of letter grades and that grade 5 will be internationally benchmarked. Grade 4 will be equivalent to the old grade C at GCSE, with grade 9 being the highest grade.

We can look at examination standards in a couple of ways - in terms of what people have to do to get the grades (content standards) or how many attain the grades.

Ofqual conducted some research comparing the content standards of A-level equivalents across countries and A-levels came out pretty well, but the same exercise has not yet been carried out for GCSEs.

In any case, we can safely assume that a new Ofqual consultation on its proposals means that fewer students are expected to get grade 5 or above in the toughened GCSEs of the future.

So who are these high performers that we need to benchmark with? Taking the PISA 2012 results, the top five performing jurisdictions in mathematics were Shanghai-China, Singapore, Hong Kong-China, Chinese Taipei and Korea.

Japan and Finland are also regular top performers in these league tables. In comparison, the UK generally performs very averagely in PISA, although we typically do better than average in science and did well in problem solving. As we invest an average amount in education compared with the other countries included in these studies, our average results are a typical return on investment.

One issue for the government will be just which of these high performers we want to set our new standards by. For the sake of argument, let’s choose Shanghai-China and see where that takes us.

The difference between our average score in mathematics and Shanghai-China’s was 119 points for the 2012 PISA test. In a little-known 2011 department for education (DfE) report, a DfE researcher had a go at translating that into what would need to change in terms of GCSE point scores.

To match Shanghai-China’s outcomes, a student with 8 grade C GCSEs would need to pull her socks up to 3 A* and 5 grade As; an improvement of 19 grades overall.

One could quibble with the methodology of that paper, but however we look at the outcomes, it looks like we would have to raise the bar quite a lot to match the maths performances of 15-year-olds in Shanghai.

Or would we? Can we rely on PISA to tell us this? It turns out that comparing educational outcomes across nations is not as easy as pie. There are many controversies about the tests and how they are conducted, including the effects of test translation, how motivated the students are and how well the tests match the curriculum.

To make the comparisons you have to be pretty sure that you have a representative sample of students from each country. OECD’s Andreas Schleicher has recently admitted to Parliament’s Education Select Committee that the PISA coverage of 15 year-olds in Shanghai was only 73%, whilst the figure is more like 90% in other countries.

Tom Loveless at the Brookings Institute in the US has explained how poor, migrant workers originating from rural China have trouble getting their children into school in Shanghai.

So it is highly likely that the missing 27% are children from economic groups that we know do worst at school, statistically speaking.

So what should we make of this? Do we want higher standards in our education system? Of course. It is worth looking to these international tests for the information they provide, but they should not be the only measure Ofqual uses to set its new standards.


Jo-Anne Baird gives assessment consultancy advice to, and receives research funding from, governments and commercial organisations, including Ofqual and Pearson.

This article was originally published on The Conversation. Read the original article.