Academics love games. We can’t resist playing them – but the Research Excellence Framework (REF) is our favourite. As the results of the UK’s 2014 assessment of university research are digested, academics have been torn between worry at how damaging this huge waste of time and money is, and pride that their university has done well. The point is that playing the REF game changes our behaviour. That is of course the desired consequence – better research, environment and now impact. But there are unintended consequences.

From the outset, concerns were raised on the effect of the REF on interdisciplinary research that crosses different subjects. There were many concerns voiced following the previous assessment of research in 2008 that the highly discipline-focused judging panels found it difficult to favourably assess research that spanned a number of subject areas. Asking more than one panel to review interdisciplinary research outputs didn’t help – fragmented pieces never look impressive.

“Many high profile academics argue that since “interdisciplinary research often lacks rigour”, it turns into a “safe-haven for the lesser-talented”, suggested Ismael Rafols of Sussex University’s Science and Technology Policy Research Unit (SPRU) in a 2012 blog post. The SPRU team published a paper showing the negative effect that use of journal citation data would have on interdisciplinary work within business and innovation research.

Changing the incentives

Incentives that urge us to leave the borderlands of our disciplines and dive for the safe burrows of their heartlands are very worrying, as my colleague Veronica Strang and I have argued recently. Our most pressing global challenges of energy, sustainability and health security are all inherently interdisciplinary. The conversation between disciplines should really be conceived as the heart of a healthy academia, not its periphery.

In response to criticism, the Higher Education Funding Council for England (HEFCE), which runs the REF, decided to take four new measures to support interdisciplinary research in 2014. The fewer, broader, units of subject assessment would contain a wider range of expertise and additional assessors with experience in interdisciplinary research would be co-opted to panels. Outputs could also be labelled as interdisciplinary and rather more care would be taken in cross-referring between subject panels.

In addition to this, probably the greatest positive incentive in support of interdisciplinary research was the inclusion of “impact” outside of academia in the way the REF 2014 was calculated. Not only would many public benefits of research be naturally interdisciplinary, but resulting case studies could in principle be submitted under all the relevant subject panels – and contributing as an entire case study for each one.

Signs of success

The question is whether these measures have worked. At first glance, analysis of the results suggest that they have. Across the REF results, 20% of research was marked as four-star or “world-leading” – whether it was marked as “interdisciplinary” or not. This was the same across the spectrum of research, as the graph below shows.

CC BY

HEFCE policy adviser Alex Herbert has written: “Panels observed increasing trends towards inter- and multi-disciplinary research and recognised excellence in these outputs.” To take one example, the English panel this time contained members who covered linguistics, philology, film and theatre, science, medicine and philosophy. They felt much more able to evaluate radical new interdisciplinary research such as in medical humanities, and noted that a huge increase in creative writings opened up a panorama of subject matter.

Initial analysis of the impact case studies undertaken by Digital Science and Kings Policy Institute (who will publish a full analysis in March 2015) suggests that 70% of impact case studies were interdisciplinary. This is encouraging, but until we know more, it is not enough. Claiming that research is interdisciplinary is not the same as doing it.

The truly interdisciplinary impact case studies will reveal themselves by appearing in more than one subject of assessment – the Kings Policy study needs to examine the numbers closely. Qualitative evidence from panels also needs to be garnered while the experience is fresh in their memory.

Clearer picture still to emerge

We need to know whether the cross-referencing and advice from experts outside the subject area really did help panels to assess whether a piece of interdisciplinary research represented a whole greater than the sum of its parts. Other questions need to be answered, such as whether the process gave panels a better idea of how to judge the quality of interdisciplinary research and whether the new measures encouraged submissions that would have been judged too risky in the former exercise. It will also be important to see how the panels evaluated a university’s support of interdisciplinary research in their awarding of scores under the “environment” part of the assessment.

Beyond these immediate analyses we need to have a deeper conversation. There is an inherent tension now within the REF between the interdisciplinary nature of research impact and the entrenched framing of the whole exercise in subject silos. Even if the REF has achieved as balanced a position as it can, we know that academic journals, research funders, and university human resources teams all find the evaluation of interdisciplinary research challenging.

We need to look at everything that motivates our daily game-playing to ensure that it supports what we value the most: deep scholarship and learning, fresh understanding of ourselves and the world, and imaginative ways forward driven by a creative academy serving our communities.

Click here for more coverage of the Research Excellence Framework and a Q&A about why it is so important for universities.

Tom McLeish receives funding from the Arts and Humanities Research Council, Engineering and Physical Sciences Research Council, Procter&Gamble and the Templeton Foundation.