Artificial intelligence enthusiasts claimed a new legal sector breakthrough today by announcing that a computer program had beaten human lawyers in a direct competition to predict the outcome of civil claims. 

The week-long Lawyer Challenge pitted 112 lawyers against software produced by a Cambridge start-up company, CaseCrunch, in assessing PPI mis-selling claims. According to CaseCrunch, the software predicted outcomes with an accuracy of 86%, while the lawyers' average was 62.3%. 

While the company commented that it 'could not be happier about the outcome', it was careful to stress it did not see lawyers as enemies. 'We are not necessarily adversaries in this game, the systems like ours can make the legal world more effective for everyone,' managing director Jozef Maruscak said. 

The challenge involved clearly defined sets of data and binary decisions. Participants were presented with 775 factual scenarios of PPI mis-selling claims, and asked to predict 'yes or no' as to whether the financial ombudsman would succeed in the claim. The same factual scenarios were given to CaseCrunch.

Ludwig Bull

Ludwig Bull

It is the latest in several attempts to demonstrate AI's ability to predict case outcomes. Last autumn, researchers at University College London, the University of Sheffield and the University of Pennsylvania said they had predicted judicial decisions of the European Court of Human Rights with 79% accuracy. However, the CaseCrunch challenge may have been the first to put human expert lawyers head to head with AI. 

A technology judge and a legal judge independently verified the fairness of the competition. The legal judge was University of Cambridge lecturer Dr Felix Steffek; the technical judge was Ian Dodd, UK director of legal analytics company Premonition. 

CaseCrunch scientific director Ludwig Bull said that the results do not mean that machines are generally better than human lawyers at predicting outcomes. Rather, they show that 'if the question is defined precisely, machines are able to compete with and sometimes outperform human lawyers. The use case for these systems is clear. Legal decision prediction systems like ours can solve legal bottlenecks within organisations permanently and reliably'.