News focus: Algorithms and 'Schrödinger’s justice'

Inside a sealed black box is a decision involving the law. It might be whether a civil case is worth pursuing, whether an alleged offender is eligible for bail or even whether they are innocent or guilty. As with the fate of the cat in the renowned thought experiment in quantum physics, we can discover which way the judicial die has been cast simply by opening the box. Call it Schrödinger’s justice.

What cannot be discovered is the logical route by which the decision was reached. The reasoning is locked inside a proprietary algorithm driving an artificial intelligence (AI) computer program.

How can we trust it? The scenario is not science fiction. In the US, risk-assessment systems to make decisions about whether accused offenders should be eligible for bail are widely used. They rely on algorithmic calculations to predict whether an accused person is likely to reoffend, or to fail to turn up for the trial –  classifying them as low-, medium- and high-risk. In theory, the scoring is based on objective facts about each offender and is scrupulously race-neutral.

However, a study published last year of decisions made by one such system, Correctional Offender Management for Alternative Sanctions (COMPAS), found a worrying apparent bias. An examination of the cases of more than 7,000 people arrested in Broward County, which covers Fort Lauderdale, Florida, found that in assessments that turned out to be mistaken, black people were almost twice as likely as white people to be falsely labelled at high-risk. White people were more likely to be mislabelled as ‘low-risk’.

In a 37-page rebuttal, the company behind the COMPAS system, Northpointe, vigorously denied any bias. It says that the apparent racial imbalance is a natural consequence of using unbiased scoring rules for groups that happen to have different distributions of scores. Black people in Florida are disproportionately likely to tick the risk-box of ‘homeless’, for example.

But a British expert on crime data, Professor Allan Brimicombe of the University of East London, says this is exactly the problem. Of the data items used to score risk, 65-70% ‘would strongly correlate with race in America’, he told the annual conference on Trust, Risk, Information and the Law (TRIL) last week. And this leads to another problem: the COMPAS system, like many of its competitors, relies on a proprietary algorithm. It is in effect a ‘black box’ whose decisions cannot be challenged in court.

With recent breakthroughs in AI, the question of when a black box can be trusted is becoming ever more pertinent. It features in many debates about driverless cars: in an instance of the famous ‘trolley problem’ in ethics, should a driverless car be programmed to sacrifice its passengers to avoid ploughing into a line of schoolchildren? But the debate also needs to be had in the context of the law, as interest in AI grows in criminal justice and other parts of the legal system.

Hence the focus of this year’s TRIL conference on issues raised by ‘artificial and de-personalised decision-making, machine learning, artificial intelligence and drones’.

So far the UK approach to ‘artificial and de-personalised decision-making’ in criminal justice has been more cautious than in the US. Sheena Urwin, head of criminal justice at Durham Constabulary, laid heavy stress on transparency and careful evaluation when she described a system currently being tested for its ability to assess risks of reoffending.

The constabulary is carrying out a randomised controlled trial of a ‘harm assessment risk tool’ to see if it agrees with custody sergeants’ assessments of which offenders are at medium risk of reoffending within two years. The idea is to identify individuals who might be eligible for the force’s Checkpoint programme, which offers people such as long-term drug offenders a good behaviour contract as an alternative to prosecution. The programme is available to people judged as at ‘moderate risk’ of reoffending; a strong concern is to avoid ‘false negatives’ – offenders classified as low-risk who go on to commit crimes.

‘We still have some way to go,’ Urwin told the conference, saying that results of the trial would be published in a peer-reviewed journal. And she stressed that the risk-assessment system will not replace human custody sergeants. ‘This is a decision-support tool, not a decision-maker,’ Urwin said.

On the general question of trust in black boxes, the conference heard from a research project called UnBias, which is attempting to come up with  proposals to ensure ‘fair’ algorithms in modern life. The usual techie answer, to require that systems are based on open source-code, is not sufficient, the project’s leader Dr Ansgar Koene of the University of Nottingham said. Transparency ‘needs to be meaningful’. One possibility would be to require key algorithms used in public life to be independently certified as trustworthy, but that might be seen as merely kicking the problem upstairs. Which robot will guard the robot guardians?

Brimicombe, meanwhile, warned that total transparency could be a double-edged sword, with algorithms being challenged line by line ‘to the point where the courts no longer function’.

Whatever the answer, the search for ways of trusting black boxes is bound to involve lawyers – especially as the legal profession is itself on the cusp of its own AI revolution. The conference also heard from one of the UK’s leading academic experts in the burgeoning field, Professor Katie Atkinson, head of computer science at the University of Liverpool. Atkinson’s research on modelling legal reasoning made headlines in the legal world two years ago when fixed-price legal services business Riverview Law announced a collaboration with the department to find ways of applying AI to legal processes. Atkinson hinted that the first results of the collaboration would be announced at the world’s premier conference on AI and the law, which comes to London next month.

However, she conceded that there are ‘some challenges’ to AI being accepted in legal practice. ‘It needs buy-in from end-users,’ she said. In particular, academic research needs tailoring to business-specific problems. Another challenge is getting the right data into the black boxes in the first place. ‘The output is only as good as the data,’ she emphasised. If the future is to be Schrödinger’s justice, this seems a good place to start.