Last week, the Gazette carried an article about ChatGPT’s potential for use in legal services. ChatGPT – for those who missed the article – is an AI text generator tool, or chatbot, developed by San Francisco-based OpenAI. As chatbots go, ChatGPT is more University Challenge material than many of the frustratingly limited ones sometimes encountered on airline or retail websites, but how smart is it? Can it do things lawyers do, like applying law to the facts?
Joanna Goodman reported the performance of ChatGPT when subjected to questions from the US Multistate Bar Examination (it got about 50%). She then pondered, ‘It would be interesting to conduct the same experiment with the multiple-choice SQE1 (Solicitors Qualifying Examination)’. This article is about my attempt at that experiment.
The SQE was launched by the Solicitors Regulation Authority (SRA) in September 2021 with the aim of providing a standardised assessment process forming part of the new route to qualification as a solicitor in England and Wales. The SQE is comprised of a functioning legal knowledge part (SQE1) assessed by multiple-choice questions (MCQs) and a practical legal skills and knowledge element (SQE2). In April 2022, the SRA published 90 sample SQE1 questions and these are available on its website. In my experiment I posed each of these 90 questions to ChatGPT using the Free Research Preview, 9 January, version. Before I reveal its score, here are a few observations.
The chatbot worked out it was being asked MCQs and, except on a couple of occasions, always replied first with the answer expressed as one of the 5 responses labelled A to E. In addition to the A to E answer it would also usually give an explanation of why that answer is correct. For example, when asked about the legal implications of a man tearing up his will (question 88), it rightly said this action revoked the will and added that the intestacy rules would thereafter apply: an extra mark for the smart bot at the back of the class. It also knew it was being asked questions about English and Welsh law and limited its elaborations to relevant material from those jurisdictions.
A few questions were submitted a second time to see if ChatGPT returned the same answer. Interestingly, when asked to work out a solicitor’s professional fees (question 22), on the first attempt it simply stated the wrong answer, whereas the second time round it got the correct answer and, impressively, showed its working out. However, there were plenty of occasions when its conclusion or its reasoning, or both, were wrong.
So, how did it do? It scored a pretty decent 50% (45 out of 90), mirroring its US Bar Exam score and only narrowly failing to pass SQE1. The passing score for the November 2021 SQE1 sitting was 57%, with a 53% pass rate. The passing score for the July 2022 sitting was 56%, with a 53% pass rate. Students in both sittings who had a third class undergraduate degree averaged scores in the high 40s and those with a 2:2 averaged in the low 50s. ChatGPT is currently performing around that level but it is learning all the time. Bearing in mind that SQE1 questions require candidates not only to know the law but also to understand and apply it to novel fact scenarios, shows how far AI has come in recent years. Another attempt at SQE1 might well see ChatGPT on its way to being admitted to the roll.
Dr Andrew Gilbert is a non-practising solicitor and senior lecturer at The Open University Law School
4 Readers' comments