Generative AI: a jetpack for legal

The last few weeks have seen a huge amount of speculation across the tech industry – including legal tech – around OpenAI’s ChatGPT, an AI text generator based on the company’s GPT-3.5 (Generative Pretrained Transformer) language processing model that was released on 30 November 2022.

ChatGPT is a large language model (LLM) system that writes short responses to pretty much any question you ask it. It can also write technical articles, scripts and poems and generate computer code. You can try the (free) public version here.

ChatGPT was trained using massive text databases from the internet (around 300 billion words) and Reinforcement Learning from Human Feedback (RLHF), where human trainers input questions and assess/correct/rank the output. The model has been fine-tuned multiple times. It continues to learn from experience and feedback, so it is constantly improving.

ChatGPT creates text in response to prompts. If you ask it a specific question it generates a clear, coherent answer, which means it is easily aligned to exam questions and coursework. You can also ask it to rewrite existing text. In the Guardian last week, Alex Hern wrote that academic institutions were already so concerned about AI-generated plagiarism, that they are working on reliable ways to identify AI-generated content and exclude it. And OpenAI was looking at watermarking its output.

However, although ChatGPT creates cogent, rational responses, because it is probabilistic – that is, it predicts the next word in a sentence based on words and phrases that are commonly linked together, and previous feedback – its output is not always factually correct. Sam Altman, co-founder and CEO of OpenAI, tweeted: ‘ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness. It’s a mistake to be relying on it for anything important right now. It’s a preview of progress; we have lots of work to do on robustness and truthfulness.’

One might infer from this that generative AI has limited application to legal, so why has it created such a buzz in the legal and legal tech community?

Legalese and the Bar Exam

First, it addresses head-on the fundamental challenge for legal technology: the complexity of legal language, with its long sentences and context-specific definitions. Understanding legal language and terminology is also a key part of legal education. This realisation led serial legal tech entrepreneurs Daniel Katz, professor of law at Chicago-Kent College of Law (Illinois Tech), and Michael Bommarito, adjunct professor at Michigan State University College of Law, to conduct research. This resulted in their draft paper ‘GTP Takes the Bar Exam’, which documents their evaluation of the performance of OpenAI’s GPT-3.5 – they used the commercial API (application programming interface) version – in the multiple-choice section of the Multistate Bar Examination (the Bar Exam). This uses scenario-based questions designed to test legal knowledge and comprehension.

Without fine-tuning the model, GPT-3.5’s first choice answers scored over 50% in three categories in the Bar Exam, and close to 50% in a fourth category. Katz and Bommarito repeated the exercise with previous GPT models. They discovered that in under three years, large language models have progressed from achieving scores that were indistinguishable from random chance to near parity with humans in at least two categories. The conclusion was that a future GPT or similar LLM is on track to meet or exceed human performance within one to two model iterations. It would be interesting to conduct the same experiment with the multiple-choice SQE1 (Solicitors Qualifying Examination).

Legal tech use cases

I asked ChatGPT, ‘How can ChatGPT be used in legal services?’ it responded with four use cases that are already covered by current legal tech solutions:

‘ChatGPT is a variant of the GPT (Generative Pretrained Transformer) language model that is specifically designed for use in chat applications. It is trained to generate human-like text based on a given prompt and can be used in a variety of applications, including in legal services. Some possible ways that ChatGPT could be used in legal services include:

1. Providing legal information: ChatGPT could be used to provide general information about legal topics and help people understand their legal rights and options.

2. Answering legal questions: ChatGPT could be used to answer specific legal questions and provide guidance on legal issues.

3. Generating legal documents: ChatGPT could be used to generate legal documents, such as contracts or legal notices, based on user input and pre-defined templates.

4. Providing legal analysis: ChatGPT could be used to analyse legal issues and provide recommendations or opinions on legal courses of action.

It’s important to note that ChatGPT is a tool and should not be relied upon as a substitute for the advice of a qualified lawyer.’

I then asked for advice on common consumer legal issues: how to get a landlord to make repairs to a rented apartment and how to resolve issues with noisy neighbours. The response to each question offered sensible ways to negotiate resolution and suggested formal legal action only as a last resort.

While Katz and Bommarito did not train GPT-3.5 on the preparatory materials for the bar exam, their study highlights the rate at which the technology is improving. And use cases such as DoNotPay’s integration of GPT and voice technology (see box) show that generative AI can be incorporated into process automation in ways that make the concept of a ‘robot lawyer’, or at least an AI legal assistant, a real possibility.

GPT-3 for contract drafting

In commercial legal services, GTP-3 has the advantage of integration with Microsoft applications. Microsoft invested $1bn in OpenAI in 2019 and is in talks to invest a further $10bn.

 In December, Seattle-based legal tech startup Lexion launched AI Contract Assist, a GPT-3 plug-in for Word which helps to draft and redline legal documents. CTO co-founder Emad Elwany worked for Microsoft Research on NLP (natural language processing) enterprise AI applications before building Lexion’s AI capabilities at the Allen Institute for AI in Seattle. Contract Assist is the only Lexion AI feature that uses third-party AI – the same GPT API that Katz and Bommarito evaluated in their Bar Exam experiment.

Contract Assist works in a similar way to ChatGPT – you type a prompt into the query box and it creates a suggested clause or amendment. Elwany explains: ‘The feature looks at the surrounding context within a contract, the requested prompt from the user, and some prompts and hyperparameters that we optimise it with to provide an answer. It doesn’t currently learn within the same session, but we are running experiments to train it on [a company’s] entire corpus of contracts, negotiated redlines, and the activity around them, to see how well it starts to mimic [a] company’s contracting process.’

Elwany believes that the Microsoft integration will accelerate the take-up of generative AI for legal. ‘Legal professionals are devoted to Microsoft Office products. Like any well-designed product, the user experience has to meet users where they already work. If lawyers had to visit a webpage to get suggestions, they would be far less likely to use them, but they live in Word, and delivering value there is the most useful.’

Having experimented with GPT-3.5 and its predecessors, Katz believes that ‘while there has not been a deep exploration of what problems it is good at, and what it is not good at, as long as you keep a human in the loop, it can be applied to many everyday legal problems’. He is working on the possibility of running technologies similar to GPT behind the firm’s firewall, to overcome potential security and confidentiality issues.

Jetpack for legal

Generative AI involves working differently – using a series of prompts in an iterative way – and its output still needs to be checked by a human to overcome concerns around factual and numerical accuracy as well as explainability, liability and security.

But as Katz put it, it opens up opportunities to ‘jetpack’ legal processes, beyond what can be achieved by current AI applications. ‘Although it won’t immediately make current legal tech offerings obsolete, generative AI is the next step, and incumbent vendors will need to work out how to bring it into their portfolio – or buy a company that will teach them,’ he says. Launching ChatGPT over the holidays was a genius marketing strategy on the part of OpenAI, he adds.

In a discussion hosted by Professor Daniel Linna of the Northwestern University Law and Technology Initiative, Professor Kris Hammond wondered whether law firms will need to employ prompt engineers to help lawyers develop the right prompts to collaborate with generative AI in a way that adds value to their work.