GenAI – the power of conversation

'We are still very much in the exploratory stage, so while there are established use cases for GenAI, its ROI incorporates potential as well as current value'

Chris Tart-Roberts, Macfarlanes

Identifying the return on investment (ROI) on the latest technology is complicated. Notwithstanding the legal sector’s speedy adoption of generative AI, with firms identifying use cases, building bespoke solutions, and working with start-up and established big tech and legal tech vendors.

Chris Tart-Roberts, head of lawtech and chief knowledge and innovation officer at Macfarlanes, makes three observations: ‘We are still very much in the exploratory stage, so while there are established use cases for GenAI, its ROI incorporates potential as well as current value. So when we onboard a new GenAI tool, we are investing in its potential value too.

‘Secondly, there’s a bit of bifurcation in the way GenAI tech is being leveraged. It supports day-to-day tasks like drafting, text analysis and interpretation, and while it’s only saving each user a small amount of time, the benefits are incremental. So while we can look at usage data, see who is using the tech, and record prompt numbers, measuring value is dependent on user feedback.’

Tart-Roberts notes that ‘GenAI in legal is often used in conjunction with other tech resources, so its value cannot be measured in isolation’.

While some advantages of GenAI are measurable – for example, speeding up document review is likely to increase client satisfaction and retention – others are transformative in the long-term. Deploying GenAI effectively is not just about working differently – it is about changing mindsets and expectations – including within the IT function.

Tipping point

GenAI adoption is reaching a tipping point, but it is also clear that the various tools on offer have specific strengths and use cases. Using them effectively for legal work means putting together solutions from a toolkit of resources.

This is because start-ups have led the way in GenAI – OpenAI’s ChatGPT made it accessible, and Harvey was a first mover in legal – before the established legal tech vendors made it part of firms’ IT infrastructure.

As Karen Jacks, chief technology officer at Bird & Bird, explains, working with start-ups presents challenges and benefits: ‘We have to spend time ensuring that they meet our risk and compliance requirements. And they don’t always understand that adoption is always challenging in a law firm and that we need to test everything properly before we buy.

‘On the other hand, start-ups are more flexible than established vendors. They are willing to listen to user feedback and build it into their product roadmap. Start-ups founded by former lawyers find quick ways of doing the grunt work that they didn’t enjoy. For example, Office & Dragons takes the grunt work out of M&A documents. It saves time, helping to meet deadlines, and reduces write-off.’ So the ROI is both quantitative, in terms of time written off, and qualitative in terms of client satisfaction.

Bird & Bird recently announced a six-month proof of concept trial of GenAI platform Leya, which automates tasks and processes, accessing public and internal data, drafting contracts and reviewing documents. Leya has been rolled out to 400 users across eight countries. ‘It’s a big investment, and the ROI is complex because some benefits are less tangible than others,’ says Jacks. While engagement is not an issue in Bird & Bird, changing mindsets is an important consideration. Shifting from search – which is the underlying concept of machine learning extraction and analysis software – to GenAI’s conversational interface means adding learning and development to the mix, which changes the cost-benefit analysis.

It isn’t a search, it’s a conversation

‘We have combined the Leya roll-out with user education, including prompt training,’ says Jacks. ‘We did this because we want people to use the tools effectively, and not as a search engine. As Max Junestrand [Leya CEO and co-founder] said, this isn’t a search, it’s a conversation, and if you have a conversational mindset you get better results.’

Jacks notes: ‘Our senior lawyers needed to learn to instruct it as they might instruct a junior lawyer, perhaps to read something and write some key bullet points. And when they get the results, they can then ask more questions, unlike search.’

She adds: ‘Our global head of legaltech and innovation, Hélder Santos, organised non-technical training sessions for our lawyers with Uwais Iqbal of Simplexico and Josh Kubicki [professor of legal business and entrepreneurship at Indiana University].’

Michael Kennedy, senior manager, innovation and legal technology at Addleshaw Goddard, has also focused sharply on training and outreach across the firm to promote AI adoption.

The results are impressive. Almost exactly a year after it was launched, AGPT, the firm’s internal GenAI tool, is used by 82% of the firm, including around 60% of partners. There are 1,000 recorded users during a rolling 30-day period.

‘Adoption is better than it has been for previous tech tools,’ he says. ‘We conducted a use case survey which showed a mixture of drafting/redrafting, queries around specific points of law, document review and translation, and we are about to do a relaunch, which will include a series of four presentations, covering all the stats, prompts and use cases.’ As well as developing AGPT, Addleshaw Goddard is using Microsoft Copilot, Thomson Reuters CoCounsel, and start-ups including Harvey, Leya and Spellbook (see below).  

Kennedy and his team also deliver GenAI webinars and workshops to the firm’s clients, covering GenAI essentials, use cases and products, as well as using CoCounsel for document review in live projects. ‘Clients say they want to use [GenAI] for triaging instructions, contract review and approval, playbooks, etc and we help them build out what they need.’

AI agents – are they the future?

AI agents are software systems that can execute and automate multiple tasks, and respond to a series of natural language instructions. Spellbook Associate, launched last month, claims to be the first AI agent for legal work. It can break down projects into plans, execute end-to-end legal work across multiple documents and apps, and even check its own work.

A useful feature is the ability to go back to any point in the workflow and prompt again – a bit like delegating to a junior associate, and then asking them to redo something. However, firms have long been using (less sophisticated) AI to expedite and automate legal processes and workflows. Jacks and Kennedy both consider Leya to be an AI agent in all but name, as its capability includes building and executing a thread of instructions.

Another complexity around GenAI’s ROI is that optimal results are often achieved by combining GenAI and conventional tools. As Tart-Roberts explains, this makes it difficult to calculate the ROI of each separate tool. Furthermore, GenAI is in addition to existing AI tools rather than replacing them.

‘GenAI is fulfilling the old expectation gap,’ adds Kennedy. ‘Machine learning clause extraction tools, like Kira and eBrevia, can identify, say, a termination clause, but the lawyer still has to interpret the output. These latest tools go further along the delivery curve, but they still leave the final analysis to the lawyer.’

As GenAI is not replacing anything it represents additional cost, which is unusual in tech procurement. ‘I might still use Kira for clause extraction, and pull in AGPT, Leya, Harvey or CoCounsel to do the analysis. The challenge is that you have to pay for all of them,’ adds Kennedy.

‘People use AGPT for tasks that aren’t super-complicated, but they are often saving time that is less valuable, or doesn’t get billed. There is also work that doesn’t directly save time but adds value – say, by identifying a problem immediately. Our user survey showed that about 40% of users get value out of AGPT that doesn’t involve saving time.’

It’s not IT support; it’s a relationship

Another indirect outcome of GenAI (and cloud computing, which underpins AI functionality) is the evolution of the law firm IT function from tech support to internal tech consultancy. ‘We have a big IT support team, and as we move [core systems] to the cloud their roles are changing,’ says Jacks.

‘Rather than focusing on helping people use the tools, our approach is to find out what the issue/task is and engineer a toolkit to expedite it,’ she explains.

‘We are working with our IT team on selling the new tools into the business and encouraging support staff to start seeing themselves as IT relationship managers. We have centralised so much that the role of IT is changing from support to relationship management.’