ChatGPT tested on common conveyancing questions

A widely publicised AI chatbot service gave factually correct, but also confusing, incorrect or contradictory responses to basic conveyancing questions, a law firm has found.

East Midlands firm Bird & Co wanted to see if ChatGPT, an example of so-called 'generative AI', could answer the most common first-time buyer queries accurately. 

Several major firms, including magic circle practice Allen & Overy, are already implementing systems based on generative AI. Legal futurologist Professor Richard Susskind last night described the technology as the most impressive AI development in 40 years. 

At Bird & Co, ChatGPT was asked the definition of a first-time buyer as well as questions about mortgages, first-time buyer schemes and stamp duty land tax.

On the definition of a first-time buyer, ChatGPT replied: ‘A first-time buyer is a person who is purchasing a home or property for the first time. This term is commonly used in the real estate industry to refer to individuals who have never before owned a property, or who have not owned a property in the recent past.

‘First time buyers are often eligible for special programs or incentives offered by governments, lenders, or real estate agents to assist them in purchasing their first home. These programs may include lower interest rates, down payment assistance, or tax credits.’

Bird & Co partner Daniel Chard said the service provided a ‘great in-depth answer’. He also praised the service’s response to whether first-time buyers are guaranteed a mortgage.

However, other responses fared less well. A question on how much stamp duty a first-time buyer has to pay was incomplete, out of date and lacked explanation on the specifics, Chard said. A ‘detailed and informative’ but also ‘somewhat contradictory’ answer was given to a question on help-to-buy ISAs.

Chard concluded that ChatGPT provided what looked like detailed and intricate answers but in many cases, the details were not always accurate and the information was often reductive.

He said: ‘When we heard about ChatGPT, we had no choice but to jump on the bandwagon and ask it questions to compare to how property professionals would respond. On the whole, we were impressed with elements of its capabilities in terms of answering first-time buyer queries. It is clear to see how ChatGPT could be disruptive for many sectors, including the legal sector as it improves.

‘However, the AI chat has many limitations in terms of giving accurate legal advice. These limitations include providing incorrect information on occasion, providing basic answers to complex queries, and failing to provide tailored responses. ChatGPT does accept there are limitations to the AI tool.’

The concern for solicitors, Chard said, ‘lies with the fact it is a completely free tool to use, which is 100% cheaper than going through a solicitor. It appears to provide answers that inexperienced people could believe to be factually correct’. But he also suggested that, as AI technology becomes more sophisticated, solicitors could use it to reduce their workload, for instance answering very basic queries.

This article is now closed for comment.