AI's mixed bag: groundbreaking study reveals selective impact on legal work
This recent study by the University of Minnesota Law School was set up as a randomised controlled trial, where sixty law students were asked to complete four different legal tasks, either with or without GPT-4.
The researchers then blind-graded the results (quality) and measured how long it took (speed). Interestingly, they also compared the benefits of AI for both ‘skilled’ and ‘lower-skilled’ students to see if there was a difference. Spoiler: the benefits weren’t equally distributed!
The four tasks were:
drafting a complaint
drafting a short 2 page contract from scratch
creating a section of an employee handbook
drafting a client memo
The findings
Speed: The students who used GPT-4 completed tasks more quickly than those who didn’t. This was measurable across all tasks, and all experience levels.
Impact of using ChatGPT4 on speed of drafting a contract Quality: Interestingly, when it came to the quality of the output, using GPT4 didn’t really increase the quality by a lot, except for lower skilled students. Students who on other tests (not using GPT4) scored lower than their peers, saw the biggest boost in quality.
Another way to look at these findings is that experienced lawyers will mostly see speed improvements as opposed to quality improvements.
But that inexperienced junior lawyers will be able to punch above their weight in terms of quality too by using AI tools.
Implications
I believe the implications of this are:
Junior lawyers: Junior lawyers will be able to take on more legal tasks and to a higher standard than before. It’ll make more sense to increase the % of junior lawyer on any team.
Law firm billing model: Given all tasks will take less time, law firms should move to billing on fixed fees vs hourly rates.
In-house leverage: In-house legal teams will have more leverage to negotiate law firms down on price. The “we’ll otherwise do the work in-house” alternative is more believable. After all, generalist in-house lawyers often outsource work they’re not specialised in. But now AI tools could help them plug that gap.
Experienced in-house lawyers should spend as much of their time as possible on tasks that require “in the moment” expertise. Things like:
brainstorming options with a commercial colleague
live negotiations on sizeable deals
higher level risk management (including setting fallback positions for contract negotiations)
being the legal voice in company wide strategy sessions
management of junior team members + hiring
These are all tasks that junior lawyers with AI won’t be able to replicate to a high standard that easily.
Thanks for being here,
Daniel