Some real data on what happens when lawyers use AI

Here's the thing about legal AI tools - everyone's making big claims, but there's surprisingly little objective data out there.

Most data you see comes from the vendors themselves (at DraftPilot, we’re guilty of this too!).

E.g. "10x faster this", "80% more efficient than." It's not that these numbers are wrong, but they often come from controlled tests or cherry-picked examples.

But something interesting just happened. Axiom, who provides interim lawyers to in-house teams globally, ran what might be the most objective test of legal AI we've seen yet.

They ran a structured 8-week pilot where their lawyers used our tool (DraftPilot) on real client work (across 27 different in-house legal teams), followed by detailed surveys and interviews.

What makes this data different? The lawyers were working on actual client contracts, not test scenarios.

And importantly, this wasn't a vendor-run study (we didn’t run the questionnaires etc) - it was Axiom gathering feedback from their own lawyers to decide if the tool was worth rolling out more broadly.

I’ll summarise the findings, but if you’re curious you can find their full report here:

1. Speed gains were real, but varied by task

• Creating playbooks and summarizing contracts saw the biggest gains (60%+ time savings). One lawyer noted: "The summary feature cut a task that would take approx. five days down to approx. one day."

• Drafting redlines saw 40% time savings, although a few users mentioned bigger savings on this, eg seeing time spent on redlines drop from 2 hours to 30 minutes

  Here are a few of the stats on time savings:

  

  

2. Work quality was the surprise

• 89% of lawyers reported quality improvements

• Most lawyers rated the output as comparable to a mid-level associate

• The consistency of output was highlighted repeatedly

• Lawyers noted it often caught things they might have missed

What I found particularly interesting was the comment: "It suggests things that wouldn't have crossed my mind."

This matches what we've seen elsewhere - AI isn't just about speed, it can actually enhance the quality of legal work by bringing more perspectives to the table.

3. The "Goldilocks" use cases emerged clearly

Not every legal task benefited equally. The sweet spots were:

• Reviewing third-party contracts

• Creating or updating playbooks

• Summarizing agreements

• Generating pushback narratives

This makes sense - these are tasks that require both pattern recognition (which AI is good at) and traditionally take quite a lot of time.

4. The 5-minute implementation surprise

This was fascinating - one of the biggest unlocks had to do with onboarding / implementation. DraftPilot has the ability to generate a contract playbook instantly by just pasting in a template agreement.

This is a feature that most people like the look of, but isn’t seen as such a huge deal when we demo it. But we learned that in practice it often ends up being the difference between a lawyer adopting the tool or not.

This reminds me of something music producer Rick Rubin once said: "Never try to judge an idea based on the description of the idea. Always musically try the idea."

His point was that there's often a gap between how something sounds when described versus how it works in practice.

That's exactly what happened here. When you describe the feature - "paste a template contract, get a playbook" - it sounds.. a bit basic?

But in practice, it solved a big pain point for interim lawyers: they don't have weeks to spend implementing tools or building playbooks from scratch when they are dropped into a new client environment.

Instead of requiring hours of setup and manually creating detailed rules, lawyers could start reviewing actual client contracts within 5 minutes of getting access. As one lawyer put it: "I saved four hours using this to create a playbook last night."

(If you’re curious, you can see how this instant playbook creation works in this 3-minute demo video.)

What does this mean for in-house teams?

Given these results, Axiom has decided to roll out DraftPilot to their lawyers globally and we’re obviously really excited about that!

But I think the bigger point here is that we're finally getting real data about how AI performs in actual legal practice. Not vendor claims, or based on vendor demos, but feedback from practicing lawyers doing real work for real clients.

The picture that emerges is nuanced: AI isn't magic, and it won't revolutionize everything overnight. But for contracting work, it can deliver meaningful improvements in both speed and quality.

Thanks for being here,

Daniel

CEO at DraftPilot
LinkedIn