Anthropic just launched a Word add-in for Claude, and the legal tech world is treating it like the end of purpose-built legal AI.
Watch today to see what most people missed, including the Easter eggs I found, the capabilities and gaps, and what it all means for lawyers and legal tech builders.
00:00Hi, I'm Anna, a lawyer and the founder of Legal Benchmarks. Today, I am reacting to the demo video that Anthropic recently launched on its Claude Microsoft Word plugin. Every Claude announcement recently feels like it's dropping an atomic bomb into the legal tech world with a lot of people chatting death to legal tech and rooting for Anthropic. But, I don't think people are actually looking close enough at what the demo actually contains. I will go through the demo
00:31through the lens of a lawyer that has drafted, reviewed, and redlined hundreds of contracts as both a private practice and in-house lawyer. You'll definitely want to stay until the end because I will share my take on what this means for legal teams that are deciding between buying a general-purpose solution or going with a purpose-built solution. And also, for the legal tech builders out there that's wondering whether all of these announcements means it's doom and gloom for them.
01:02Spoiler, I don't think it does. But, enough talking, let's dive into the demo. Okay, pausing here. First of all, I love the funky music that Anthropic always uses in their demos and their ads. I can feel myself like grooving to it. It also fits the cool kids vibe of Anthropic
01:32compared to, you know, OpenAI, which has a very different vibe. Of which I will not name the vibe here. Anthropic is definitely dropping us an Easter egg in this first demo page. If you just look at the top right corner, you'll see in the top toolbar is a Copilot add-in. I see this as an open dig at Microsoft. Many companies have Copilot installed but are still adding Claude because Copilot, even though it
02:02is native to Microsoft, isn't good enough for what a lawyer needs. And so, therefore, they're installing OpenClaude. Okay, is this a new model that only the Anthropic team has access to OpenClaude and it's too powerful to be released in the public? That's why we have Claude, but they have OpenClaude. Or is this like a challenger to OpenClaude? I'm not sure whether this is intentional or not. Let's take a look at the demo doc. And a few things immediately jump out from a
02:33lawyer's perspective that is a bit bizarre. The first is, if you look at the file name, this is the fourth version of the draft mutual NDA between Acme and Vertex, which are two companies. However, there is a law firm name, Morrison and Calloway LLP, in the header of the mutual NDA. This is just extremely bizarre because if it's two companies that's negotiating a mutual NDA, they would never use a
03:03template that has a law firm's header. Having this law firm name in the header makes this document feel unrealistic to me. And then, just the overall vibe of the document feels very synthetic to me. Like, I'm 99% sure that this is a AI-generated document that's meant to look like an NDA, but it isn't really an NDA. I'm 1% speculating that could it be potentially Anthropic's own internal mutual NDA template?
03:33Would it be possible? Could be. So, what is being demoed here is actually a more sophisticated use case compared to what they shared before, which was just simple contract review because of two reasons. The first is Claude is right now in Word, which is an environment that lawyers usually open and analyze contracts in. So, that's one major difference. The second difference is it is not only analyzing a document and reviewing it and highlighting risks, it is reviewing a redlined document. So,
04:06the use case is now a lot more tailored to a lawyer. Let me go. Let's look at the proposed redlines themselves because all the legal tech solutions out there seem to offer the same features. Every solution does
04:36redlining, they do review, they do drafting, they do legal research. But, the reality of the situation is just because you claim you have a capability doesn't mean perform well on those capabilities. And that's exactly the reason why we started Legal Benchmarks, which is to measure the claims of those capabilities from the different AI solutions out there for the legal function. So, let's take a look at the two proposed redlines. What draws my attention is the proposed redline to section five. One of the risk flags earlier from Claude was that the
05:07counterparty removed the 500k liability cap and then asked for a limited liability, which isn't clear in this draft at all. It's not shown. But, when I did take a deeper look, you can see that the first sentence is probably original draft, which includes a 500k liability cap, whereas the second sentence is probably what was added by the counterparty. And it says, "Liability under this agreement shall be unlimited and subject to the prevailing parties' discretion."
05:37Okay, so there's so many things that's very peculiar about the second sentence from the counterparty. First of all, it does not make any sense that a counterparty would ask for a limited liability for both parties. Usually, what we see is they would ask for unlimited liability to the other side, not for themselves. And second of all, it is unlimited, so why is it still subject to the prevailing parties' discretion? And prevailing party is usually relevant in disputes. you have a
06:07dispute, you have a winner and a loser. But, the whole point of a limitation of liability clause is to decide on what is the cap to the liability. So then, it shouldn't be subject to the prevailing parties' discretion. And so, like, within the sentence itself, it contradicts each other. I I just don't think that in this case, whoever wrote this sentence is a lawyer. I mean, they did use lawyerly language like subject to and prevailing party, but it just is legally incoherent.
06:40Okay, so if we look at how the redline is actually applied here, it's really helpful that there's a comment bubble from Claude explaining what it's done. It would definitely be more helpful if they inserted comment bubbles that explain the user's rationale of why they are pushing back to the counterparty. And I know that Claude has ability to insert those comments. If I was making a demo video for Claude Word add-in for lawyers, I would definitely include that here because that's a very important piece of the workflow explaining to the
07:10other party why you decided to push back on a certain clause. And it's sadly not mentioned here, which I think is of some value that's left not shown. Hopefully, the Claude team can pick up and showcase next time. The last point I want to bring up is about surgical redlines. You would see the word surgical being used a lot in the advertisements for legal tech companies. They're because it reflects a user demand that I don't think Claude this far has picked up. An example of this is clause three. On the surface, Claude deleted subsection C and replaced
07:41it with the language that the user has accepted. However, it has deleted the entirety of what the counterparty has proposed and then just block inserted the new language, which I noticed this being the case when I use Claude Co-work to do redlining as well. If you are not a commercial or transaction lawyer, you would be like, "Okay, yeah. So what? It's achieved the goal." But, if you're a contracts lawyer and you receive comments from the other side completely deleting your language and language. And
08:12vice versa, if you did it to the counterparty, you would have two very pissed off counsel. There are certain redlining etiquettes that lawyers try to follow when they are negotiating a document, which is to minimize the edits. And this in turn improves the readability of the document and removes a lot of the friction. So, the goal should be to assert your position in the document with as minimal edits as possible, which then got coined the term surgical edits, which is obviously not what is being demonstrated in section three here. It's tiny nuances like this that legal tech companies, because
08:42they're so obsessed with solving a lawyer's problem, that they would invest a lot of effort into fine-tuning their system to be able to provide those surgical edits. Whereas, for Claude, because it's a general-purpose tool that's trying to attack 100 different knowledge professions at the same time, it just wouldn't do at this point. But, who knows? Maybe after this video comes out, they will add surgical edits. But, until then, the users themselves would probably have to add additional instructions or set up a skill. The last step of the legal workflow that Claude is demonstrating in this demo is QA.
09:14At this point, we can be pretty certain the user from Anthropic is definitely not a lawyer. And there cannot be a better telltale sign than this prompt instruction right here, "QA the whole doc before we send it back." Lawyers will never use the acronym QA, which stands for quality assurance. It's something that folks in maybe product or engineering would use. Let's see what it catches. On the surface, Claude it was able to demonstrate the concept of being able to proofread a
09:45legal document for its users, whether the users are lawyers or non-lawyers, as we've come to find out in the demo. But, what is a little bit disappointing is the execution of it in the demo, because either the mock-up just wasn't done carefully enough for the document to actually match the suggested actions, or alternatively what what is even more concerning is if this was actually the output from Claude for the proofreading query, it's not very
10:16reliable if both of those outputs are incorrect. So, that's the Word add-in demo for Claude. And here are my thoughts. Claude first moved to meet engineers where they are, which is in the terminal, and now it has moved to meet lawyers where they are in Microsoft Word. Frontier Model Labs are just as
10:46interested in being the supplier of legal tech companies as they are being the competitors. I mean, this isn't new. In the business world, this is just the beginning of vertical integration. But what's more interesting to me about the demo is how important human expert still is. And you can see this in the demo, that Claude clearly lacks legal awareness both in terms of creating a demo doc that looks realistic to
11:17its lackluster performance in proofreading a legal document. And that's actually quite important, because I'm sure their technical team is able to build out all of the features that a lot of legal tech companies can, but do they understand enough about the lawyers and the legal workflows in order to build out a process that is tailored and feels customized to the legal teams. I think it's an open question at this point. And
11:47if the demo, which is the starting point of where you build trust with the users, already contains quite a number of errors and places where it looks inauthentic, then I think that leaves a lot of room for improvement and is the point where the legal experts can really add value. As a user, you can see that this is a very simple legal task of reviewing, redlining, and proofreading a document. And for the most part, AI isn't perfect, and this is not just applicable to the Claude demo alone. In
12:18my own personal use of Claude co-work and also of other legal tech tools, they hallucinate, and sometimes their answers lack legal nuance. That's where the human legal experts are still very important to have in terms of verification and setting up the system so that it actually delivers not just convenient and speedy outputs, but also high-quality outputs. This is very important to keep in mind as we start to build agentic systems that can execute
12:48multiple steps along this workflow to make sure that we don't just amplify these errors as humans become gradually designed out of the loop. Here's how I think about it. For legal teams, general-purpose solutions like Claude is a sensible starting point. It can get you probably 60 to 70% of the way there for the common legal tasks that you see like review, drafting, but there's a gap there because Claude currently lacks that legal awareness. If you can cultivate legal engineers internally or
13:18bring them on externally to help you customize these tools and adapt these workflows, it can probably help you close that 30% gap by customizing the tools to fit your workflows. However, if you want just something off-the-shelf that works and is more adapted to your current legal workflows, then a purpose-built solution is definitely something that is worth exploring. I cannot emphasize enough how important it is to test these solutions and to also retest them after a few months, because
13:49these capabilities are just changing so quickly. So, so for example, we haven't even heard from Gemini and ChatGPT, and so we don't know. And therefore, we need to be more comfortable with uncertainty and more willing to experiment as we move forward. And if you need any help with navigating this very confusing process, then you can visit the Legal Benchmarks website, where we have so many resources that we co-created with hundreds of legal professionals around the world on how to select the right tool for your team and also benchmarks
14:19on AI performance within the legal domain. For legal tech builders, I definitely don't think this is doom and gloom. In fact, to the contrary, I think this Claude um Word add-in demo is a great thing that you can actually show your clients has so many details in the demo to showcase a lack of legal awareness that you can point to and say, "Hey, we can do that better." Only if you actually do it better. And as an example of that is like surgical redlining and drafting hygiene and also actually accurate proofreading capabilities. This also shows that legal tech builders have to go beyond the surface level capabilities today,
14:50because everyone is claiming similar features. And so, if you are also offering the same suite of functionalities, the depth of those functionalities and how well they actually work and have that legal awareness really matters. And look, Anthropic is a massive, well-respected company that is worth billions dollars, and yet they just shipped a demo that isn't perfect. It's encouragement for all of us founders and overthinkers like me out there in the world that you don't have to be perfect before you share an
15:21idea with the world. Anthropic isn't, and yet with every new launch, they blow up the internet. So, just ship it. So, that's it for today. In the upcoming weeks, I'll be comparing Claude with other legal tech solutions out there in the market and also exploring what's happening in the legal tech industry. If you found this episode useful, please subscribe to the channel or join the Legal Benchmarks community, and I will see you in the next one. Bye.
