Leaderboard

We test 64 tasks built around real-world legal requests lawyers are already using AI to help with. The set splits into two categories.

Contract drafting accounts for 34 tasks: producing or amending contract language from a lawyer’s instruction, including clauses, side letters, amendments, and “tidy this up” requests.

Information extraction accounts for 30 tasks: answering questions about a document, such as locating clauses, defined terms, governing law, values, and obligations. Tasks vary by how broad the question is, how clean the document is, whether one document or several, and whether the answer is present, absent, partial, or contradictory.

The set is heavy on commercial, IP, employment, M&A, and competition work. Every task is single-turn: one instruction, one answer, no follow-up.

Where the tasks come from

Practitioner-authored tasks make up the core of the set. Lawyers from the global Legal Benchmarks community wrote the instructions, reference files, and answer keys based on real scenarios from their own work.

Synthetic tasks make up the rest. We built a pipeline to generate additional tasks from the failure modes identified in our earlier Info Extraction and Contract Drafting reports, and from published legal-AI benchmarking work in the industry. Every synthetic task is validated by a human expert before it enters the set.

How this builds on our prior research

This is the third installment in a series we’ve been publishing since April 2025. Our Phase 1 report tested 6 AI assistants on 18 information extraction tasks; Phase 2 tested 13 AI tools and a human-lawyer baseline on 30 contract drafting tasks, and introduced the two-axis split between reliability and usefulness that this benchmark inherits. The 64-task set consolidates both into a single, repeatable evaluation, with the failure modes identified in Phases 1 and 2 now seeding the synthetic tasks.

We built an evaluation pipeline to collect AI outputs and measure their quality on two independent axes, reliability and usefulness, scalably and repeatably.

Collecting each answer

For an AI model, we send the instruction and documents to the model directly via API through our evaluation pipeline. System prompts are standardized across API models so the only variable between runs is the model itself. Documents are sent in native form rather than pre-converted to text, so reading them is part of the capability we test: a model that can read a file does so with its own vision, and a model that cannot fails tasks that depend on the document.

For a commercial application like Claude or ChatGPT, we use a browser agent that drives the application’s own interface the way a lawyer would, and we grade whatever it produces, including any document it drafts and attaches, not just its chat reply.

Model variations tested

Every API model is called through OpenRouter at the provider’s default reasoning setting, with no reasoning or reasoning_effort override, so the only variable between runs is the model itself.

Two-axis scoring

Quality is reported on two independent axes, reliability and usefulness, kept separate rather than blended into a single score. A memo can be technically accurate and unreadable, or beautifully formatted and wrong; reporting the axes separately preserves that distinction.

Reliability answers the question “is the answer right?” It captures accuracy, completeness, relevance to what was asked, and honesty about what the AI does not know rather than inventing. We report the share of tasks the AI gets fully right.

Usefulness answers the question “could a practitioner use it as written?” It captures whether the output is clear, well formatted, and the right length. We report the average of three sub-scores (clarity, length, structure), each rated 1 to 3, where 1 means needs rework before use, 2 means usable with edits, and 3 means use as-is.

How each task is graded

Each task ships with a checklist of requirements written by the lawyer who authored it, covering both substance and wording. For reliability, each item is marked pass or fail, and missing even one fails the whole task. This follows error-analysis-first eval discipline: binary judgments, no false-precision Likert scales hiding inside the task definition.

The checklist is also where we encode the fact that there is rarely only one right answer in legal work. For example, where a lawyer instructs the AI to draft an unenforceable clause, the criteria are written to pass any of several legitimate response shapes: refusing to draft and explaining why, drafting the requested clause with an explicit risk flag, or proposing a compliant alternative with the reasoning for the substitution. What fails is silent compliance with the bad instruction. The same logic runs through the extraction tasks: where a defensible answer can be “yes,” “no,” or “uncertain,” the criteria accept the valid variants and fail only the stated must-avoids, such as hallucination, missing a core element, or flattening a conditional answer into a flat one. The benchmark records the shape the model chose, and the criteria meet practice where a competent lawyer would actually land.

Reliability is graded by a single LLM judge (Claude Sonnet 4.6) against the task’s checklist. Usefulness is scored by a panel of LLM judges (Claude Sonnet 4.6 and Gemini 3.1 Pro) running in parallel for every cell, and their scores are averaged into a consensus.

Grader validation

LLM graders are only as useful as the evidence that they agree with practitioners. We run two checks.

Cross-grader agreement. Across roughly 1,700 paired scores, the two usefulness graders gave the exact same 1 to 3 score about 82% of the time. Agreement is highest on clarity (around 97%) and structure (83% to 94%), and lowest on length (56% to 64%); length is the most subjective sub-criterion and the one most worth a human eye. Cells where the two judges disagreed by the maximum 2-point spread are flagged for human review.

Favoritism check. We test whether a grader scores its own maker’s model higher. It does not show up in the data. Sonnet does not top the usefulness leaderboard in either category, and on the length sub-criterion the Anthropic models actually sit at the bottom, the opposite of what a biased grader would produce.

Results and rankings

Each model run produces, for every task, a reliability verdict (a per-criterion pass/fail tally that resolves to “fully right” or not) and a usefulness score (the mean of clarity, length, and structure from the two-judge panel).

We then aggregate across the 64 tasks in two cuts. Per category, contract drafting and information extraction are ranked separately and never combined: a model’s reliability score in a category is the share of that category’s tasks it got fully right, and its usefulness score is the mean of its task-level usefulness means in that category. Per sub-criterion, clarity, length, and structure are also reported individually, because they fail differently and a single overall number hides that.

Rankings are refreshed each time we re-run the benchmark.

Accessing the tasks

The task set, reference documents, and per-task criteria are not publicly distributed. Releasing them in full would let vendors train or tune against the benchmark, which would quickly erode the signal we are trying to publish. Researchers and academics who want to inspect the methodology in more detail can request access by getting in touch. Vendors who want their tool measured against the set can submit it through the route below, and we run the evaluation on their behalf.

The benchmark gives a useful read on where models stand today, but it has limits worth flagging.

Task count and coverage. The set is 64 tasks. It leans commercial, IP, employment, M&A, and competition work, and it is English only with a US and UK skew. Practice areas and jurisdictions outside that scope are thin or absent.

Single-turn only. Every task is one instruction and one answer. We do not yet test multi-turn workflows, longer matters, or back-and-forth refinement, which is closer to how lawyers actually use these tools.

Snapshot, not a trend. Models and applications change month to month. The leaderboard reflects a point-in-time run, and a result from one month may not hold the next.

LLM judges. Reliability is graded by a single LLM judge and usefulness by a panel of two. Cross-grader agreement is high overall but lower on length, and the judges are validated by human experts on a spot-check basis, not on every task.

Application coverage. Application testing is limited to Claude, ChatGPT, and Gemini, and the Apps vs. Models table is carried over from the prior run rather than re-tested this cycle. Other legal AI products, including purpose-built tools, are not yet represented on the leaderboard.

File handling. Documents are sent in native form, so a model that cannot read a file will fail file-dependent tasks. That is by design — reading the document is part of the capability we test — but it means text-only models score lower on file-heavy tasks than they might with pre-extracted text.

API system prompts. System prompts are standardized across API models so the only variable is the model itself. A model that benefits from a more tailored prompt may look weaker here than it does in practice.

Cost figures. Per-task cost is based on listed API pricing at the time of the run. It does not include retries, application overhead, or enterprise discounts.

Consistency and drift not tested. We grade a single run per task. We do not yet measure how much a model’s output varies across repeated runs of the same task, which matters for anyone relying on the tool day to day.

We re-run the benchmark every month, and a few things are already in train.

• Benchmark applications. Tool builders can submit their products to see where they rank across these task categories.
• Wider coverage. We want to expand into more task categories that both in-house lawyers and private-practice lawyers face day-to-day, and, more importantly, into more geographic regions and languages.
• More practice areas. The set leans commercial, IP, employment, M&A, and competition. We are adding tasks in the areas that are thin today.
• Beyond single-turn. Every task today is one instruction and one answer. Multi-turn work, longer matters, and tool use are on the list.