Does the Legora BAR really set the bar for agentic benchmarking?

On July 24, Legora released its benchmark for agentic reasoning, the Legora BAR. It is designed to demonstrate product improvement to clients and guide internal product development.

At its core, the BAR asks two questions.

• The first is about the model: How good, fast and cost-effective is it when it runs inside Legora? To answer this, the BAR scores three things—the quality of the output, how that quality trades off against speed and cost, and how well the model cites its sources.
• The second question is about the product: Is Legora itself getting better over time? By re-running the same tasks as the software evolves, the BAR tracks whether Legora’s harness, and with it its application operating system (aOS), is improving.

This article looks beneath the headline scores to ask what the BAR really proves, what it may overlook and what a trustworthy legal-AI benchmark would need to do differently.

TL;DR:

BAR is a meaningful step toward measuring legal AI at the level that users actually experience it: the product. The evidence published so far, however, reads more like a sophisticated vendor-led product evaluation than a benchmark the market can independently rely on.

1. The task design does not fully confront the hardest agentic problems.

The M&A diligence extension released by Harvey LAB deliberately highlights and targets an unsolved research problem for technical teams viz.:

Context management: In a diligence exercise, lawyers review and surface (i) issues present in independent documents in the data room; and (ii) issues arising in the transaction by connecting the story across multiple contracts. Thus, diligence requires both aggregating these millions of tokens across individual contracts in the data room into a coherent story, and then identifying issues within that context as well. Agents must read and make connections across information stretching many times their context window. Novel approaches to memory and compaction are required to allow them to effectively parse and retain key information while identifying and tracing risks.

Exhaustive rather than efficient search: Most agents are trained to identify the relevant portion of a large data space. Their bias is to search efficiently [using keyword searches, semantic searching, RAG (chunking, embedding using mathematical calculations instead of relying on logic, etc.)], not completely/exhaustively the way human lawyers review every part of the documents. The difference between exhaustive and efficient search can be seen in practice: an AI agent is able to identify and flag a named change of control clause and associated conditions mentioned thereunder, but misses change of control conditions present under a completely unrelated clause heading. Diligence requires reversing the intuition of efficient search and teaching them to check, and double-check, every possible issue. 

The representative task released by Legora on GitHub contains reference/input documents which are in multi-modal formats like PDF, Word, Excel and PowerPoint. That is a genuine strength, since lawyers weave context across formats in real work. But a closer look shows most of the information sits in fairly structured form.

Real matters are rarely assembled as clean, long-form question-and-answer word documents. Lawyers commonly reconstruct the facts from fragmented communications, evolving requisition lists, inconsistent terminology and incomplete records.

This has two drawbacks:

• The sample tasks demonstrate the ability to work across formats, but do not clearly establish performance on the messier problems that define reliable legal agency.
• The task set clearly misses the granularity required to highlight, test, and address the current limitations and drawbacks in performance of AI agents such as improving context management and exhaustive rather than efficient search mentioned above. Since these results feed internal product development and are shown to clients as proof of improvement, missing this level of granularity leaves the benchmark painting an inflated picture of reliability that neither product teams nor clients can safely act on.

2. The grading problem: absolute versus weighted

a. Weighted grading sets a lower ceiling than absolute grading

Legora rejects all-or-nothing grading on the ground that a partner does not treat every mistake as equally serious. It instead gives criteria high, medium or low importance according to the risk created by a failure.

The basic logic is defensible. Missing a material change-of-control provision is ordinarily more serious than a formatting error.

The problem lies in how those priorities are assigned.

Weighted grading builds in the margin of error seniors/partners allow for considering learning curves and human limitations. A strict absolute pass criterion sets the bar higher, holding the agent to the expected standard without that discount.

Given that AI carries limitations typically absent when humans do the same work, discounting AI performance for errors native to humans is hard to justify.

For the industry, this presents the following question:

Are we looking to replace the existing system as is, or to create and adopt a system that addresses and eliminates the errors currently prevalent in our systems?

b. Weights don’t align with real lawyer priorities

The Quality Rubric Score, which BAR uses to assess Legora's own performance, shows clearly which areas the platform considers more important than others. Checkpoints are categorized high, medium and low.

The criteria which are typically critical and important from a client deliverable perspective sit in the medium and low bands. For example, clarifying the commercial impact of key risks is classified as medium importance.

Grading Illustration shown in the announcement

This is astonishing given where the task set puts its focus: M&A, IP and tech, real estate, governance, litigation, capital markets, PE/VC, trusts and estates, funds, tax and commercial. All of these practice areas are heavily governed by the commercial aspects of the transaction, which is what determines whether a fact pattern is a legal risk in the first place.

c. The weighted grading is barely doing any work.

The public 175-criterion tax case also labels only three criteria as low importance. Those criteria include removing all template instructions, using the correct currency consistently and adapting headings to the client’s SaaS business model.

That distribution raises two questions.

• If almost everything is medium or high importance, how much value does weighting add?
• Should factual and document-production details receive reduced weight merely because they appear mechanical?

One of the principal reasons to delegate work to an AI agent is to remove repetitive, cumbersome and voluminous checking from the lawyer. Correctly identifying factual details across a large volume of data is exactly the kind of work a lawyer wants to offload to an agent. The burden of verifying factual details extracted across a voluminous set is far heavier than reviewing the strategic positions taken in a document and correcting them.

On both counts, getting factual detail right should be a strict criterion in benchmarking AI performance, not a medium or low weight.

A missed zero can change a transaction value. An incorrect year can make an analysis inapplicable. An inconsistent defined term can create the ambiguity that later becomes a dispute. In these situations, errors and inconsistencies in factual detail are no better than hallucinations.

Getting the legal concepts right while getting the factual details wrong is like identifying the correct grounds and pleadings but filing them before the wrong court. In high-fidelity knowledge work, technicalities are part of correctness.

3. The example prompts supply legal judgment that the harness should provide

Two of Legora’s published prompts are revealing.

Sample Prompts provided in the Legora BAR

The NDA prompt tells the system to focus particularly on the term, the definition of confidential information and the non-solicit clause. The litigation prompt expressly instructs the system to restrict the pleading to what the record supports.

These are sensible instructions from a lawyer. But they also raise a product-design question.

A specialized legal tool claims an advantage over a general-purpose model partly because it incorporates recurring legal workflows and professional context. A capable NDA-review system should ordinarily know that term, confidentiality scope and restrictive covenants require attention. A litigation agent should ordinarily avoid pleading facts unsupported by the record.

The user should still be able to modify priorities. But if good performance depends on the user supplying basic legal-review discipline in every prompt, part of the claimed value of the specialist harness remains unrealized.

The ideal system should combine the lawyer’s matter-specific instruction with a default layer of legal-process knowledge. The lawyer should provide the commercial position and exceptions—not repeatedly restate the elementary requirements of competent legal work.

4. Testing and evaluation methodology has several glaring gaps

a. On model Judge, model bias, variance:

Both Harvey and Legora use expert-written criteria followed by automated LLM judging. Harvey discloses that it uses Claude Sonnet 4.6 as the judge, while Legora stays silent, only noting that it runs each task three times and averages the result.

Running the same prompt thrice addresses variance, but withholding the judge model costs credibility.

Averaging multiple runs also answers a different question from the one most users face. A lawyer often generates an output once and decides whether it is usable. A system that returns two strong answers and one materially deficient answer can still post a respectable average on the BAR — but to the lawyer stuck with that one flawed output, it has failed.

Recent research on long-form LLM judging finds that judge reliability can remain unstable across scenarios even when rubrics and references are provided. This can be addressed by having at least two judges evaluate the outputs against the identified rubric criteria. Ideally the two judges should be from different model families to eliminate bias and mitigate limitations that models from a particular model family may inherently carry.

Borderline results and failures on high-importance criteria should receive mandatory expert legal validation.

Automated judging can scale an evaluation. It should not become an invisible layer on which the credibility of the evaluation itself depends.

b. On evaluation rubric:

When AI outputs are graded using LLM judges, the best practice is to break up the judgment criteria into as many independent rubrics as possible. This ensures that if an output fails on a particular rubric, it is clearly identifiable what the AI output has missed out on. This is particularly relevant from a product development perspective. Conversely, if the rubric has multiple factual details or criterion merged as one criterion, as is the case below, the LLM judge will fail the task on a criterion, despite the AI output meeting 2-3 out of 7 sub-components out of the larger rubric without actually identifying the failure.

Judging Criteria (blue markings and highlights added)

Additionally, across the criterion, the citation sections include cross references (highlighted in yellow in the above picture) of other criterion supported by the citation. Does the LLM judge also grade the ability of the AI output to identify all criterion where the said citation is applicable? If yes, the relevance of this check is unclear.

5. The task set created only represents one out of three target audience segments of Legora.

Despite the “for in-house” positioning in Legora’s broader messaging, the recognized partners contributing to the task set distinctly misses in-house teams and mid-size law firms. Consequently, the representation of use cases from such categories of their target audience is missing. The product evaluation report seems to demonstrate its ability to meet the Big Law deliverable assembly line.

6. Uneven jurisdictional coverage leaves some buyers short-changed.

Where Harvey LAB's task set was mostly US-based, Legora spread its set across more jurisdictions — the US, UK, Europe, Africa, Brazil, and parts of APAC. This is inferred from the named partners and their global reach, since the full task set is not public. Even so, many regions stay underrepresented, including much of LATAM, the Middle East, and smaller APAC countries.

That has a real cost for buyers in those regions: they pay the same price but get a weaker product, because the training set and harness are not tested or tuned for their local nuances — which directly undercuts the product's reliability and usefulness there.

7. The benchmark asks to be trusted on a sample too small to check for bias, difficulty, or judging consistency.

Most cases and rubrics are private; BAR runs on real matters supplied by recognized firms, anonymized and synthesized. Legora has published one synthetic tax case — a prompt, 175 evaluation rubrics and the input documents for a transfer-pricing memo — to demonstrate its methodology. That is a meaningful gesture, but one public case cannot establish the reliability of a benchmark spanning 5,161 cases and 28 practice areas. External reviewers cannot tell whether the published subset is representative, whether difficult cases are sampled consistently, whether model-specific configurations are comparable, or whether the judge applies the criteria reliably across models.

8. The question of credibility.

A private dataset by the Vendor will carry credibility concerns since it cannot be verified whether the dataset is designed to systematically capture difficult exceptions, poor-quality source documents, jurisdictional variation and adversarial ambiguity—or whether it overrepresents workflows where Legora already performs well. The trust sits more naturally with an independent body holding a private set than with a vendor evaluating its own product.

Conclusion: the right level of measurement, with the wrong level of independence

BAR gets one foundational point right: customers do not buy a foundation model in isolation. They buy an application.

Legal output is shaped by prompt construction, document processing, retrieval, memory, workflow design, citation systems and post-processing. Testing models inside Legora's production environment therefore measures something commercially real. BAR's focus on application-level performance - is directionally more useful to customers than another abstract model leaderboard.

That is the contribution worth preserving. However, the publication reads closer to a well-polished product-evaluation report than to an independently reproducible research benchmark.

The next step is independent benchmarking. A credible third party could maintain a private, contamination-resistant task set; ensure representation across organization types and jurisdictions; audit task selection and model configuration; use multiple judges; and require expert review of critical or disputed results. Vendors could still test their full production systems, while customers would gain evidence that does not depend solely on the vendor's own claims.

Legora has helped move legal AI evaluation to the correct level: the product customers actually use. To set the bar for the market, however, the evaluation must also move to the correct level of independence and address the known limitations.

This article is Part 1, from a two-part series where we look below the headline scores. Part 2 of the series reads between the lines and discusses what the announcement reveals about Legora as a product—and about the legal-tech market around it.