- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces Enterprise’s Last Exam (ELE), a benchmark for what the authors call “organizational judgment”: making correct decisions that depend on company-specific information such as entity relationships, precedents, policy versions, approval authority, timing, and conflicting records across systems. ELE contains 630 text scenarios across six reasoning dimensions and eight business domains, split into a Core set (384), a harder Challenge set (156), and a Counterfactual set (45 pairs, where changing one key fact changes the correct answer). Seven models are evaluated with a single direct prompt at temperature 0, scored by exact match plus an LLM judge. Core accuracy ranges from 77.4% to 97.4%. Precedent-based exception handling, cross-system synthesis, and policy-version reasoning are clearly harder than the other three dimensions.
Critical strengths
The problem is well motivated and clearly defined. Separating “choosing the right decision” from “executing it in software” is a useful framing that existing enterprise and agent benchmarks do not isolate. The six-dimension taxonomy is intuitive and makes it easy to see where different models fail; for example, the category results show that several models score above 95% on approval chains but below 80% on precedent handling, which a single aggregate score would hide.
The counterfactual pair design is a good idea. Changing a single decision-critical fact so that the correct action must change is a more direct test of whether a model actually relies on the governing evidence, rather than on surface patterns or generic business norms. The release of per-item transcripts, Wilson intervals, and a paired McNemar test also makes the reported results easier to audit.
Critical weaknesses
There is little evidence for data quality. The abstract describes the scenarios as “practitioner-authored,” but Section 4.1 says they were drafted from practitioner-informed templates and then reviewed and rewritten by the authors. These are quite different claims. The paper reports no inter-annotator agreement, no human baseline, and no check that each gold answer is unambiguous; the gold answers are also withheld during review. This matters because some results are hard to interpret without such evidence. For example, llama4-maverick outperforms claude-opus-5 on the Challenge set (87.8% vs. 85.9%), and claude-opus-5 drops 11.5 points from Core to Challenge. This could reflect real differences in capability, but it could also reflect ambiguous questions or wrong gold labels, and the paper has no way to tell these apart. With about 104–106 items per dimension and confidence intervals of 16–20 points, some of the dimension-level conclusions are also within noise.
The benchmark is not very difficult, and the evaluation setting is simpler than the title suggests. The best model already reaches 97.4% on Core, so the claim that “no model saturates the benchmark” is weak, and the benchmark may have a short useful life. All relevant evidence is placed directly in the prompt, so the task is closer to careful reading comprehension than to reasoning under “fragmented enterprise context”: the model never has to find, retrieve, or decide what information is missing. On scoring, about 26.5% of responses are graded by claude-opus-4-6, which comes from the same model family as the top-ranked claude-opus-5. This creates a possible self-preference bias, and the paper does not validate the judge with human checks. Finally, the counterfactual set does not yet report whether a model’s answer flips in the direction required by the changed fact, which is the main point of the design; pair-level “both correct” accuracy only partly captures this.
Questions for authors
How exactly were the scenarios created? What did practitioners contribute versus the authors, and were LLMs used to draft scenarios or gold answers? Can you report agreement between independent annotators on the gold decisions, or a small human-expert baseline, to show that the answers are unambiguous and that the Challenge results reflect model capability rather than label noise?
Since the judge (claude-opus-4-6) and the top model (claude-opus-5) come from the same family, can you re-score the judge-path responses with a judge from a different provider, or report a human audit of a sample of judged responses? Does the model ranking, especially the gap between claude-opus-5 and gpt-5.6-terra, still hold?