acceptodds
Under review as a conference paper at ICLR 2027

Enterprise’s Last Exam: A Benchmark for Organizational Judgment under Fragmented Enterprise Context

Abstract

Frontier language models perform strongly on benchmarks of coding, mathematics, knowledge, and general reasoning. However, enterprise deployments often require a different capability: making decisions whose correctness depends on organization-specific entities, historical precedent, conflicting systems of record, changing policy versions, distributed or informal authority, and temporal context. We introduce Enterprise's Last Exam (ELE), a benchmark for this capability, which we term organizational judgment. ELE complements benchmarks of generic knowledge, artifact production, and software interaction by directly measuring whether models identify and correctly apply the organization-specific evidence that governs a decision. ELE comprises 630 practitioner-authored scenarios across three complementary model-blind evaluation sets: a Core set (n=384), a difficulty-focused Challenge set (n=156), and 90 counterfactual scenarios forming 45 paired cases in which a single decision-critical fact changes. Scenarios span six dimensions: entity resolution under ambiguity, precedent-based exception handling, cross-system synthesis, policy-version reasoning, approval-chain reconstruction, and temporal decision consistency. We evaluate seven proprietary and open-weight models under a controlled direct-prompt condition at temperature zero. On the primary Core set, accuracy ranges from 77.4% to 97.4%; no model saturates the benchmark. Performance varies sharply by reasoning type: models are strongest on entity resolution, approval-chain reconstruction, and temporal consistency, but substantially weaker when they must apply precedent or reconcile conflicting evidence across systems; cross-system synthesis ranges from 67% to 90%. Challenge accuracy ranges from 73.7% to 87.8%. On the 45 counterfactual pairs, the fraction for which models answer both variants correctly ranges from 62% to 89%, indicating substantial variation in robustness when a single governing organizational fact changes.

Primary area:
datasets and benchmarks
Keywords:
organizational judgment evaluation, enterprise reasoning, benchmark, evaluation, counterfactual robustness, large language models, agentic AI, precedent reasoning, cross-system synthesis, fragmented context

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_iM9COpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces Enterprise’s Last Exam (ELE), a benchmark for what the authors call “organizational judgment”: making correct decisions that depend on company-specific information such as entity relationships, precedents, policy versions, approval authority, timing, and conflicting records across systems. ELE contains 630 text scenarios across six reasoning dimensions and eight business domains, split into a Core set (384), a harder Challenge set (156), and a Counterfactual set (45 pairs, where changing one key fact changes the correct answer). Seven models are evaluated with a single direct prompt at temperature 0, scored by exact match plus an LLM judge. Core accuracy ranges from 77.4% to 97.4%. Precedent-based exception handling, cross-system synthesis, and policy-version reasoning are clearly harder than the other three dimensions.

Critical strengths

The problem is well motivated and clearly defined. Separating “choosing the right decision” from “executing it in software” is a useful framing that existing enterprise and agent benchmarks do not isolate. The six-dimension taxonomy is intuitive and makes it easy to see where different models fail; for example, the category results show that several models score above 95% on approval chains but below 80% on precedent handling, which a single aggregate score would hide.

The counterfactual pair design is a good idea. Changing a single decision-critical fact so that the correct action must change is a more direct test of whether a model actually relies on the governing evidence, rather than on surface patterns or generic business norms. The release of per-item transcripts, Wilson intervals, and a paired McNemar test also makes the reported results easier to audit.

Critical weaknesses

There is little evidence for data quality. The abstract describes the scenarios as “practitioner-authored,” but Section 4.1 says they were drafted from practitioner-informed templates and then reviewed and rewritten by the authors. These are quite different claims. The paper reports no inter-annotator agreement, no human baseline, and no check that each gold answer is unambiguous; the gold answers are also withheld during review. This matters because some results are hard to interpret without such evidence. For example, llama4-maverick outperforms claude-opus-5 on the Challenge set (87.8% vs. 85.9%), and claude-opus-5 drops 11.5 points from Core to Challenge. This could reflect real differences in capability, but it could also reflect ambiguous questions or wrong gold labels, and the paper has no way to tell these apart. With about 104–106 items per dimension and confidence intervals of 16–20 points, some of the dimension-level conclusions are also within noise.

The benchmark is not very difficult, and the evaluation setting is simpler than the title suggests. The best model already reaches 97.4% on Core, so the claim that “no model saturates the benchmark” is weak, and the benchmark may have a short useful life. All relevant evidence is placed directly in the prompt, so the task is closer to careful reading comprehension than to reasoning under “fragmented enterprise context”: the model never has to find, retrieve, or decide what information is missing. On scoring, about 26.5% of responses are graded by claude-opus-4-6, which comes from the same model family as the top-ranked claude-opus-5. This creates a possible self-preference bias, and the paper does not validate the judge with human checks. Finally, the counterfactual set does not yet report whether a model’s answer flips in the direction required by the changed fact, which is the main point of the design; pair-level “both correct” accuracy only partly captures this.

Questions for authors

How exactly were the scenarios created? What did practitioners contribute versus the authors, and were LLMs used to draft scenarios or gold answers? Can you report agreement between independent annotators on the gold decisions, or a small human-expert baseline, to show that the answers are unambiguous and that the Challenge results reflect model capability rather than label noise?

Since the judge (claude-opus-4-6) and the top model (claude-opus-5) come from the same family, can you re-score the judge-path responses with a judge from a different provider, or report a human audit of a sample of judged responses? Does the model ranking, especially the gap between claude-opus-5 and gpt-5.6-terra, still hold?

Reviewer_Bud7OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces a benchmark (ELE) to evaluate whether LLMs can take correct actions based on organization-specific information such as company policies, past decisions, and approval structures. The authors first define six reasoning dimensions relevant to enterprise setting and then construct 630 scenarios spanning these dimensions. The scenarios are divided into Core, Challenge and Counterfactual sets. Seven LLMs are evaluated on the benchmark, with overall accuracy ranging from 76.4% to 94.1%.

Critical strengths

The major strengths of the paper are:

  • Evaluating LLMs in enterprise settings is an interesting problem. The authors attempt to give a structure to evaluation by introducing six reasoning dimensions.
  • The creation of counterfactual scenario pairs, where changing one critical fact changes the correct answer, is useful.
  • The benchmark covers 8 enterprise domains and evaluates 7 proprietary and open-weight LLMs. The authors observe that LLMs perform differently across reasoning dimensions. For example, models perform better on some dimensions such as identifying approvers but struggle with handling exception cases and combining information across systems.
Critical weaknesses

Below are critical weakness of the paper:

  1. Several existing benchmarks, such as ENTLORE, EnterpriseBench, CRMArena, already evaluate different aspects of enterprise reasoning. Although the authors distinguish ELE from some of these benchmarks by emphasizing that ELE focus on decision-making based on organization specific knowledge. However, novelty is still not fully clear. A more concrete comparison, possibly with examples, would help explain what specific capabilities ELE evaluates that existing benchmarks do not.
  2. Section 4.1 mentions that the scenarios are self-contained and include evidence required to answer it. Does this mean the benchmark is mainly evaluating how well LLMs can read a passage and apply the given rules?
  3. Several details about dataset construction are missing. How many practitioners were involved, and what were their backgrounds? Were the scenarios created manually or using LLMs, or both? What are the practitioner-informed templates? Was inter-annotator agreement measured, and how were ambiguous scenarios resolved? How is challenge level measured to decide the split?
  4. How were the six dimensions identified and validated. Does these cover every kind of failure or how much do they overlap?
  5. The LLM-as-judge scoring needs further validation. The authors first use exact matching and then send non-exact responses to Claude Opus 4.6. How was the 0.9 correctness threshold chosen? Is the judge equally reliable across all seven models? How often does the judge disagree with human annotators?
  6. With paper titled "Enterprise's Last Exam", I expected a challenging dataset for the current generation of models, similar to Humanity's last exam (HLE) and Agent's last exam (ALE) dataset when those were released. With performance accuracies ranging from 77.4% to 97.4% on ELE, the are two possibilities: either the dataset scenarios are not complex enough or the strongest models have "almost" reached AGI for enterprise tasks! From my direct experience, most likely it is former.
Questions for authors
  1. There are no scenario examples in the paper or appendix, making it difficult to understand the nature and complexity of the tasks.
  2. Explain how ELE scenarios differ from tasks in existing enterprise benchmarks such as ENTLORE, CRMArena, EnterpriseBench? What reasoning capabilities does ELE evaluate that are not covered by these benchmarks?
  3. Details of Data construction pipeline. Few questions in Weakness point (3)
  4. LLM-as-judge validation. Some questions in Weakness point (5)
  5. Did the authors think about defining some measure for complexity of the task? That would help quantify the task's relation to the performance of the model, rather than just reporting the accuracies.
Reviewer_D9jvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Enterprise's Last Exam (ELE), a benchmark that evaluates the LLM's decision capacity in enterprise environments. It mainly focuses on whether the LLM can make the right decisions based on enterprise's records, rules and historical information. The benchmark includes 630 scenarios, which involves six reasoning dimensions. The authors conduct experiments on seven LLMs and analyze the results.

Critical strengths

S1. This paper focuses on the decision-making in enterprise environments, which is a meaningful problem. S2. The experiments are comprehensive. The benchmark includes diverse types of reasoning tasks.

Critical weaknesses

W1. The novelty is not clear enough. Existing works have already studied enterprise reasoning [1,2] and decision-making [2, 3]. Although this paper reorganizes these task, it is still necessary to explictly clarify the differences between these existing works.

W2. The setting of the benchmark is too idealized to reflect the complexity of the real enterprise environment. All of the relevant information is directly supplied to the LLM, which only needs to choose an answer based on the given text. Real environment is much more complex. Information is dispersed across different department, which means that the LLM needs to retrieve relevant information in a complex environment.

W3. We need more details about experimental evaluations. The paper does not report the agreement between human-judge and LLM-judge, while 26.5% of the answers are scored by LLM-as-a-Judge. And there is also no human expert performance as a reference. Furthermore, the best model has already achieved 97.4% accuracy on the Core set, which suggests this benchmark might have limited ability to distinguish among frontier models.

[1] L. Boisvert et al. WorkArena++: Towards compositional planning and reasoning-based common knowledge work tasks. NeurIPS'24. [2] K.-H. Huang et al. CRMArena: Understanding the capacity of LLM agents to perform professional CRM tasks in realistic environments. NAACL'25. [3] S. Yao et al. τ-bench: A benchmark for tool-agent-user interaction in real-world domains. ICLR'25. [4] F. F. Xu et al. TheAgentCompany: Benchmarking LLM agents on consequential real world tasks. NeurIPS'25.

Questions for authors

See W1~W3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.