acceptodds
Withdrawn from ICLR 2027

One Analyst Is Not Ground Truth: Grading Agent-Built Financial Models Against Observed Professional Practice

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language model agents can now construct complete financial models, but it remains unclear whether they have learned the financial judgment that gives those models meaning. Existing benchmarks often anchor correctness to an expert-authored solution (e.g., numerical targets or detailed rubrics) which introduces a implicit assumption: one expert solution can serve as ground truth. This is appropriate when finance provides a unique answer, but not when judgment is required. Evidence from professional practice challenges this assumption: When financial models built by different analysts for the same company are graded against one another, the median score is only 0.33 under standard tolerances, revealing that single-reference grading confounds professional disagreement with error. We therefore introduce GAUGE, a benchmark that decomposes financial-model evaluation into deterministic checks, rubric judgments and numerical rules. Built from 1,001 professional valuation spanning 922 companies and all 25 GICS industry groups, GAUGE checks mechanical properties deterministically and evaluates judgment-bearing quantities against ranges in professional practice. We then validate GAUGE as a measurement instrument by testing expertise ordering, held-out professional values, and robustness to LM judgements. Our findings show that current LM agents are far better at constructing financial models than at deriving the company-specific assumptions that drive valuation, a gap that even persists after fine-tuning.

Primary area:
datasets and benchmarks
Keywords:
Agent-Built Financial Models, benchmarks

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GQTyOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces a new benchmark that decomposes financial model evaluation into deterministic checks (e.g., numerical consistency), rubric judgments by LLM judges, and numerical rules. The paper finds that LLMs can construct financial models well but are limited in making solid judgments or company-specific assumptions.

Critical strengths

[1] I agree with the authors’ motivation that using a single analyst’s valuation as a ground truth is problematic.

[2] The authors’ effort to collect more than 1,000 models and diverse human answers is commendable. If released publicly, the dataset will be of interest to other researchers.

[3] GAUGE’s conceptual motivation is useful.

Critical weaknesses

[1] The paper’s main question is whether AI models can ‘perform’ financial tasks as well as human analysts. As the authors do not compare AI’s valuations with actual future returns or performance, we still do not know whether AI’s valuation has economic value or not.

[2] BigFinanceBench uses reference answers for ‘verifiable’ questions only. FrontierFinance does use a reference model but evaluates whether sales forecast is based on past trends or management guidance, whether assumptions are rational, and (or) whether predictions are accompanied by sufficient explanations. Therefore, I do not necessarily agree with the authors’ criticism that existing benchmarks largely rely on a single reference. The authors should make these discussions accurate and clarify their motivation.

Furthermore, GAUGE also uses a single reference benchmark for EBIT margin and valuation multiples. As another benchmark, the authors could use analyst consensus and forecast dispersion.

[3] If my understanding is correct, AI models are given a very limited set of information such as basic financials while human experts are given an abundance of data including earnings calls. This is problematic because providing ample firm-specific information is essential for the model to understand firm economics and make tailored financial judgment. I understand that it is difficult to control what humans know, but at least the authors should match the information set given to both humans and AI models. It would be interesting to start from the basic input pack of GAUGE and incrementally include 10-Ks and earnings calls, and report the changes in judgment scores.

[4] We do not know the ‘release dates’ of models. This is very important because analysts’ predictions become remarkably accurate as earnings date approaches. The authors should report at least (i) the written dates of analyst models, (ii) the number of days between analyst models and earnings dates. If these are materially different, one cannot assure that the difference in prediction is due to genuine disagreement among analysts. Instead, the difference could be due to different information sets that analysts have.

[5] If my understanding is correct, ϕ(2)=100\phi(2) = 100, ϕ(1)=60\phi(1) = 60, and ϕ(0)=0\phi(0) = 0, and 29 deterministic items get either 0 or 1, 23 LLM judge items get 0, 1, or 2, and 4 numerical rule items get max 100 points. If one gets a perfect score in all 56 items, the max score should be 79. The senior’s average is reported to be 88.3, which is above the theoretical limit. The authors should clarify this discrepancy.

Questions for authors

Please respond to my comments in the Weakness section.

Reviewer_G3veOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a professional financial-analysis benchmark built around workbooks and structured tasks. Its scoring framework combines deterministic mechanical checks, judged professional-quality facets, reference envelopes, and validity gates. The evaluation includes multiple agents, human groups, expert agreement, an alternative judge, and envelope and ranking analyses. The resource and multidimensional evaluation design are potentially valuable. My main concern is whether the published scoring specification is consistent with the reported human scores, followed by whether envelope validation directly covers the rules used in the actual evaluator.

Critical strengths
  1. The benchmark targets substantive professional artifacts and distinguishes mechanical correctness from professional judgment. This is an informative evaluation setting with clear resource value.
  2. The validation effort is substantial. The expert audit, human groups, alternative-judge evaluation, and ranking analyses provide evidence beyond a single judge’s outputs. In particular, changes in rankings after removing judged facets should not themselves be interpreted as unreliability, since those facets are intended to measure additional capabilities.
Critical weaknesses
  1. I cannot reconcile the reported Senior mean score of 88.3 with the published scoring specification. Section 3.3 maps raw scores 0/1/2 to 0/60/100, deterministic facets are binary, and aggregation is a bottom-up mean across facets, sub-capabilities, and five pillars. Under the mandatory-facet and activation rules in Tables 5, 6, 9, and 11, a deliberately generous upper bound appears to be 85 before gates: P1≤73⅓, P2≤78⅓, P3≤100, P4≤100, and P5≤73⅓. For example, P5’s three mandatory sub-capabilities have maxima 86⅔, 73⅓, and 60. Gates can only reduce the score. This is a conditional inconsistency in my reading of the published rules, not an allegation about the underlying data; an undisclosed normalization or activation rule could explain it. Nevertheless, clarification is essential for reproducibility and the human–agent comparison.

  2. Some envelope validation does not directly evaluate the complete rule used for scoring. In particular, coverage of an industry interval does not establish coverage of its intersection with a reference-centered tolerance band. This concern is localized—the reported rank stability after removing R facets is reassuring—but the validity claim should match the operational evaluator.

Questions for authors
  1. Could you provide a complete scoring trace for at least one Senior attempt: raw facet scores, applicability masks, sub-capability scores, pillar scores, and the final gated score? Please explain how the 88.3 group mean is compatible with binary D facets mapping to at most 60, or identify the normalization/aggregation rule missing from my reading.

  2. What are coverage and counterfeit-rejection rates for the actual intersection-based reference rules, rather than only their constituent industry or multi-method envelopes?

Reviewer_ntobOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces GAUGE, a benchmark for evaluating financial-modeling artifacts when multiple professionally defensible analyses may exist. It combines tasks derived from professional workbooks with a grader that separates mechanical correctness from financial judgment and accommodates variation in numerical assumptions. Model evaluations, human studies, and grader audits are used to examine the difficulty of the tasks and the reliability of the evaluation framework.

Critical strengths
  1. The paper identifies an important limitation of evaluating financial analysis against a single reference solution. The use of professional artifacts and analysis of expert disagreement give the motivation a strong empirical basis.
  2. The evaluation goes beyond model rankings to examine human expertise, grader reliability, and the contribution of different scoring components. This makes the benchmark informative both as an evaluation resource and as a study of how financial-modeling quality should be measured.
Critical weaknesses
  1. The paper provides substantial validation of the benchmark, but the central claim of accommodating legitimate professional disagreement remains only indirectly supported. Disagreement among professional workbooks does not establish that both analyses are sound, and accepting held-out professional values within a numerical range does not demonstrate that distinct, defensible analyses of the same task receive appropriate scores. The worked example illustrates how an individual output is graded, rather than how valid alternative approaches are treated. This gap is particularly relevant because some deployed numerical rules remain tied to a reference workbook. More direct evidence would help distinguish recognizing legitimate analytical diversity from simply allowing numerical variation.
  2. The appendix reports rule coverage and conditions for excluding inapplicable criteria. However, some exclusions also depend on whether the required quantities can be measured from the output. It remains unclear how the grader distinguishes genuine inapplicability from failure to provide a requested result, and whether other checks consistently penalize the latter. This is a concern about score comparability, not about the absence of a documented exclusion policy.
Questions for authors
  1. Is there an existing example or analysis in which different approaches to the same task are independently judged professionally defensible and both receive appropriate scores from GAUGE? A comparison with single-reference grading, alongside a substantively flawed alternative, would help demonstrate the intended benefit beyond wider numerical tolerances.
  2. How do the reference-based numerical rules avoid penalizing a defensible analysis that uses different assumptions from the reference workbook? Please connect the existing validation to this specific concern.
  3. When a required numerical output is absent or cannot be extracted, which grading rule distinguishes that failure from legitimate inapplicability? Do the existing gates or other criteria ensure that omission cannot improve the resulting score?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.