- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces a new benchmark that decomposes financial model evaluation into deterministic checks (e.g., numerical consistency), rubric judgments by LLM judges, and numerical rules. The paper finds that LLMs can construct financial models well but are limited in making solid judgments or company-specific assumptions.
Critical strengths
[1] I agree with the authors’ motivation that using a single analyst’s valuation as a ground truth is problematic.
[2] The authors’ effort to collect more than 1,000 models and diverse human answers is commendable. If released publicly, the dataset will be of interest to other researchers.
[3] GAUGE’s conceptual motivation is useful.
Critical weaknesses
[1] The paper’s main question is whether AI models can ‘perform’ financial tasks as well as human analysts. As the authors do not compare AI’s valuations with actual future returns or performance, we still do not know whether AI’s valuation has economic value or not.
[2] BigFinanceBench uses reference answers for ‘verifiable’ questions only. FrontierFinance does use a reference model but evaluates whether sales forecast is based on past trends or management guidance, whether assumptions are rational, and (or) whether predictions are accompanied by sufficient explanations. Therefore, I do not necessarily agree with the authors’ criticism that existing benchmarks largely rely on a single reference. The authors should make these discussions accurate and clarify their motivation.
Furthermore, GAUGE also uses a single reference benchmark for EBIT margin and valuation multiples. As another benchmark, the authors could use analyst consensus and forecast dispersion.
[3] If my understanding is correct, AI models are given a very limited set of information such as basic financials while human experts are given an abundance of data including earnings calls. This is problematic because providing ample firm-specific information is essential for the model to understand firm economics and make tailored financial judgment. I understand that it is difficult to control what humans know, but at least the authors should match the information set given to both humans and AI models. It would be interesting to start from the basic input pack of GAUGE and incrementally include 10-Ks and earnings calls, and report the changes in judgment scores.
[4] We do not know the ‘release dates’ of models. This is very important because analysts’ predictions become remarkably accurate as earnings date approaches. The authors should report at least (i) the written dates of analyst models, (ii) the number of days between analyst models and earnings dates. If these are materially different, one cannot assure that the difference in prediction is due to genuine disagreement among analysts. Instead, the difference could be due to different information sets that analysts have.
[5] If my understanding is correct, , , and , and 29 deterministic items get either 0 or 1, 23 LLM judge items get 0, 1, or 2, and 4 numerical rule items get max 100 points. If one gets a perfect score in all 56 items, the max score should be 79. The senior’s average is reported to be 88.3, which is above the theoretical limit. The authors should clarify this discrepancy.
Questions for authors
Please respond to my comments in the Weakness section.