1° My main concern is whether the evaluation methodology is reliable enough to support the paper's central claims.
The authors emphasize an "Analyst-Calculator Dichotomy," suggesting that models perform relatively well on qualitative financial analysis but struggle with numerical reasoning. However, these capabilities are evaluated in very different ways.
For numerical approximate questions, the reference answer comes from Gemini's interpretation of the charts, then verified by experts, and predictions were accepted within a 5% relative tolerance. Yet, human experts only achieve around 63% accuracy on this task. I’m thus wondering whether the models are genuinely struggling with numerical estimation, or if the evaluation wouldn’t be too restrictive for values that must be estimated visually? The problem seems even worse when several estimated values are combined. This is why I would have liked to see a tolerance sensitivity analysis or some measure of variability between independent human estimates.
On the other hand, open-ended questions are evaluated by three LLM judges. The validation in Table 9 mainly establishes agreement with humans at the level of model rankings, rather than individual answers. The automated judges also tend to assign higher scores than humans. This matters because the claim that GPT-5.4 outperforms human experts on open-ended questions relies on a relatively small difference (4.41 vs. 4.25).
Finally, the overall score averages objective accuracies with normalized subjective ratings. I understand the motivation, but I'm not sure what we can really conclude from such an aggregate score.
To me, these are not minor evaluation details. They directly affect the reliability of the benchmark and the interpretation of its main findings.
2° I'm also not entirely convinced by what the benchmark actually establishes beyond existing work.
The paper repeatedly emphasizes document-level reasoning, but the questions are generated from extracted charts rather than the documents themselves. This makes me wonder to what extent the benchmark evaluates reasoning over financial documents, rather than chart-based QA with an additional evidence-localization challenge. The authors identify a "visual grounding bottleneck," but this seems to cover several different difficulties: locating relevant charts, interpreting them, combining information across figures, and performing calculations.
I don't think a benchmark paper necessarily needs to explain the mechanisms behind model failures. Still, although the cropped-chart experiment and the error analysis in Appendix C.3 are useful, I'm not convinced they sufficiently establish which of these difficulties is actually responsible for the observed failures. In particular, the error analysis groups localization and visual data extraction together, while the cropped-chart experiment does not isolate numerical reasoning from chart interpretation. I would have liked to see a more fine-grained analysis of these errors, perhaps with a tool-assisted calculation baseline. Without this, I find the conclusion that visual grounding is the main bottleneck somewhat stronger than what the experiments establish, especially since the authors recommend agentic approaches without evaluating them.
More generally, several analyses remain descriptive, and some interpretations seem a bit far fetched if I may. The Chinese/English comparison, for example, involves different document distributions, making the claimed "Home Field Advantage" difficult to attribute to language alone.
Finally, I would have appreciated a clearer positioning relative to existing work, particularly FinMMDocR (AAAI 2026), which addresses a closely related document-level reasoning setting or the Scribe Finance dataset (Mouilleron et al., FNP 2026), which reports similar difficulties with financial chart interpretation, especially in conversational scenarios.
Thank you for the detailed response and additional experiments. I think most of my concerns have been addressed.
However, I still have some concerns about the open-ended evaluation. Although the authors responded to W2, I think that the fact that some models achieve higher open-ended scores than human experts still raises questions about the evaluation's reliability, especially given their substantially lower performance on multiple-choice questions. This makes me wonder whether the current evaluation method for open-ended questions can reliably assess finance-specific reasoning abilities, as the authors acknowledged that response length, coverage, and presentation style may contribute to higher scores for LLM answers. I think a more robust evaluation method here may be needed for these open-ended questions.
I understand that this may be difficult to fully address in the current paper, and open-ended questions are only one part of the benchmark. I will raise my score to reflect the authors' efforts in addressing my other concerns, but I would like to keep this concern on record and leave it for the AC to consider in the final decision.