acceptodds
Under review as a conference paper at ICLR 2027

FinDocMRE: A Benchmark for Document-Level Financial Multimodal Reasoning Evaluation

Abstract

While Large Multimodal Models (LMMs) excel in general visual tasks, their deployment in specialized financial contexts remains insufficient. Existing benchmarks prioritize isolated charts, often overlooking the need to integrate data from text, tables, and images within financial documents. To address this limitation, we introduce FinDocMRE, a multi-image document-level benchmark designed for financial multimodal reasoning. We construct the dataset via a semi-automated pipeline that combines Visual-Centric Generation with Expert Verification, thereby minimizing text bias and ensuring high annotation quality. Spanning twelve domains, the benchmark comprises 12,207 samples derived from 2,878 financial reports, designed to evaluate multi-image processing and document-level understanding across five distinct task types. Extensive experiments with eleven LMMs reveal that no model surpasses an overall score of 65, highlighting challenges in integrating visual grounding with logical reasoning within complex document environments. Specifically, we observe a significant performance divergence across tasks, where models exhibit proficiency in semantic narrative construction but struggle with numerical estimation and cross-page visual grounding.

Primary area:
datasets and benchmarks
Keywords:
Financial Benchmark, Document-Level Reasoning, Multimodal Evaluation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_yWBGOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This study represents a continuation of the trend of recent years to introduce new benchmarks for financial data. The new benchmark proposed by the authors is a multi-image document-level benchmark produced via a semiautomated pipeline that combines Visual-Centric Generation with Expert Verification. Eleven LLMs are evaluated on this new benchmark and the results reported in the study demonstrate moderate performance.

Critical strengths

S1. The new benchmark is a useful instrument for the researchers working in the field.

Critical weaknesses

W1. The authors remark the modest performance of LLMs, but do not propose any solution to improve it.

W2. In the “Conclusion”, the authors propose the use of agentic architectures in the future without considering carefully in the article the previous works that already used such architectures.

W3. The analysis of errors is focused on the final result, not on the errors in the reasoning that led to an incorrect result.

Questions for authors

Q1. It is well-known that the real-life financial document analysis heavily relies on precise multi-step arithmetic. Why were not considered tool-augmented agent baselines (e.g., LMMs paired with Python Code Interpreters/Calculators)?

Q2. For dataset construction, three human experts applied a unanimous consensus rule to discard approximately 45% of the auto-generated QA pairs. How was inter-annotator agreement formally measured among the three experts prior to enforcing consensus?

Q3. For open-ended questions, evaluations rely on an ensemble of three LLM judges scoring text responses against reference answers. However, the text-only LLM judges do not inspect the source visual chart images during scoring. Which guarantees exist that these three judges do not reward hallucinations in the model's generated text that look plausible but are visually inaccurate?

Reviewer_dsokOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FINDOCMRE, a multi-image document-level benchmark designed for financial multimodal reasoning. It includes 12207 samples derived from 2878 financial reports, designed to evaluate multi-image processing and document-level understanding across five distinct task types. The author evaluates 11 models against a human baseline and finds that no model surpasses an overall score of 65.

Critical strengths
  1. A carefully designed benchmark with a three-expert verification process.
  2. Provide an evaluation of 11 models and employ comparative analysis.
  3. The paper is well-written and easy to follow.
Critical weaknesses
  1. I found that the error analysis is too general and does not reveal any finance-specific failure patterns, limiting the scientific insights of this benchmark. The Visual Grounding Errors and Logical Reasoning Errors can be applied to almost any visual failure task.

  2. In Table 3, some models outperform human experts on open-ended questions but perform worse on MCQs. This raises concerns about whether the open-ended task design or evaluation can reliably assess financial reasoning capabilities, or whether something about the AI-generated open-ended QA pairs introduces shortcuts that models can exploit. Could the authors explain this further?

Questions for authors
  1. As a benchmark paper, I would like to see evaluations of other open-source models beyond the Qwen family. Also, it would be great to include finance-specialized multimodal models.
  2. Please clarify the weaknesses, especially the weakness 2.
1 comment

Thank you for the detailed response and additional experiments. I think most of my concerns have been addressed.

However, I still have some concerns about the open-ended evaluation. Although the authors responded to W2, I think that the fact that some models achieve higher open-ended scores than human experts still raises questions about the evaluation's reliability, especially given their substantially lower performance on multiple-choice questions. This makes me wonder whether the current evaluation method for open-ended questions can reliably assess finance-specific reasoning abilities, as the authors acknowledged that response length, coverage, and presentation style may contribute to higher scores for LLM answers. I think a more robust evaluation method here may be needed for these open-ended questions.

I understand that this may be difficult to fully address in the current paper, and open-ended questions are only one part of the benchmark. I will raise my score to reflect the authors' efforts in addressing my other concerns, but I would like to keep this concern on record and leave it for the AC to consider in the final decision.

Reviewer_KKjDOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces FinDocMRE, a benchmark of roughly 12k bilingual Chinese/English questions derived from nearly 3k financial reports. Unlike most financial multimodal benchmarks, which focus on isolated charts, FinDocMRE evaluates models on full documents, with about half of the questions requiring information from multiple charts. The dataset is generated using Gemini-2.5-Pro from extracted charts and subsequently filtered by three financial experts. The authors evaluate eleven multimodal models and report a substantial gap with human experts (64 vs. 79 overall). Their main finding is what they call an "Analyst-Calculator Dichotomy": models perform relatively well on open-ended financial analysis but struggle with numerical reasoning and visual grounding. I think this benchlmark addresses a real problem, and I can see it becoming a useful resource for evaluating multimodal models on financial documents. The scale is substantial, the expert verification is a welcome effort, and the visual dependency checks are convincing. What I'm less convinced by is the interpretation of the results. The paper makes fairly strong claims about models being good financial analysts but poor calculators, yet the two abilities are measured in very different ways. More generally, the experiments show where models fail without really getting into the why. To me, this is a promising benchmark paper where some empirical claims currently go a bit further than what the evidence supports.

Critical strengths

As I said above, I think FinDocMRE addresses a relevant problem in these times where many people blindly trust LLMs to evaluate charts and pdf papers. Here, I believe that trying to evaluate models on full financial documents rather than isolated charts is a necessary step toward more realistic applications. The scale is substantial, and the diversity of question types, domains, and languages makes this a potentially useful resource for the community. I particularly appreciate that roughly half of the questions require information from multiple charts.

I appreciate the effort put into dataset construction, particularly the involvement of three financial experts in validating the generated questions. The generation and verification procedures are also reasonably well documented. The chart-masking ablation in Appendix C.5 is a useful sanity check, although I wouldn't consider it a particularly strong contribution on its own.

Critical weaknesses

1° My main concern is whether the evaluation methodology is reliable enough to support the paper's central claims.

The authors emphasize an "Analyst-Calculator Dichotomy," suggesting that models perform relatively well on qualitative financial analysis but struggle with numerical reasoning. However, these capabilities are evaluated in very different ways.

For numerical approximate questions, the reference answer comes from Gemini's interpretation of the charts, then verified by experts, and predictions were accepted within a 5% relative tolerance. Yet, human experts only achieve around 63% accuracy on this task. I’m thus wondering whether the models are genuinely struggling with numerical estimation, or if the evaluation wouldn’t be too restrictive for values that must be estimated visually? The problem seems even worse when several estimated values are combined. This is why I would have liked to see a tolerance sensitivity analysis or some measure of variability between independent human estimates.

On the other hand, open-ended questions are evaluated by three LLM judges. The validation in Table 9 mainly establishes agreement with humans at the level of model rankings, rather than individual answers. The automated judges also tend to assign higher scores than humans. This matters because the claim that GPT-5.4 outperforms human experts on open-ended questions relies on a relatively small difference (4.41 vs. 4.25).

Finally, the overall score averages objective accuracies with normalized subjective ratings. I understand the motivation, but I'm not sure what we can really conclude from such an aggregate score. To me, these are not minor evaluation details. They directly affect the reliability of the benchmark and the interpretation of its main findings.

2° I'm also not entirely convinced by what the benchmark actually establishes beyond existing work.

The paper repeatedly emphasizes document-level reasoning, but the questions are generated from extracted charts rather than the documents themselves. This makes me wonder to what extent the benchmark evaluates reasoning over financial documents, rather than chart-based QA with an additional evidence-localization challenge. The authors identify a "visual grounding bottleneck," but this seems to cover several different difficulties: locating relevant charts, interpreting them, combining information across figures, and performing calculations.

I don't think a benchmark paper necessarily needs to explain the mechanisms behind model failures. Still, although the cropped-chart experiment and the error analysis in Appendix C.3 are useful, I'm not convinced they sufficiently establish which of these difficulties is actually responsible for the observed failures. In particular, the error analysis groups localization and visual data extraction together, while the cropped-chart experiment does not isolate numerical reasoning from chart interpretation. I would have liked to see a more fine-grained analysis of these errors, perhaps with a tool-assisted calculation baseline. Without this, I find the conclusion that visual grounding is the main bottleneck somewhat stronger than what the experiments establish, especially since the authors recommend agentic approaches without evaluating them.

More generally, several analyses remain descriptive, and some interpretations seem a bit far fetched if I may. The Chinese/English comparison, for example, involves different document distributions, making the claimed "Home Field Advantage" difficult to attribute to language alone.

Finally, I would have appreciated a clearer positioning relative to existing work, particularly FinMMDocR (AAAI 2026), which addresses a closely related document-level reasoning setting or the Scribe Finance dataset (Mouilleron et al., FNP 2026), which reports similar difficulties with financial chart interpretation, especially in conversational scenarios.

Questions for authors

1° How does accuracy change, for both models and humans, with tolerances of 2%, 5%, 10%, or 20%? How much do experts disagree when independently estimating values from the same charts, particularly when questions require combining several approximate values?

2° Appendix B.2 reports agreement between human and LLM judges at the aggregate model-ranking level. Could the authors also report item-level agreement (e.g., Spearman correlation or weighted Cohen's kappa)? Also, were the human baseline's open-ended answers evaluated using the same LLM judges as the models' answers?

3° Did both financial analysts independently answer all 12,207 questions, or was the human baseline evaluated on a subset? What were their time constraints, and how were their responses aggregated?

4° In the Appendix (C.4) you mention that longer financial reports frequently contain vector charts, but the extraction pipeline is described primarily in terms of embedded images. How were vector charts and tables handled? Could the extraction and filtering procedures systematically exclude certain types of visual information?

Minor remark

The licence of the dataset should be made clearer: were all document allowed to be redistributed ?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.