acceptodds
Under review as a conference paper at ICLR 2027

FinVQA-Chart: A Diagnostic Benchmark Isolating the Multimodal Fusion Bottleneck in Financial Chart Understanding

Abstract

Vision–Language Models (VLMs) report high aggregate accuracy on multimodal benchmarks, yet a single accuracy number cannot localise where a model fails in visual perception, in language reasoning, or in the cross-modal integration that binds them. We introduce FinVQA-Chart, a diagnostic benchmark that separates these three abilities via modality-controlled question triplets: for each underlying problem, we construct a visual-only, a text-only, and a fusion variant, where the fusion variant is solvable only by combining a cue present exclusively in the image with a cue present exclusively in the text. The benchmark contains charts rendered from authentic OHLCV data for U.S. equities (, seven volatility regimes) paired with five-option multiple-choice questions stratified by reasoning category, modality, and difficulty. We define Fusion Efficiency (FE) as multimodal accuracy normalised by the stronger unimodal channel, and, unlike prior "diagnostic" benchmarks, we provide a construct-validity study showing that FE carries signal beyond raw multimodal accuracy and beyond question difficulty: on difficulty-matched items, human Fusion Efficiency reaches while the strongest model stays at , showing the gap reflects integration, not item difficulty. Evaluating contemporary VLMs (frontier, reasoning-specialised, chart-specialised), the strongest model achieves overall, points below human experts, with FE vs for human experts (chance-corrected). Neither extended chain-of-thought nor chart-domain specialisation closes the gap; error stratification localises the residual to numerically grounded operations rather than perception. We release the dataset, an identity-controlled contamination-safe variant, a real-chart transfer set, the evaluation harness, prompts, and baselines. All artefacts derive from authentic market data via standardised rendering with template-controlled language; we provide evidence that the design does not create trivial shortcuts (a model fine-tuned on in-template data reaches only , below frontier zero-shot) and that findings transfer to real charts (rank correlation r=).

Primary area:
datasets and benchmarks
Keywords:
Vision-Language Models, Financial Chart Understanding, Multimodal Reasoning, Benchmark Dataset, Technical Analysis, Candlestick Patterns, Numerical Reasoning, Visual Question Answering, Domain-Specific Evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_V6VoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors propose FinVQA-Chart that separates these three abilities via modality-controlled question triplets. It includes 149400 five-option multiple-choice questions stratified by reasoning category, modality, and difficulty. The paper evaluates 45 VLMs. showing the strongest model achieves 72.3% overall, 19.5 points below human experts.

Critical strengths

1 A comprehensive evaluation includes 45 VLMs, which is rare to see in other benchmark papers.

2 The benchmark size is quite large with 149400 questions.

Critical weaknesses
  1. All figures are vague in the paper. More importantly, Figure 1 shows one specific case. But the bars in the two charts look different to me, although they are from the same case and should be exactly the same. Could the authors explain why this difference happens?
  2. The authors use deterministic algorithms to generate ground-truth answers, but it is unclear how they check whether these answers are correct. For example, if one candlestick satisfies the conditions of multiple patterns, how do the authors ensure that only one answer is correct in the MCQ?
  3. The difficulty design is unclear to me. How do the authors classify questions into different difficulty levels, and what domain knowledge or criteria are used to determine these levels?
Questions for authors

Please check the weakness.

Reviewer_NrhFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a large-scale diagnostic benchmark designed to isolate cross-modal integration from unimodal visual perception and textual reasoning in financial candlestick chart understanding. The benchmark evaluates 45 VLMs using a proposed Fusion Efficiency (FE) metric, concluding that cross-modal fusion represents the primary performance bottleneck across current models.

Critical strengths
  1. The motivation to separate textual and visual reasoning is strong, and several of the empirical findings are insightful. (but the motivation is not that novel from related work)
  2. The dataset quality appears well-maintained through multiple quality control steps.
Critical weaknesses
  1. Positioning and Related Work
  • The cited papers are dated, and the paper does not situate itself well within recent literature. In particular, it misses recent chart understanding paper. For instance, separating "visual vs. text vs. fusion" has already been explored in ChartMuseum, as well as recent work on cross-modal binding probes.
  • Cross-chart comparative reasoning is not unique to the proposed benchmark; existing benchmarks such as CharXiv, ChartMuseum, and ChartQAPro also feature cross-chart comparative reasoning.
  1. Method and Experiment Result Calrity
  • The paper states that "Each triplet shares one latent answer y and three question variants." However, in Figure 1, the answers to all three question variants are different.
  • The multiple-choice options and ground-truth labels in the Figure 4 qualitative examples are confusing, and it is unclear whether answering them requires external background knowledge. Does background knowledge count as text reasoning?
  • In Line 269, it is not clear what variable or metric the reported correlation is measuring.
  1. Inconsistencies Across Figures, Tables, and Main Paper
  • In Figure 1, the plot compares o3-mini-high and human performance, whereas the caption states that it compares Claude Opus 5 and human performance. Furthermore, the "15.2 point accuracy drop" cannot be derived from the plotted bars, the "-21.5 points" callout calculation is mathematically incorrect, and the claim that "human expert performance remains perfectly stable across modalities" is not supported by the figure.
  • In the Abstract, the fine-tuned model is reported to reach 64%, whereas Section 3.4 states that the fine-tuned model reaches 53.7%. These numbers are inconsistent, and the main paper does not clarify which model was fine-tuned.
  • Between Figure 3 and Table 2, the category assignment of Grok 4.6 is inconsistent (Gen C — Reasoning in Figure 3 vs. C-Frontier in Table 2).
  • In Table 2, the numbers in the Overall column should be the average of the three equal-sized modality columns on the right (Visual, Text, and Multi-Modal), but based on direct calculation, none of the rows match the average of the three columns.
  • The average performance per model is inconsistent between Table 2 and Table 4, and averaging across models in Table 3 does not make sense.
  • The number of questions reported for CharXiv in Table 1 (~1,000) is incorrect. The "Descriptive/reasoning only" questions in CharXiv include more than 1,000 examples.
Questions for authors

Please make the positioning of the paper clearer and address those identified conflicting statements and results.

Reviewer_eRwKOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper constructs a VLM diagnostic benchmark, FinVQA-Chart, for financial candlestick charts. It includes 96 US stocks, 9960 multi-panel charts, and 149,400 multiple-choice questions, setting up three types of tasks: pure visual, pure text, and multimodal fusion. The authors propose fusion efficiency (FE), normalizing multimodal accuracy using the optimal single-modal accuracy. The evaluation is conducted on 45 large visual models and human experts, analyzing numerical reasoning, domain transfer, tool enhancement, and generalization to real charts. The authors argue that cross-modal fusion is the primary bottleneck, with most other errors stemming from numerical reasoning.

The topic is meaningful, and the controllable benchmark is valuable. However, whether FE can effectively identify fusion defects is questionable; there are inconsistencies in the numerical values ​​throughout the manuscript, a key appendix is ​​missing, and the core conclusions are currently unreliable.

Critical strengths

The benchmark design shows potential: three types of supporting questions are constructed on standardized financial charts, samples are generated using real OHLCV data to reduce annotation noise, and hierarchical analysis is supported; if the diagnostic logic is validated, it can serve as a domain resource.

The evaluation scheme is comprehensive, with appropriate control of variables: it includes multiple models, human baselines, data contamination, difficulty matching, tool usage, and cross-scenario transfer validation, meeting diagnostic benchmark specifications; however, the experiments and data still need verification.

Critical weaknesses

Fusion efficiency (FE) cannot isolate pure fusion errors. Multimodal questions require both visual and textual cues to be correct simultaneously; even if the fusion itself is error-free, FE can still be less than 1. The results of Claude Opus 5 can be interpreted by the independent joint probability of cues, and may not necessarily indicate fusion errors. Manual difficulty matching cannot solve the recognition problem; conditional accuracy analysis should be used. The definition of ternary samples is self-contradictory; FE supports the core argument of the entire text, which is a major flaw.

There are multiple numerical conflicts caused by non-rounding: the mean modal accuracy is inconsistent with the overall accuracy in the report; the sample count in the figure does not match the total number of data points; the model labels in the accompanying figures are confusing, and the corresponding models are not native visual models; the evaluation settings are unclear.

Questions for authors

Q1: Distinguish whether FE (Feature Analyzer) characterizes fusion failure or merely reflects the difficulty of the joint task. Output conditional multimodal accuracy and joint baseline; clarify the definition of triples, and disclose difficulty matching and effectiveness analysis.

Q2: Investigate numerical biases, explain the accuracy aggregation rules, the number of questions in each slice, and the scoring code; correct counting errors, and clarify the model and image link in Figure 1.

Q3: Supplement baseline validity materials: appendix, model output, version, annotation process, sample-by-sample prediction, script, and transfer results, focusing on verifying fusion gaps and the diagnostic power of FE.

If the response is sufficient, a re-evaluation may be conducted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.