acceptodds
Withdrawn from ICLR 2027

CROSSREF-VQA: Benchmarking Cross-modal Reference Resolution in Structured Graphics

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

While MLLMs perform well on everyday visual question answering (VQA) tasks, they remain unreliable when required to combine visual and textual information at the level of individual elements. Interpreting structured graphics demands exactly this: elements are marked with short references such as legend keys and colour codes with the corresponding semantic identities in a separate text section, leaving the reader to cross-reference. To study this limitation, we propose CrossRef-VQA, a benchmark explicitly designed to isolate element-precise cross-modal reference resolution. CrossRef-VQA renders structured data as graphics, replaces every semantic label in the image with an anonymous reference, supplies the reference-to-element mapping as text, and computes every answer programmatically from the underlying data. We intensionally incorporate human designed templates to avoid circular LLM evaluation and support scalable data generation, enabling stable large-scale assessment. Unlike traditional VQA benchmarks that evaluate a single, confounded setup – where full semantic information is redundantly present in both the image and the text – our framework introduces a set of controlled experimental conditions. By keeping the anonymised graphic fixed and systematically varying only the need to perform cross-modality referencing and de-referencing, we can directly compare performance across conditions to cleanly isolate cross-modal integration from pure visual perception. Across 8 chart types and 11 MLLMs, we find that requiring cross-referencing causes a significant overall performance drop of 15–30%. Notably, the two directions fail differently: referencing is imprecise when the question specifies an element, whereas de-referencing is skipped, where models correctly locate the visual element but return its raw label most of the time. These findings demonstrate that the fundamental failure point is not visual understanding, but the cross-modal reference resolution itself.

Primary area:
datasets and benchmarks
Keywords:
Benchmark, MLLM, Cross-modality integration, interpretability, chart question answering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_gQYzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces CrossRef-VQA, a benchmark for evaluating cross-modal reference resolution in multimodal large language models under Original, Anonymous, and Disjoint conditions. It analyzes performance degradation and errors when models must resolve anonymized chart labels using external semantic mappings.

Critical strengths
  1. The paired evaluation design provides a useful comparison of reference resolution with and without external semantic mappings.
  2. The manually designed question templates and programmatically generated ground truth ensure reproducible evaluation without relying on an LLM judge.
  3. The detailed error analysis distinguishes different failure types and provides insights across question categories, mapping directions, and chart types.
Critical weaknesses
  1. The comparison with existing chart question-answering benchmarks is insufficient, and the additional diagnostic value of CrossRef-VQA is not clearly demonstrated.
  2. Table 2 contains apparent inconsistencies in the O−A values. For example, 78.8−76.7 equals +2.1 percentage points, rather than the reported −0.7, raising concerns about the calculation and the related conclusions.
  3. The interpretation of Mapping Gap is overstated, as improvements and regressions between conditions A and D may cancel each other out. A near-zero gap alone cannot establish robust mapping or support the claim that the model perfectly bridges the modalities.
Questions for authors
  1. How does CrossRef-VQA differ from existing benchmarks such as ChartQA and PlotQA, and what additional diagnostic insights does it provide?
  2. How are the O−A values in Table 2 calculated, and would correcting the inconsistencies affect the conclusions?
  3. Can the authors report paired outcome statistics for conditions A and D to verify whether a near-zero Mapping Gap reflects robust reference resolution?
Reviewer_thxcOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents CROSSREF-VQA, a benchmark for diagnosing cross-modal reference resolution in structured graphics. Specifically, the benchmark evaluates whether MLLMs can resolve references between anonymized graphcics and external textual mappings. Paired original, anonymous, and disjoint conditions are introduced. The main finding is that the MLLMs' performance drops when external mappings are required.

Critical strengths
  1. The task is a every-day task relevant to interpreting charts, diagrams, and other graphics with detached legends or reference keys.
  2. The finding that MLLMs often identify the correct visual element but return its anonymous label is interesting.
Critical weaknesses
  1. The mechanistic conclusions is too "strong" given the conducted experiments. Specifically, performance differences and output errors do not confirm failures in cross-modal working memory or the vision encoder. Similarly, results across one model family, Qwen3.5, do not confirm that parameter scaling is generally ineffective.
  2. The paper does not test whether simply prompting “identify the reference, then translate it using the key” resolves the skipped-mapping errors.
  3. Although the identified model failures are interesting, the overall insights from the findings are limited.
Questions for authors
  1. Does simple promoting strategies narrow the gap?
  2. The O−A column in Table 2 do not match the reported O and A?
Reviewer_Hj86OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces CrossRef-VQA, a generator for benchmark evaluation sets that tests wether multimodal LLMs (MLLM) can resolve references and dereferences between a chart image and a separate text key. The charts are rendered from LLM-populated schemas, semantic labels are replaced by anonymous references and answers are computed programmatically from the underlying data. The generated evaluation set has 154 charts and 1,305 question-answer pairs across eight chart types. Eleven state of the art MLLMs are compared under an Original (O), Anonymous (A) and Disjoint condition. The authors report a 15 to 30 point drop in accuracy from A to D, which they call the mapping gap and attribute it mainly to dereferencing failures, where models return the annonymized dummy label instead of the semantic name. They conclude that the bottleneck is cross-modal reference resolution rather than visual understanding and that it does not scale with model size.

Critical strengths

• The benchmark data appears well constructed. The eight chart types and five question categories give good coverage. • Answers are computed programmatically, so ground truth does not depend on an LLM. • Results and error types are broken down by question category and required operation (referencing and dereferencing or both), which gives more insight than a single accuracy. • The paired O,A,D design is a good idea for separating perception from cross-modal mapping.

Critical weaknesses

• The prompt is not explored, which undermines the main claim. It is unclear whether the models are worse at mapping or whether they were simply never told which kind of answer to give. According to Fig. 2, most errors under D are Label errors (the correct element is found but the dummy label is returned), not wrong answers. If the text says “L1 = Solar” and the model answers “L1”, that is arguably correct, since the prompt never asked for semantic names. The prompts are not shown, and there is no prompt ablation. • There is no mapping-only sanity check. Nothing tests whether a model can resolve “L1 = Solar” from the text key alone. Together with the better referencing results, this suggests models can do the mapping, and the headline gap may partly disappear with a better prompt. • The claim that any degradation in D reflects an inability to integrate the textual mapping is too strong. The models may be able to do it and simply not consider it necessary. The paper cannot distinguish “unable” from “not prompted to”. • Chance levels are mentioned but never stated, so the unimodal ablations cannot be compared against chance without the numbers. • O−A is inconsistent with the accuracies in Table 2. For GPT-6 Sol, 78.8 − 76.7 = 2.1, not −0.7. Either O−A is computed differently (which is not explained) or there is an error. This matters because the claim that anonymisation is harmless rests on the O−A column. • The claim that mapping robustness does not correlate with parameter count is unsupported. The parameter counts of the evaluated models are not given, so the claim cannot be checked.

Questions for authors

• What exact prompt was given to the models? Were they instructed to answer with the semantic names from the key? • How exactly is O−A calculated? Why do the values not equal the difference of the reported accuracies? • What are the chance levels for the question types? • What are the parameter counts of the evaluated models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.