acceptodds
Under review as a conference paper at ICLR 2027

Question-Specific Table Values Reduce Sensitivity to the Chart in Vision-Language Models

Abstract

When a model receives both a chart and an extracted table, it can in principle check the table against the chart. We test whether restating the table entries as the two quantities a question compares makes correct answers less likely when these sources disagree. From 96 synthetic specifications, we render opposite-answer image pairs and cross them with tables that match only one image. Three 8B–12B models receive an experimenter-supplied assistant report that either repeats four table entries or states the two quantities the question compares; both versions come from the same table and are padded to equal token length. Under direct answering, question-specific values lower conflict accuracy from 59–68% to 0–13% while raising agreement accuracy from 84–90% to 96–100% across JSON and prose. In a separate study of model-extracted values on 96 ChartQA-derived images, adding a calculation report to each model's own work record improves average accuracy by 9–17 percentage points across five models, including additional Qwen and InternVL 4B checkpoints. Among the 26 conditions where the extracted values reverse the comparison, however, correct final answers fall from 13 to zero. A chart-directed reasoning request attenuates or reverses the controlled representation effect in seven of eight model–format combinations on four 2B–4B checkpoints (24 reused specifications). On a new panel with the main models, cutting report attention in the middle third of the decoder raises accuracy with question-specific values by 26–33 percentage points, and jointly cutting image attention removes that gain. The raw/values contrast differs significantly between report and equally long text-control cuts for Qwen and InternVL, but not Gemma. These results show that chart sensitivity depends on the form of a table-derived report and on the answering protocol.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, chart question answering, image-text conflict, visual grounding, behavioral evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Uq1COpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates how table-derived reports influence vision-language models’ reliance on chart images. Using synthetic chart pairs with opposite answers, it compares reports repeating raw entries against reports presenting question-specific quantities. Under direct answering, the latter improves accuracy when chart and table agree but reduces accuracy when they conflict. Additional experiments examine reports computed from model-extracted values on ChartQA-derived images, alternative answering protocols, and attention interventions. Chart-directed reasoning attenuates or reverses the representation effect, while middle-layer report attention cuts improve performance. The findings characterize how intermediate report representations and answering protocols jointly affect model behavior when visual and textual information disagree.

Critical strengths
  1. The attention intervention analysis offers useful diagnostic evidence. Blocking access to report tokens in middle decoder layers substantially improves accuracy under the Values condition, motivating further investigation into how models combine visual evidence with intermediate textual reports.

  2. The paired experimental design provides a clear comparison: image changes preserve the text, while report replacements preserve the image, question, and table. Grouping statistical analyses by base specification appropriately accounts for dependence among derived conditions.

Critical weaknesses
  1. Limited experimental scope. The main study uses only 96 synthetic base specifications and three simple comparison tasks. Testing many conditions on these same specifications does not show whether the findings hold on more varied charts or questions. The limited model coverage also leaves their applicability to stronger models uncertain.

  2. The observed effect has several unresolved explanations. Values may make one answer option easier to select, or models may simply trust the inserted assistant report instead of checking the chart. The experiments do not clearly separate these possibilities. Consequently, it remains unclear what the results reveal about chart understanding beyond sensitivity to how the report presents information.

  3. The main protocol may shape the reported failures. Models receive an experimenter-written assistant report and must answer without written reasoning. This differs from directly asking a model to inspect a chart and solve a question. The paper does not adequately establish whether the same effect persists without the inserted report or the answer-only restriction.

  4. Raw and Values differ in more than precomputation. The comparison also changes which quantities are selected, how they align with answer options, and which operation cues are provided. Matching token length does not control these differences. The read-only experiment shows that arithmetic is unnecessary for the effect, but does not establish which other change causes it.

  5. The follow-up does not isolate what causes the reversal. Chart-directed reasoning weakens or reverses the effect, even though the original prompt already directs models to the chart. However, the follow-up changes both the wording and permission to reason. Without testing these changes separately, it is unclear which explains the reversal and when the main finding should hold.

Questions for authors
  1. The evaluation relies on only 96 synthetic specifications, which provides limited evidence for the robustness of the conclusions. Can you test whether the main findings hold on substantially more independent charts and stronger models? Repeating conditions on the same specifications does not address this concern.

  2. The auxiliary results suggest that the findings depend strongly on the prompting protocol. Which changes drive the reversal: the chart-directed wording, allowing reasoning, or both? Does the main conclusion still hold with ordinary chart-question prompts that allow reasoning and omit the answer-only restriction?

  3. The Raw–Values comparison changes several factors together, including quantity selection, A/B labeling, operation cues, and arithmetic precomputation. The current results therefore do not isolate which factor drives the difference. Can you vary these components separately, particularly testing precomputation while keeping the selected information and option alignment unchanged?

Reviewer_LYd5OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This paper explores how providing information as text alongside a chart will influence the model’s performance in ChartQA. Specifically, the paper tests whether there is a difference in effect between A) providing the raw parsed values and B) providing intermediate calculated values. To study this, the paper tests providing both correct and incorrect information as text.

For direct answer, the results show that providing intermediate values as text has a larger effect on model accuracy than providing the raw values. The paper also shows that providing this information as JSON vs. prose may have some differences based on what model is being tested, but the formatting option with higher influence is inconsistent.

For chain-of-thought, the effect flips: the results show that providing raw values as text has a larger effect on model accuracy than providing the intermediate values.

The paper also experiments with blocking attention to different positions in the context (image vs. text) to observe the effect on final answer accuracy.

Critical strengths

None

Critical weaknesses
  1. This paper’s presentation is exceptionally poor. It is extremely difficult to read, and obscures the paper’s takeaways and purpose.
  2. It is unclear what the value of this work is. How can the paper’s findings be used to improve how future models are built, or improve how we apply current models to specific use cases?
Questions for authors
  1. What are the practical takeaways from this work? How can this paper's observations help us improve future models, or how we use current ones?

  2. Can the authors carefully proofread this work and make major changes to improve overall clarity and make the paper’s contributions clear? Some suggestions:

L043: “If the answer reverses the comparison”

This is confusing because at this point in the introduction, It has not been made clear that the paper specifically tests comparison tasks. Please explain the task so that the reader can understand what “reverses the comparison” means.

L087: What does “lower signed chart response” mean?

L091–095: This introduction paragraph is confusing because it has not been made clear what the “contrast” being examined is, or how it is measured. Without this, it is unclear why it is significant that something “reverses its sign”. Also, “middle band report” and “values-arm” have also not been defined at this point in the paper.

L151: “We estimate the effect of this complete replacement; matching token length does not isolate these components or equalize the salience of the relevant quantities.”

Hard to understand. What does it mean to “equalize the salience of the relevant quantities”. Why do you say that you “estimate the effect of this complete replacement”—i.e., what is the alternative thing that could be estimated, that you’re trying to make clear that you’re not doing?

L153: This paragraph is called “Crossed worlds.” While this is catchy and sensational-sounding, I have no idea what it means.

L156-158: What does “two rendered worlds” mean? This passage uses the word “alternative” multiple times without explaining what it means in this context—e.g., what does “correct canonical alternative ww” mean? What does “alternatives rr” mean? Does “alternative” just refer to the fact that the model has binary choices in each case?

L159-160: Please explain A) what the “two option orders” are, B) what it means to have “annotated or unannotated marks”, and C) what the two “representations are. Perhaps a figure with an example would help, and you can also show the JSON and prose formats as well.

L172-190: What does “canonical +1 alternative” mean? What does “balanced finite design” mean? What are “repeat/backend discrepancies”?

L195: What does “a shift toward the table” mean?

The remainder of the paper is similarly difficult to read. The above suggestions should be taken as an indication of the density and extent of the edits to polish this paper.

Reviewer_z7iGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how question-specific numerical summaries derived from chart tables affect the reliance of vision-language models (VLMs) on visual evidence. Through controlled experiments involving conflicting chart and table information, the authors demonstrate that providing precomputed, question-relevant values substantially increases models' reliance on textual information, even when it contradicts the chart. The paper further examines this phenomenon through natural chart extraction, reasoning prompts, and attention interventions, highlighting potential reliability issues in multimodal reasoning.

Critical strengths

1 The paper investigates a practically important failure mode in multimodal reasoning, showing how seemingly helpful numerical summaries can inadvertently reduce VLMs' reliance on visual evidence.

2 The authors systematically manipulate chart–table agreement, report formats, chart types, and question types while controlling for report length, providing convincing empirical evidence for the observed phenomenon.

3 Beyond the main controlled experiments, the paper includes natural chart extraction, reasoning interventions, and attention-based analyses across multiple VLMs, offering complementary perspectives on the robustness and potential mechanisms of the observed behavior.

Critical weaknesses

1 The comparison between raw table entries and question-specific values simultaneously changes several factors, including the selection of relevant information, alignment with answer options, explicit operation cues, and numerical precomputation. Although the authors control for token length, this does not isolate the contribution of each factor. Consequently, it remains unclear whether the dramatic reduction in chart sensitivity is primarily caused by numerical computation, answer-option alignment, or simply the increased salience of task-relevant textual information. More fine-grained ablation studies are needed to support the proposed interpretation.

2 The attention intervention experiments provide useful observations but do not establish a clear causal mechanism explaining why question-specific values reduce visual reliance. Although suppressing attention to the report in the middle decoder layers improves accuracy under conflicting inputs, cutting image attention alone already reduces performance to approximately chance level. Moreover, the report-cut effect is not significantly different from the matched text-control effect for Gemma. These findings make it difficult to identify a specific report–image interaction or a unique computational mechanism responsible for the observed behavior. Stronger causal analyses would be necessary to substantiate the mechanistic interpretation.

3 The manuscript is generally well written, but several passages could be made more precise and easier to follow. In particular, some descriptions in the abstract use compressed terminology (e.g., “report attention” and “raw/values contrast”), and the discussion of the protocol results in Section 6 contains a potentially ambiguous reference to “the exception.” Clarifying these expressions and improving terminological consistency would further improve readability.

Questions for authors

1 Will the source code and datasets be made publicly available in the future?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.