- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates how table-derived reports influence vision-language models’ reliance on chart images. Using synthetic chart pairs with opposite answers, it compares reports repeating raw entries against reports presenting question-specific quantities. Under direct answering, the latter improves accuracy when chart and table agree but reduces accuracy when they conflict. Additional experiments examine reports computed from model-extracted values on ChartQA-derived images, alternative answering protocols, and attention interventions. Chart-directed reasoning attenuates or reverses the representation effect, while middle-layer report attention cuts improve performance. The findings characterize how intermediate report representations and answering protocols jointly affect model behavior when visual and textual information disagree.
Critical strengths
-
The attention intervention analysis offers useful diagnostic evidence. Blocking access to report tokens in middle decoder layers substantially improves accuracy under the Values condition, motivating further investigation into how models combine visual evidence with intermediate textual reports.
-
The paired experimental design provides a clear comparison: image changes preserve the text, while report replacements preserve the image, question, and table. Grouping statistical analyses by base specification appropriately accounts for dependence among derived conditions.
Critical weaknesses
-
Limited experimental scope. The main study uses only 96 synthetic base specifications and three simple comparison tasks. Testing many conditions on these same specifications does not show whether the findings hold on more varied charts or questions. The limited model coverage also leaves their applicability to stronger models uncertain.
-
The observed effect has several unresolved explanations. Values may make one answer option easier to select, or models may simply trust the inserted assistant report instead of checking the chart. The experiments do not clearly separate these possibilities. Consequently, it remains unclear what the results reveal about chart understanding beyond sensitivity to how the report presents information.
-
The main protocol may shape the reported failures. Models receive an experimenter-written assistant report and must answer without written reasoning. This differs from directly asking a model to inspect a chart and solve a question. The paper does not adequately establish whether the same effect persists without the inserted report or the answer-only restriction.
-
Raw and Values differ in more than precomputation. The comparison also changes which quantities are selected, how they align with answer options, and which operation cues are provided. Matching token length does not control these differences. The read-only experiment shows that arithmetic is unnecessary for the effect, but does not establish which other change causes it.
-
The follow-up does not isolate what causes the reversal. Chart-directed reasoning weakens or reverses the effect, even though the original prompt already directs models to the chart. However, the follow-up changes both the wording and permission to reason. Without testing these changes separately, it is unclear which explains the reversal and when the main finding should hold.
Questions for authors
-
The evaluation relies on only 96 synthetic specifications, which provides limited evidence for the robustness of the conclusions. Can you test whether the main findings hold on substantially more independent charts and stronger models? Repeating conditions on the same specifications does not address this concern.
-
The auxiliary results suggest that the findings depend strongly on the prompting protocol. Which changes drive the reversal: the chart-directed wording, allowing reasoning, or both? Does the main conclusion still hold with ordinary chart-question prompts that allow reasoning and omit the answer-only restriction?
-
The Raw–Values comparison changes several factors together, including quantity selection, A/B labeling, operation cues, and arithmetic precomputation. The current results therefore do not isolate which factor drives the difference. Can you vary these components separately, particularly testing precomputation while keeping the selected information and option alignment unchanged?