Question-Specific Table Values Reduce Sensitivity to the Chart in Vision-Language Models

When a model receives both a chart and an extracted table, it can in principle check the table against the chart. We test whether restating the table entries as the two quantities a question compares makes correct answers less likely when these sources disagree. From 96 synthetic specifications, we render opposite-answer image pairs and cross them with tables that match only one image. Three 8B–12B models receive an experimenter-supplied assistant report that either repeats four table entries or states the two quantities the question compares; both versions come from the same table and are padded to equal token length. Under direct answering, question-specific values lower conflict accuracy from 59–68% to 0–13% while raising agreement accuracy from 84–90% to 96–100% across JSON and prose. In a separate study of model-extracted values on 96 ChartQA-derived images, adding a calculation report to each model's own work record improves average accuracy by 9–17 percentage points across five models, including additional Qwen and InternVL 4B checkpoints. Among the 26 conditions where the extracted values reverse the comparison, however, correct final answers fall from 13 to zero. A chart-directed reasoning request attenuates or reverses the controlled representation effect in seven of eight model–format combinations on four 2B–4B checkpoints (24 reused specifications). On a new panel with the main models, cutting report attention in the middle third of the decoder raises accuracy with question-specific values by 26–33 percentage points, and jointly cutting image attention removes that gain. The raw/values contrast differs significantly between report and equally long text-control cuts for Qwen and InternVL, but not Gemma. These results show that chart sensitivity depends on the form of a table-derived report and on the answering protocol.