- Rating
- 3
- Confidence
- 3(+1)
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates how answer requirements affect multimodal language models’ use of visual representations. It introduces the Guided Answer Space (GAS), an explicitly supplied set of candidate answers, and measures visual dependency depth through layer-wise interventions on visual token states. Across eleven models, converting TextVQA and GQA questions into two-choice selection reduces this depth by a median of 42% of model depth. Controlled experiments further show that increasing candidate count extends visual dependence under both option-label and answer-text outputs. Transfer and Freeze interventions demonstrate that intermediate visual states contribute substantially to open-ended answering, including when their values are held fixed in subsequent layers. Finally, a two-stage procedure constructs candidates from other models and selects an answer without ground-truth access. Across three target models and two datasets, it improves pooled accuracy by 3.56 percentage points and exceeds frequency voting on identical candidate pools by 2.56 points.
Critical strengths
-
A broad and carefully controlled investigation of visual dependence. The central comparison holds images, questions, and target answers fixed across answering settings, and the depth ordering holds across all eleven models on both TextVQA and GQA. The harder-distractor control, alternative state replacements, unchanged-prompt likelihood analysis, and candidate-count-by-output-format experiment address several obvious alternative explanations (Sections 3–5; Appendices A–D). This makes the contribution more substantial than a behavioral observation that multiple-choice questions are easier than open-ended questions.
-
Complementary evidence connecting internal representations to practical answering. Transfer and Freeze test different properties of intermediate visual states: their causal contribution to answering and their usefulness without subsequent state updates. The practical experiment also separates candidate generation from selection by comparing methods on identical pools. Reporting reverse errors, example-clustered uncertainty estimates, and image-replacement controls during selection strengthens the evidence for a real, if modest, inference benefit (Section 4; Appendix E.1). These analyses give the paper both scientific and practical value.
Critical weaknesses
-
W1: The interventions do not directly establish the cause of the model’s original errors. Transfer measures recovery from artificial image corruption, while Freeze measures how much performance survives when intermediate visual states stop updating. These experiments establish that the states contain useful information, but their aggregate results do not show that the information needed to answer originally misanswered examples was already available and subsequently misused. Moreover, freezing visual states still permits substantial downstream computation through text states. The high noise-image selection accuracies in Table 8, particularly on GQA, also show that candidate-conditioned language cues contribute considerably. The harder-distractor and unchanged-prompt controls help, but do not fully connect the representation interventions to the original failure cases. This limits the strength of the Answer Bottleneck interpretation, although it does not invalidate the observed dependency-depth differences.
-
W2: The answer-scoring protocol leaves the accuracy claims insufficiently validated. Appendix A.1 accepts a response whenever it contains the reference answer as a substring, and Appendix E.1 uses this rule for the practical inference results. This can accept incorrect numeric strings or responses containing incompatible answers; for example, a reference of “2” would match “12.” Consistent use of the rule does not ensure equal bias across free generation, candidate likelihood, voting, and selection, because these procedures can produce different response distributions. Appendix D.1 provides scoring sensitivity for the candidate-count experiment, but does not establish robustness of the headline practical gains. Rescoring with benchmark-standard metrics and auditing disagreements would materially strengthen confidence in the reported 3.56-point improvement. TextVQA’s benchmark evaluation follows the VQA accuracy protocol.
Questions for authors
-
Clarification and supporting analysis for W1: Can you report the intervention results separately for examples answered correctly and incorrectly under the original clean open-ended prompt? What evidence specifically connects useful intermediate representations to the examples whose original errors are corrected by GAS, rather than demonstrating preservation of already successful answering?
-
Clarification and robustness analysis for W1: On the same examples and with harder distractors, how much of the selection-versus-generation depth difference remains when evaluation is restricted to questions that require the image in both settings? Table 9 provides a related answer-text analysis, but not this paired comparison.
-
Validation of W2: Do the practical gains over original generation and frequency voting persist under benchmark-standard scoring? Please report the rescored gains and paired confidence intervals, together with the frequency of disagreements with substring matching.
4 comments
Comment 2 of 2
[Q2] Does the depth difference hold on the same questions when both settings require the image?
Because the question and options alone can suffice for some selections (Table 8), we follow your suggestion and compare both settings on the same questions, keeping only those that neither setting can answer without the image.
1. Questions that require the image in both settings. We keep TextVQA questions that both settings answer correctly with the real image and incorrectly when the image is replaced by a uniform gray image. Selection uses a distractor of the same answer type, and with the gray image the model must choose this distractor in both option orders. These distractors are thus plausible without the image, and the image makes both settings correct.
2. Depth on these questions. d is the sustained-crossing depth (Appendix A.2), which requires accuracy to stay within 5 points of the unmodified run at that layer and every later tested layer; we divide d by the number of decoder layers L (new; Section 3.2, Appendix B.2, Table 9):
| TextVQA model | n | Selection d/L | Open-ended d/L | Gap (fraction of L) |
|---|---|---|---|---|
| LLaVA | 24 | 0.44 | 0.94 | 0.50 [0.28, 0.58] |
| Qwen | 24 | 0.57 | 0.93 | 0.36 [0.07, 0.39] |
| InternVL | 19 | 0.57 | 0.96 | 0.39 [0.29, 0.39] |
On the same image-dependent questions, selection depends on visual tokens through 0.36L–0.50L fewer layers than open-ended answering, consistent with the 0.42L median in Section 3.2.
3. The result does not depend on the gray image or the 5-point threshold. Because a gray image differs from natural images, it could mark some questions as image-dependent for the wrong reason. To rule this out, we also identify image-dependent questions with the image of another TextVQA question, a natural image that does not answer the question. Selection remains shallower when questions are identified with this image, when both images must lead to failure, and when the allowed drop is 2 or 10 points instead of 5.
[Q3] Practical gains under benchmark-standard scoring
1. Rescoring. As the review notes, substring matching can accept incorrect answers, and its bias may differ across methods. To test whether the gains depend on this rule, we rescored all saved outputs with the standard ten-annotator VQA accuracy for TextVQA and exact match for GQA, without rerunning any model (new; Section 5.3, Appendix E.1).
2. Gains. The gain persists for the lowest-cost configuration in Table 30, in which each of the other two models generates one answer and the target model selects between them. Accuracy increases from 73.52% to 76.01%.
| Direct selection vs. (1,800 predictions) | Gain, pp |
|---|---|
| Direct generation | +2.49 [1.37, 3.62] |
| Frequency voting over identical candidates | +2.73 [1.29, 4.23] |
| Self-consistency over 8 target samples | +2.54 [1.29, 3.77] |
3. The four-answer configuration benefits from other models' answers. The +3.56 pp in the submitted abstract comes from the configuration in which each of the other two models generates four answers. Under the standard metrics, frequency voting over these eight answers raises accuracy by 1.25 pp [0.08, 2.42], whereas voting over eight samples from the target leaves it unchanged (−0.05 pp [−0.48, 0.37]). There, answers that recur across samples indicate the common answer; with one answer per model, no answer recurs, and direct selection supplies the gain. In both configurations, the gain comes from correct answers that other models generate and the target does not, as the Answer Bottleneck predicts.
4. Selection uses the image. The image-replacement control noted in the review (Appendix E.1), which uses four answers per model, is reproduced under the standard metrics: replacing the selector's image with a gray image or an image from another question reduces accuracy by 3.51 pp [1.58, 5.41] and 4.17 pp [2.16, 6.19], respectively.
5. Disagreement with substring matching. Standard and substring scores disagree on a similar fraction of predictions for every method, 14.2–14.9%, owing to additional references, partial credit and substring false positives. The rescoring thus affects all methods similarly.
Revision (in red). The new analyses appear in Section 4.1 and Appendix C.3 (Q1), Section 3.2 and Appendix B.2 (Q2), and Appendices A.1 and E.1 (Q3); the abstract, Introduction, Section 5.3, Figure 7 and Table 1 report the configuration with one answer per model.
Response to Authors
Thank you for the detailed response and the additional experiments. The analyses on originally incorrect examples provide a clearer connection between intermediate visual evidence and the errors corrected by GAS. I also appreciate the paired comparison on image-dependent questions and the benchmark-standard rescoring, which address my concerns about the depth comparison and the robustness of the practical gains.
These additions resolve my main concerns and strengthen the paper’s conclusions. I will maintain my rating and increase my confidence from 2 to 3.
Thanks a lot for the follow-up! Your suggestions really helped strengthen the paper.
Comment 1 of 2
Thank you for these constructive questions. Intervals are paired 95% CIs that resample questions, and pp denotes percentage points. Several analyses intervene at ℓ_sel, the selection dependency depth of each model and task (Section 3.2): in two-choice selection, zeroing visual token states from this layer onward leaves accuracy nearly unchanged.
[Q1] Do the interventions help originally incorrect answers, and what links them to the errors that a GAS corrects?
Sections 4.1–4.2 pool all questions, so, as the review notes, their gains could come mainly from questions that the model already answers correctly. To examine the original errors themselves, we restrict two analyses to the questions that each model originally answered incorrectly.
1. On original errors, the interventions still produce correct answers (new; Section 4.1, Appendix C.3, Table 15). If the Clean runs on these questions encoded evidence for the correct answer, supplying their intermediate visual states should produce correct answers that the original run did not give. Transfer (Section 4.1) runs the model on a noise image and, at ℓ_sel, replaces its visual token states with those of the clean-image run; the gain is measured over the noise-image run. Freeze (Section 4.2) holds the visual token states fixed from ℓ_sel onward; the gain is measured over zeroing them. On TextVQA, where answers must be read from the image, results pooled over eleven models are (n counts model–question cases):
Because the clean run scored 0 on the original errors, every correct answer in the left column is one that the original run did not give, so these gains cannot come from preserving successful answers. Both interventions yield such gains, Transfer in all eleven models; these Clean runs had thus already encoded evidence for the correct answer at ℓ_sel.
2. When a GAS corrects these errors, it uses this intermediate evidence (new; Appendix C.3, Table 16). When the correct answer is offered as one of two options, selection corrects a median of 91% of open-ended errors across eleven models (Appendix E.2, Table 33). As the review notes for Table 8, such corrections could also rely on language cues from the candidates. To separate the two sources, we let the model select on the original errors while limiting its image evidence to the intermediate states of the original run, and we compare the result with images that carry no relevant evidence.
The model chooses between the reference answer and a distractor of the same answer type. Because the image precedes the question and options, causal attention keeps its visual token states identical to those of the original open-ended run. We zero them from ℓ_sel onward, leaving these intermediate states as the only image evidence. We then repeat the choice with the image replaced by a uniform gray image, which leaves only the question and candidates, or by the image of another TextVQA question, a natural image that does not answer the question. Each question is run with the reference as option A and as option B and counts as correct only if both runs choose it. Results on the original errors among 300 TextVQA questions for which our predefined answer-type rules provide a distractor:
The real image exceeds both replacements in all three models. When a GAS corrects these errors, it therefore draws on the intermediate visual evidence of the original Clean run, beyond what the question and candidates support on their own.