acceptodds
Under review as a conference paper at ICLR 2027

Replay Before You Reply: Second-Look Prompting for Unified Multimodal Encoding

Abstract

Solving multimodal tasks requires Vision-Language Models (VLMs) to form a unified understanding of the image and the question text. However, under causal attention in current VLMs, a token can attend only to what precedes it. Whichever modality comes first is therefore encoded under partial observation of the input, uninformed by the rest of the multimodal context. We introduce Second-Look Prompting (SLP), a training-free method that repeats the image-question pair in the input. In the repeated copy, every visual and textual token can attend to the complete input, yielding a unified multimodal encoding of the pair. Moreover, the added visual tokens contribute computation beyond re-exposing the image–even a blank image in their place can improve reasoning. Across 12 benchmarks, SLP improves the overall average score of every evaluated model, spanning nine open-source models (3B-72B) and three proprietary frontier models, with gains reaching 5.6 points for Gemini 3 Flash on MATH-Vision. Our analysis reveals two complementary mechanisms: the question guides where the second image attends, and the added visual tokens act as a prefill scratchpad whose hidden states retain answer-relevant information from the image and question. These findings highlight input presentation as a simple yet effective way to improve VLM performance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Vision-Language Models, Prompting, Causal Attention, Prompt Repetition, Training-Free Methods, Prefill Computation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_hzAcOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Second-Look Prompting (SLP), a training-free approach that repeats the image-question pair to improve multimodal understanding in vision-language models. The authors argue that causal attention limits cross-modal information exchange during initial encoding, while repeated inputs enable question-guided visual representations. Extensive experiments across multiple models and benchmarks demonstrate consistent average improvements, supported by attention analysis and scratchpad probing.

Critical strengths

1.The paper is well written and presents a clear motivation.

2.The experimental evaluation is comprehensive and covers diverse models and benchmarks.

Critical weaknesses

1.Limited methodological novelty. The proposed method primarily relies on repeating existing inputs, which is a relatively straightforward modification of the prompting strategy. Although the authors provide interesting interpretations, the methodological contribution remains incremental compared with prior studies on prompt repetition and input ordering.

2.Insufficient evidence for the proposed mechanisms. The attention visualizations and scratchpad probing provide useful insights, but they do not fully establish that the proposed mechanisms are responsible for the observed improvements. In particular, the effectiveness of blank-image tokens varies across model architectures, raising questions about the generality of the scratchpad interpretation.

3.Limited performance gains and practical applicability. Despite consistent average improvements, the overall gains are relatively modest, and some tasks, such as visual grounding, experience performance degradation. Furthermore, duplicating the input increases token consumption and training costs, while the evaluation is largely restricted to single-image scenarios, leaving its applicability to more complex multimodal inputs unclear.

Questions for authors

1.Can the authors provide stronger causal evidence that question-guided visual attention directly contributes to the performance improvements, rather than merely correlating with them?

2.Beyond the existing repetition and blank-image controls, how does SLP compare with token- or compute-matched baselines? To what extent can the improvements be attributed to additional computation rather than the proposed encoding mechanism?

5 comments
AuthorsOpenReview ↗

Response to Reviewer hzAc (1/2)

We appreciate your valuable feedback and support of our work. We detail our response below and please kindly let us know if our response addresses your concerns.

To W1:

W1: Limited methodological novelty ... methodological contribution remains incremental compared with prior studies on prompt repetition and input ordering.

We appreciate your positive assessment of our interpretations and agree that the prompt modification itself is simple. Our goal is to identify and analyze the cross-modal partial-observation problem in causal VLMs, where ⟨V, T⟩ prefill leaves visual tokens unconditioned on the question. A minimal, parameter-free intervention lets us isolate this encoding gap cleanly, just as simple interventions like zero-shot chain-of-thought [1] and register tokens [2] helped reveal underlying model behaviors.

Practically, this simplicity makes SLP an easy plug-in across closed- and open-source VLMs (as noted by Reviewer 5JNq), and our new results on Qwen3.8-27B at xhigh (+2.91 points, Table R2) show that recent model advances have not closed this gap. Beyond the prompt rule itself, our mechanism and training analyses offer concrete guidance for future multimodal designs. We therefore believe that the methodological simplicity of SLP strengthens rather than diminishes its value.

References

[1] Kojima et al. Large Language Models are Zero-Shot Reasoners. NeurIPS 2022.

[2] Darcet et al. Vision Transformers Need Registers. ICLR 2024.


To W2:

W2: ... the effectiveness of blank-image tokens varies across model architectures, raising questions about the generality of the scratchpad interpretation.

We respectfully note a misunderstanding of the blank-image experiment. Blank images do not need to work across every vision encoder for the scratchpad mechanism of SLP to hold. In actual SLP (Copy V), all evaluated architectures, including Qwen2.5-VL-32B, show strong scratchpad readout and generation gains (Table R1). Although Appendix E.3 offers one possible architectural explanation for Qwen2.5-VL's near-zero blank-image readout, our argument does not rely on it. Crucially, blank images improve accuracy precisely where blank-image readout succeeds (MiniCPM-o 4.5 and Gemma-3-12B-IT) and yield no gain where readout fails (Qwen2.5-VL-32B), serving as positive-and-negative control evidence for the scratchpad mechanism (as also noted by Reviewer 5JNq). Per-benchmark results are added to Appendix E.4.

Table R1. Scratchpad probe gains (Table 3) and normal-generation Δ over ⟨V, T, T⟩ (six-benchmark average, points)

ModelCopy V (SLP) probeCopy V (SLP) Gen. ΔBlank probeBlank Gen. Δ
MiniCPM-o 4.5+34.5+0.89+35.8+0.59
Gemma-3-12B-IT+34.9+0.91+34.8+0.35
Qwen2.5-VL-32B+38.2+0.99−0.9+0.02

To W3:

W3: Limited performance gains and practical applicability ... visual grounding degradation ... token consumption and training costs ... restricted to single-image scenarios.

Thank you for raising these practical points. We agree on the scope boundaries discussed in our Limitations section, where image duplication can interfere with fine coordinate grounding on RefCOCOg and extending SLP beyond single-image ⟨V, T⟩ inputs remains future work.

Regarding practical value and cost, SLP is applied directly at inference without any need for training (our continued-training experiment only studies how the capability develops). Even on Qwen3.8-27B at its highest effort (xhigh), SLP improves all five challenging benchmarks by +2.91 points on average (Table R2, Appendix K). We additionally provide a lightweight variant, SLP-quarter, which downsamples only V₂ to a quarter of its visual-token budget. It slightly improves accuracy over full SLP while keeping the additional visual tokens within 28% of ⟨V, T⟩ (Table R3). Per-benchmark results are added to Appendix I.2.

Table R2. Qwen3.8-27B (xhigh) accuracy on five challenging benchmarks (%)

Benchmark⟨V, T⟩SLPΔ
MATH-Vision84.7785.66+0.89
BabyVision46.1352.84+6.70
RealWorldQA84.9786.41+1.44
ERQA67.2568.99+1.74
CharXiv79.2083.00+3.80
Average72.4675.38+2.91

Table R3. Cost and accuracy of SLP-quarter

PresentationVisual tokensAcc. (%)E2E latency (s)
Qwen2.5-VL-32B
⟨V, T⟩141668.895.04
SLP283369.755.32
SLP-quarter177869.955.19
Qwen3.8-27B (non-thinking)
⟨V, T⟩78459.8740.43
SLP156862.5543.96
SLP-quarter99962.9840.86
AuthorsOpenReview ↗

Response to Reviewer hzAc (2/2)

To Q1:

Q1: Can the authors provide stronger causal evidence that question-guided visual attention directly contributes to the performance improvements, rather than merely correlating with them?

We intervene on V₂'s attention during standard generation on LogicVista with Qwen2.5-VL-32B (Table R4), masking only V₂'s attention to (V₁, T₁) (V₂-no-read) using FlexAttention [1]. This lowers accuracy from 59.28% to 55.92% and removes the entire SLP gain over ⟨V, T, T⟩ (56.82%), showing that question-guided attention in V₂ directly contributes to the improvement. The ⟨V, V, T, T⟩ order control in Q2 provides converging evidence without altering the attention mask. We have added this result to Appendix E.4.

Table R4. Attention-blocking intervention on LogicVista (Qwen2.5-VL-32B, %)

⟨V, T, T⟩SLPV₂-no-read
56.8259.2855.92

To Q2:

Q2: ... how does SLP compare with token- or compute-matched baselines? To what extent can the improvements be attributed to additional computation rather than the proposed encoding mechanism?

We compare SLP with two token- and prefill-compute-matched baselines (Table R5).

  1. ⟨V, T, Spaces, T⟩ replaces V₂ with length-matched spaces. It keeps the additional computation but provides no question-conditioned second view of the image nor a visual scratchpad, and yields no average gain.
  2. ⟨V, V, T, T⟩ places V₂ before the question, which is intra-modal repetition: V₂ attends to V₁ but not to T₁. Its average gain of +0.29 points is consistent with the +0.4-point gain of ⟨V, V, T⟩ over ⟨V, T⟩ (Table 2).

SLP outperforms both baselines on all three models, with an average gain of +0.93 points. Additional computation alone thus explains little of the improvement, and resolving the cross-modal partial-observation problem brings further gains beyond intra-modal repetition. Per-benchmark results are added to Appendix E.4.

Table R5. Gains over ⟨V, T, T⟩ of token- and prefill-compute-matched baselines

Model⟨V, T, T⟩ (%)⟨V, T, Spaces, T⟩⟨V, V, T, T⟩SLP
MiniCPM-o 4.567.90+0.03+0.58+0.89
Gemma-3-12B-IT55.73+0.13+0.45+0.91
Qwen2.5-VL-32B68.76−0.16−0.16+0.99
Average64.130.00+0.29+0.93

Thank you again for your support and thoughtful suggestions, which have strengthened both our mechanism evidence and the practical case for SLP. We would be happy to discuss any remaining questions.

References

[1] Dong et al. Flex Attention: A Programming Model for Generating Optimized Attention Kernels. MLSys 2025.

Dear reviewer hzAc,

We hope our responses have addressed your questions. Please let us know if there is anything else we can clarify or improve before the discussion ends. Your feedback is very important to us, and we welcome any further discussion.

Thanks again for your time and effort in reviewing our submission.

Reviewer_5JNqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Second-Look Prompting repeats the image and question in a vision-language model prompt. This lets the second image block attend to the question before the model answers. The main evaluation covers twelve models and twelve benchmarks. The paper also investigates why repetition helps through attention analysis, restricted readout probes, and blank-image controls. The revision adds comparisons with question–image–question prompts, normal-generation controls, and tests at higher reasoning budgets.

Critical strengths
  1. The method is simple to apply, and the evaluation covers several model families and different tasks. The added question–image–question baseline addresses an important gap in the original comparison.

  2. The rebuttal provides useful evidence beyond the restricted probes. The normal-generation filler tests, attention intervention, and order control strengthen the case that the extra visual block contributes to the gains.

Critical weaknesses
  1. The practical impact appears limited. Average improvements are generally modest, while repeating the image increases input processing. Latency benefits depend on the model and setting, and the grounding results show that repetition can also hurt performance.

  2. The mechanism explanation remains incomplete. The attention intervention covers one model and one benchmark and blocks access to both the original image and question. The order control adds supporting evidence, but these results do not establish a complete explanation across model families.

Questions for authors
  1. The additional experiments address my original questions sufficiently. I have no further questions that would change my recommendation. Please retain the qualifications about the mechanism and efficiency results in the final version.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 5JNq (1/3)

Thank you for reviewing our paper and for your valuable feedback. Below, we address your concerns point by point and we’ve revised our paper according to your suggestions. We would appreciate it if you could let us know whether your concerns are addressed by our response.


To W1 and Q1:

W1 & Q1: How does question–image–question compare with SLP in accuracy, input token count, and latency?

We evaluated ⟨T, V, T⟩ (question–image–question) across all 12 benchmarks in Table R1. Repeating the image provides a clear benefit, with SLP outperforming ⟨T, V, T⟩ across all three models while ⟨T, V, T⟩ falls below single-pass ⟨V, T⟩. This difference from concurrent work [1] is reasonable because their main tests restrict outputs to 16-token Yes/No or option answers on natural-image VQA (where Qwen2.5 also does not fully recover image-first accuracy under ⟨T, V, T⟩), whereas our 12 benchmarks evaluate open-ended reasoning, diagrams, and OCR. We have added these results to Appendix C.1.

Timing both prompts on the same set (10% per benchmark on Qwen2.5-VL-32B, A100; Table R2) further shows that although SLP takes more input tokens, it generates 12.4% fewer output tokens and finishes 6.6% faster end to end, yielding better token efficiency.

Table R1. Accuracy of ⟨V, T⟩, ⟨T, V, T⟩, and SLP across all 12 benchmarks (%)

ModelPromptDynaMathMATH-VisionLogicVistaVisualPuzzlesMMMUMathVistaWe-MathReas. Avg.AI2DOCRBenchSEED-2-PlusMMBenchMMStarPerc. Avg.Overall
Qwen2.5-VL-32B⟨V, T⟩67.037.354.438.265.872.746.954.683.685.572.685.566.878.864.7
⟨T, V, T⟩67.139.456.440.963.074.249.455.878.383.570.780.564.175.464.0
SLP67.538.857.740.867.774.949.456.784.485.871.886.167.779.266.1
MiniCPM-o 4.5⟨V, T⟩61.646.051.216.469.272.157.553.480.881.865.484.966.675.962.8
⟨T, V, T⟩61.145.450.819.268.971.757.153.480.381.064.784.565.475.262.5
SLP62.546.154.115.370.874.257.054.381.482.365.785.567.776.563.6
Gemma-3-12B-IT⟨V, T⟩30.831.236.916.451.654.135.336.680.372.967.275.254.770.150.5
⟨T, V, T⟩31.130.233.115.547.148.925.733.174.967.263.875.249.366.146.8
SLP37.431.437.116.253.054.935.938.082.271.167.479.256.871.351.9

Table R2. Inference cost of ⟨T, V, T⟩ vs. SLP on Qwen2.5-VL-32B (10% per benchmark)

PresentationInput tokensOutput tokensE2E latency (s)
⟨T, V, T⟩1603.9386.66.77
SLP3014.1338.86.32

References

[1] Abhinandan et al. Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models. ECCV Workshop 2026.

AuthorsOpenReview ↗

Response to Reviewer 5JNq (2/3)

To W2 and Q2:

W2 & Q2: During normal generation, do length-matched text fillers provide gains comparable to blank-image tokens? Can a targeted attention intervention connect the information retained in the added block to the observed accuracy gains?

Thank you for pointing this out. We agree that restricted-probe readability alone does not determine how much the retained information is used during normal generation. Thus, we provide the following additional results.

In Table R3, we evaluate length-matched text fillers (Spaces, ⟨V, T, Spaces, T⟩) during normal generation. Normal-generation gains track probe readability only partially for text fillers, as Spaces yields much smaller gains than readable visual blocks (Blank and SLP) even when its probe score is high. Visual tokens thus act as a more usable scratchpad than text fillers during generation, and text-token probe scores are not directly comparable to visual-token probe scores.

Table R3. Scratchpad probe gains (Table 3) and normal-generation Δ over ⟨V, T, T⟩ (six-benchmark average, points)

Added block RProbe gainGen. Δ
MiniCPM-o 4.5
Copy V (SLP)+34.5+0.89
Blank Image+35.8+0.59
Spaces+0.2+0.03
Gemma-3-12B-IT
Copy V (SLP)+34.9+0.91
Blank Image+34.8+0.35
Spaces+29.2+0.13
Qwen2.5-VL-32B
Copy V (SLP)+38.2+0.99
Blank Image−0.9+0.02
Spaces+40.9−0.16

We also test a targeted attention intervention on LogicVista (Table R4). Keeping the SLP input unchanged, we mask only V₂'s attention to (V₁, T₁) (V₂-no-read) using FlexAttention [1]. This lowers accuracy from 59.28% to 55.92% and removes the entire SLP gain over ⟨V, T, T⟩ (56.82%).

Table R4. Targeted V₂ attention-blocking intervention on LogicVista (Qwen2.5-VL-32B, %)

⟨V, T, T⟩SLPV₂-no-read
56.8259.2855.92

To further address your question without altering the attention mask, we evaluated an equal-input-budget order control ⟨V, V, T, T⟩ under standard generation (Table R5). Placing V₂ before T₁ naturally prevents V₂ from reading T₁, and SLP still outperforms ⟨V, V, T, T⟩ on all three models (+0.31 to +1.15 points).

Table R5. Unmasked equal-input-budget order control (%)

Model⟨V, T, T⟩⟨V, V, T, T⟩SLPSLP − ⟨V, V, T, T⟩
MiniCPM-o 4.567.9068.4868.79+0.31
Gemma-3-12B-IT55.7356.1856.64+0.46
Qwen2.5-VL-32B68.7668.6069.75+1.15

To sum up, these results indicate that the normal-generation gain of SLP depends on the question-conditioned information retained in V₂: length-matched text fillers do not reproduce it, and it is removed by masking and reduced by reordering once V₂ cannot read the question. We have revised Section 3.3.2, and per-benchmark results are added to Appendix E.4.

References

[1] Dong et al. Flex Attention: A Programming Model for Generating Optimized Attention Kernels. MLSys 2025.

AuthorsOpenReview ↗

Response to Reviewer 5JNq (3/3)

To Q3:

Q3: Does the benefit persist with explicitly prompted chain-of-thought or higher reasoning budgets?

Yes. Using Qwen3.8-27B's native reasoning-effort controls on five challenging benchmarks from its official model card (MATH-Vision, BabyVision, RealWorldQA, ERQA, and CharXiv), SLP consistently improves accuracy at every effort level across a 3× increase in generated reasoning tokens (Table R6). Xhigh-reasoning even yields the greatest improvement. Scaling generation-time text reasoning therefore does not replace question-conditioned visual encoding during prefill, which aligns with our Finding 3. We also agree that our original phrasing was too broad, and have revised Section 3.3.3 to state that prefill replay complements reasoning across the tested budgets. Per-benchmark results are added to Appendix K.

Table R6. Qwen3.8-27B accuracy and total output tokens across reasoning efforts

Reasoning effort⟨V, T⟩ Acc. (%)SLP Acc. (%)Δ (points)⟨V, T⟩ output tokensSLP output tokens
low69.5972.17+2.5813741322
medium69.9872.15+2.1718861720
xhigh72.4675.38+2.9144054285

We hope the new evidence resolves your concerns about the evaluation and the mechanism evidence. If so, we would be grateful if you could take them into account in your assessment, and we are happy to address any further questions.

Reviewer_Jqf5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a second-look prompting strategy (SLP), which repeats the image-question pair in the input. The approach provides a training-free method by only changing the input presentation to improve the model's performance on 12 benchmarks, including reasoning and perception tasks. The results show that the SLP method improves the overall average score of open-source and proprietary models evaluated.

Critical strengths
  • The paper presents the motivation clearly.
  • The suggested approach is compared with previous studies in a quantitative way to show the improvement of the method.
  • Various benchmarks have been tested to show the performance difference.
  • The visuals are clear and understandable.
Critical weaknesses
  • The evaluated open-source models are old versions, and I have concerns that their results may not be valid for the current open-source models.
  • When we compare the gain vs. cost of the approach, the improvement of the method is not sufficient to cover the cost of the two-time input representation. As the image inputs are costly, my expectation from this method is an improvement of more than a couple of points.
Questions for authors
  • Can the authors explain why they selected layers 60-63 to analyze visual attention?
  • Although the experiments on proprietary models can be costly, the open-source models the authors evaluated, like QWEN 2.5, are older versions and may not have this input representation issue currently. I suggest the authors try some of their main experiments on new models like Qwen 3.8 and prove their approach on these new models.
  • Is there any method to reduce the cost of visual input to solve the gain vs. cost tradeoff? If the authors find one, can they run an experiment on that to show the changes in cost with this method vs. two-time image-question-pair input?
  • The authors mention that they evaluated proprietary models only on 1000 samples per benchmark. Is the same test set used for open-source models? If not, then can they explain how these different test sets provide reliable result comparison across model performances?
6 comments
AuthorsOpenReview ↗

Response to Reviewer Jqf5 (1/2)

Thank you for your constructive comments and suggestions. We have revised our paper according to your comments. We respond to your questions below and would appreciate it if you could let us know if our response addresses your concerns.


To Q1:

Q1: Can the authors explain why they selected layers 60-63 to analyze visual attention?

Visualizing the final decoder layers follows common practice in interpretability studies, since high-level semantics and multimodal representations consolidate in middle-to-late Transformer layers [1–4]. We therefore fixed the last four layers a priori for compact visualization without tuning layer indices.

However, for completeness we still analyzed all decoder layers on the same examples. The ⟨V, T, V⟩ − ⟨V, V, T⟩ answer-region shift is positive at every layer from 14 through 27 in all three examples and peaks in layers 20–23 rather than the final window (Table R1). While preparing these plots, we noticed a typo in Section 3.3.1 where Figure 3 plotted the final four layers of Qwen2.5-VL-7B instead of Qwen2.5-VL-32B; we have fixed this typo and added all-layer curves for both 7B and 32B models in Appendix D (Figures 6 and 7), which show similar patterns.

Table R1. Answer-region attention shift (⟨V, T, V⟩ − ⟨V, V, T⟩, points) on Qwen2.5-VL-7B

LayersSoccerRoot hairsFood chain
16–19+2.63+0.67+0.98
20–23+3.94+1.01+2.60
24–27+2.33+0.71+1.48

References

[1] Caron et al. Emerging Properties in Self-Supervised Vision Transformers. ICCV 2021.

[2] Jawahar et al. What Does BERT Learn about the Structure of Language?. ACL 2019.

[3] Tenney et al. BERT Rediscovers the Classical NLP Pipeline. ACL 2019.

[4] Zhang et al. Cross-modal Information Flow in Multimodal Large Language Models. CVPR 2025.


To W1 and Q2:

W1 & Q2: The evaluated open-source models are old versions ... try some main experiments on new models like Qwen 3.8.

We evaluated the requested Qwen3.8-27B alongside two additional recent models, Qwen3.5-35B-A3B and Gemma-4-E4B-it (Table R2a and R2b). To avoid ceiling effects on older tasks, we test Qwen3.8-27B across all four native reasoning modes on five challenging benchmarks from its official model card (MATH-Vision, BabyVision, RealWorldQA, ERQA, and CharXiv).

Notably, stronger models benefit even more from SLP, exceeding the earlier family-average gains on the same suite in Table 1 (+0.39 to +0.84 points) and improving Qwen3.8-27B across every reasoning mode. Model scaling does not remove the one-way constraint of ⟨V, T⟩ prefill, and stronger backbones appear even better able to exploit unified multimodal encoding once given a second look. We have added these results to Appendix K.

Table R2a. Qwen3.5-35B-A3B and Gemma-4-E4B-it (points)

ModelPromptMATH-VisionLogicVistaDynaMathAI2DOCRBenchMMBenchAvg.
Qwen3.5-35B-A3B⟨V, T⟩69.0571.5979.5084.2389.7086.0780.02
SLP70.6975.1783.4984.2391.7084.1381.57
Δ+1.64+3.58+3.990.00+2.00−1.93+1.55
Gemma-4-E4B-it⟨V, T⟩43.0938.0333.9175.0074.1070.7455.81
SLP45.6945.1940.5478.2175.4074.0759.85
Δ+2.60+7.16+6.63+3.21+1.30+3.33+4.04

Table R2b. Qwen3.8-27B across reasoning modes (points)

ModePromptMATH-VisionBabyVisionRealWorldQAERQACharXivAvg.
non-thinking⟨V, T⟩71.0923.7178.9554.7070.9059.87
SLP73.5523.9783.6657.4974.1062.55
Δ+2.47+0.26+4.71+2.79+3.20+2.68
low⟨V, T⟩81.2838.4084.3165.8578.1069.59
SLP83.1946.6584.4465.8580.7072.17
Δ+1.91+8.25+0.130.00+2.60+2.58
medium⟨V, T⟩82.2441.2483.6665.1677.6069.98
SLP82.6046.9184.7169.3477.2072.15
Δ+0.36+5.67+1.05+4.18−0.40+2.17
xhigh⟨V, T⟩84.7746.1384.9767.2579.2072.46
SLP85.6652.8486.4168.9983.0075.38
Δ+0.89+6.70+1.44+1.74+3.80+2.91
AuthorsOpenReview ↗

Response to Reviewer Jqf5 (2/2)

To W2 and Q3:

W2 & Q3: ... improvement is not sufficient to cover the cost of the two-time input representation ... Is there any method to reduce the cost of visual input?

We first clarify that duplicating visual tokens does not double inference cost, because prefill is parallel while autoregressive decoding dominates runtime (and SLP slightly shortens outputs on most models). Family-average E2E latency in Table 7 changes by at most ±0.16 s (as also noted by Reviewer 5JNq), while gains on recent models reach up to +4.04 points without training (Table R2a).

Still, following your suggestion, we propose downsampling only the replayed image V₂ to half (SLP-half) or a quarter (SLP-quarter) of its visual-token budget, while keeping V₁ at full resolution (all presentations timed side by side on one H100 node over 10% subsets; Table R3). Remarkably, coarsening V₂ monotonically reduces latency and further improves accuracy over full SLP, bringing E2E latency within 1%–3% of ⟨V, T⟩. We attribute this additional accuracy gain to the same mechanism hypothesized in our shifted-image control (Section 3.3.2 & Appendix F): breaking exact spatial correspondence with V₁ discourages direct V₂→V₁ attention in favor of attention to the question and computation within V₂. Thank you for motivating this practical improvement. Per-benchmark results are added to Appendix I.2.

Table R3. Accuracy and cost of downsampled V₂ replay

PresentationVisual tokensAcc. (%)Δ vs. ⟨V, T⟩E2E (s)E2E vs. SLP
Qwen2.5-VL-32B
⟨V, T⟩141668.89—5.04−5.4%
SLP283369.75+0.865.32—
SLP-half212469.91+1.025.29−0.7%
SLP-quarter177869.95+1.065.19−2.6%
Qwen3.8-27B (non-thinking)
⟨V, T⟩78459.87—40.43−8.0%
SLP156862.55+2.6843.96—
SLP-half118362.78+2.9141.75−5.0%
SLP-quarter99962.98+3.1140.86−7.0%

To Q4:

Q4: Is the same test set used for open-source models? If not, then can they explain how these different test sets provide reliable result comparison across model performances?

No. Open-source models use the full splits, while proprietary models share a seeded subset of up to 1,000 samples per benchmark due to API cost.

Our claims do not compare across model performances; they only require a reliable comparison between settings (⟨V, T⟩ vs. SLP) within each model, where both settings always use identical examples and decoding configurations. In addition, the 95% confidence intervals in Table 10 exclude zero for all three proprietary models, confirming reliable gains on these subsets.

We hope these results on recent models and the lower-cost variant address your concerns about validity and cost. Please let us know if anything remains unclear; we would be glad to clarify.

Dear reviewer Jqf5,

We would be grateful to know whether our rebuttal has addressed your concerns. Your insights mean a great deal to us, and we’d be glad to address any remaining issues to strengthen our work.

Thank you again for your time and effort in reviewing our paper.

Comments on the paper

2 comments

We thank the Area Chair and all reviewers for their time and constructive feedback. We are glad that the reviewers recognize our clear motivation and writing (Reviewers Jqf5, hzAc), broad evaluation across model families and task suites (Reviewers Jqf5, 5JNq, hzAc), practical ease of use and latency analysis (Reviewer 5JNq), and mechanistic insights from our attention and scratchpad studies (Reviewers 5JNq, hzAc).

Given the number of new ablations within the rebuttal window, unless noted otherwise, new experiments use six representative benchmarks (MATH-Vision, LogicVista, and DynaMath for reasoning; AI2D, OCRBench, and MMBench for perception). We have updated the revised manuscript (highlighted in blue) with the following additions:

  1. Newer frontier and reasoning VLMs (Section 3.2, Appendix K): Added Qwen3.5-35B-A3B (+1.55), Gemma-4-E4B-it (+4.04), and Qwen3.8-27B across four reasoning modes (up to +2.91 at xhigh).
  2. Low-resolution V₂ replay (Section 3.5, Appendix I.2): Downsampling V₂ to half and a quarter of its visual-token budget monotonically reduces latency while further improving accuracy over full SLP.
  3. Order, carrier, and all-layer controls (Section 3.3, Appendices C.1, D, E.4): Added 12-benchmark ⟨T, V, T⟩ accuracy and latency comparisons, normal-generation text-filler, ⟨V, V, T, T⟩, and V₂ attention-blocking controls, as well as all-layer attention curves (Qwen2.5-VL-7B and -32B) and window-wise spatial maps in Appendix D (also fixing a typo in Section 3.3.1 where Figure 3 plotted the final four layers of Qwen2.5-VL-7B).

Dear reviewers,

We have updated our anonymized code implementation at https://anonymous.4open.science/r/slp-iclr-anonymous/.

Thanks again for your time reviewing our paper, and we look forward to further discussions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.