- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The authors propose SCOPD, a post-training framework aimed at recovering the performance of LVLMs that have undergone aggressive visual token pruning. The work is predicated on the observation of a "representation-utilization gap"—the finding that a pruned visual context still yields correct answers under high-budget stochastic sampling (Pass@K), implying that the language model simply fails to reliably extract surviving visual evidence during greedy decoding. To address this, SCOPD employs a standard on-policy self-distillation approach, where a student model conditioned on sparse visual context is trained to match the token-level predictive distribution of a frozen, full-context teacher. A selective variant, SCOPD+, applies distillation only to a subset of tokens deemed "visually sensitive," which are identified by measuring the Jensen-Shannon Divergence (JSD) of the student’s output when the visual token budget is marginally increased.
Critical strengths
- It is important to distinguish “information absent from the retained tokens” from “information present but unreliably used by the decoder.”
- The authors evaluate their method across a wide array of 13 image benchmarks and 5 video benchmarks, testing generalization across different base models (Qwen2.5-VL and Qwen3-VL) and various pruning strategies (VisionZip, DivPrune, FastV).
Critical weaknesses
- The core “representation–utilization gap” claim is not causally established. The paper’s main claim is that sparse visual representations often retain sufficient task-relevant evidence, but the model fails to use this evidence reliably. However, the fixed-context Pass@K experiment does not rigorously establish this causal interpretation. The main evidence compares the setting with 10% retained visual tokens against a language-only baseline. The language-only control is potentially confounded. For image-dependent questions, VLM without image input may frequently produce responses such as “I cannot see the image,” refuse to guess, fail to follow the requested answer format, or otherwise behave conservatively. In that case, the near-zero language-only Pass@K does not necessarily show that language priors are insufficient.
- The comparison between “correct-image 10% tokens” and “no image tokens” is not the right causal control. It changes both the availability of relevant visual evidence and the model behavior. A more appropriate control should preserve the same number of visual tokens, while changing only whether the visual evidence is relevant. For example: ''10% task-relevant tokens'' with ''10% strictly task-irrelevant tokens''. Without proving that the recovery relies only on relevant tokens, this framework is likely just forcing the model to confidently hallucinate visual evidence when it actually cannot see the image clearly.
Questions for authors
-
In SCOPD+, the "visual sensitivity" criterion is measured using the JSD between predictions under budgets b and b+δ. However, this metric essentially measures the model's output instability or sensitivity to additional tokenspruning results. It does not necessarily prove that the model is sensitive to task-relevant visual evidence. Could you provide deeper theoretical justification or empirical validation?
-
Why does SFT underperform Vanilla at the full 100% visual-token budget, while improving over Vanilla at 20% and 10% token retention?
-
The paper appears to be an extension of a NeurIPS 2026 workshop paper. Could the authors clearly describe the relationship between the two versions and enumerate the new contributions in the current submission?
5 comments
Q2: Why is SFT below Vanilla at full context but above it at 20% and 10%?
SFT is trained exclusively at 10% visual-token retention using reference reasoning traces from LLaVA-CoT. One plausible explanation for the observed trade-off is that fine-tuning adapts the model to sparse inputs and fixed reference traces, which can also change its performance when evaluated with full visual context.
Importantly, the degradation at full context is relatively small: SFT achieves 99.17 Avg13 compared with 100.00 for Vanilla, with the largest drops on MathVerse and LogicVista (6.98 and 4.69 points). At lower retention budgets, the benefit of sparse-context adaptation outweighs this degradation: SFT improves over Vanilla from 93.42 to 95.22 at 20% retention and from 86.37 to 88.82 at 10%.
SCOPD is also trained at 10% retention but retains its full-context performance (100.67 Avg13). Unlike SFT, SCOPD uses on-policy trajectories and supervision from an EMA teacher with access to the full visual context, rather than fixed external reasoning traces. This difference in supervision may help preserve the model's original reasoning behavior while adapting it
Dear Reviewer,
We understand how busy this period is, and sincerely appreciate the time you have devoted to reviewing our paper.
With less than three days remaining in the discussion period, we wanted to briefly follow up on our rebuttal and revised manuscript. In particular, we have added the matched-budget visual controls you suggested to strengthen the evidence for our representation–utilization hypothesis, along with additional experiments and clarifications addressing your other questions.
We would be grateful if you could let us know whether these additions address your concerns, or if there is anything further we could clarify before the discussion period ends.
Thank you again for your time and thoughtful feedback!
We thank the reviewer for the detailed review and concrete suggestions. We agree that the key question is whether task-relevant information is missing from the retained visual tokens or remains available but is used unreliably. To investigate this distinction more carefully, we have added the suggested controls and further analyses.
W1 & W2: Is the representation–utilization gap causal, and does recovery depend on task-relevant evidence?
Language-only control. We examined all 32,000 language-only responses generated from 500 unique questions, with 64 responses per question, using the GPT-6-Astra judge described in Appendix A. In 25% of responses the model mentions that there is no visual information available but still guesses an answer and in 20% of responses it refuses to answer. These response-level rates indicate that refusals could contribute to the low language-only score. We therefore added the matched-budget controls below to assess the dependence of recovery on image-specific input. Our Pass@K evaluation requires not only a correct final answer but also logically consistent reasoning supported by the original image (Sec. 3.1, Appendix A). Figure 5 provides examples of accepted and rejected reasoning trajectories.
Controls with the same visual-token budget. Following the reviewer's suggestion, we repeated the fixed-context analysis on the same 500 questions using three additional conditions, each retaining 10% visual tokens: (i) tokens from an unrelated image, (ii) the least important tokens from the correct image according to VisionZip, and (iii) randomly selected tokens from the correct image. Random and least-important token selection serve as additional controls rather than strictly irrelevant visual contexts. Random tokens may still preserve useful evidence, while the least-important tokens are determined by VisionZip's importance ranking, which does not necessarily imply that they contain no task-relevant information.
The results provide stronger evidence for our interpretation. Replacing the retained tokens with those from an unrelated image reduces Pass@64 from 79.6% to 11.4%, despite preserving the same visual-token count. In contrast, random tokens from the correct image recover 68.8%, and even the least important tokens recover 60.6%. These results suggest that recovery depends strongly on image-specific visual evidence, rather than simply on the presence of visual tokens or a greater tendency to generate answers. While this does not establish which individual tokens contain the required evidence, it strengthens the representation–utilization interpretation. We have added these controls to Fig. 1b, and Sec 3.1.
Regarding the concern that adaptation may encourage confident hallucinations, we also observe improvements on HallusionBench at both 10% retention (58.57 for Vanilla, 60.46 for SCOPD, and 60.99 for SCOPD+) and 20% retention (61.51, 63.83, and 65.72, respectively). While not a direct test of hallucination frequency, these results provide additional supporting evidence that the gains are not limited to answer-generation behavior.
Q1: Does the sensitivity score reflect task-relevant evidence?
We agree that our visual sensitivity metric measures how much the model's predictions change under a small increase in the retained visual context, rather than directly identifying task-relevant visual evidence. However, we would like to clarify that our goal is not to identify which positions correspond to task-relevant evidence. Instead, as hypothesized at the end of Sec. 3.2, our motivation is that positions sensitive to changes in the visual context may be particularly useful for applying full-context teacher supervision, helping the student better utilize its sparse visual representation.
We validate this hypothesis empirically. Table 2 compares our sensitivity-based selection against several alternatives, and Appendix H extends these comparisons to all 13 benchmarks and additional baselines. SCOPD+ achieves 92.43 Avg13, compared with 90.48 for random selection, 91.29 for Top-KL, and 86.35 for selecting the least sensitive positions. Moreover, Masked SCOPD, which excludes the top 10% most sensitive positions and distills the remaining 90%, achieves only 89.46, even below dense SCOPD (90.49).
Together, these experiments support our hypothesis that visual-budget sensitivity is a useful criterion for selecting where to apply distillation, without requiring the stronger claim that the metric explicitly identifies task-relevant visual evidence.