- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes VeGo, an extension of crop-based on-policy self-distillation. The teacher receives clean and degraded crops separately, and the divergence between its prediction distributions under the two views is used to weight teacher–student distillation. A second, answer-probe-based divergence identifies a decision cliff, after which the generated suffix is penalized to reduce redundant post-decision generation while maintaining accuracy. The method is trained on 6.5K examples using Qwen3.5-9B and evaluated on five high-resolution or fine-grained perception benchmarks. Compared with Vision-OPD, average accuracy improves from 75.77% to 77.35%, while average output length decreases from 116.74 to 86.39 tokens. Ablations compare weighting signals, normalization methods, suffix masks, and adaptive reference lengths.
Critical strengths
- The reported accuracy–length trade-off on a newer open-source model, Qwen3.5-9B, is attractive. Detailed component analyses also demonstrate the method's effectiveness. Across five benchmarks, the method improves accuracy and reduces generation length relative to Vision-OPD.
- The method has a sufficiently clear and important motivation. Diagnostic experiments in Figures 1 and 2 show that token-level dependence on visual evidence is highly concentrated and that residual evidence dependence drops sharply to near 0 once the answer is determined. These findings support the subsequent method design.
Critical weaknesses
- Fine-grained visual reasoning generally involves relatively short outputs. For example, the longest output reported in Table 2 does not exceed 400 tokens and may not incur excessive inference latency. Efficiency improvements may therefore not be among the more important issues in fine-grained visual reasoning. VeGo saves only a few dozen tokens, which may not translate into a significant inference speedup in practice.
- Many details need further attention, with multiple omissions and errors throughout the paper. For example:
- The value of is inconsistent: Section 4.1 specifies , while Appendix A states “default .”
- In Table 4, the Len direction for the HR-Bench 8K and Average columns is marked () but should be ().
- The question and answer options in Figure 8 do not match: “What black logo is visible on the yellow jersey? (A) left (B) right.”
- The Table 3 caption contains the problematic wording “Bold values indicate the best and best results.”
- Several leftover notes remain in the references.
- The main text cites a nonexistent “Figure 2 (c).”
- The sensitivity analysis cites the wrong figure.
- Numerous extraneous characters and typos remain.
Questions for authors
- How exactly is the decision-cliff value defined? The paper gives only a rough description, without a specific formula or value.
- The introduction states that the method will “apply an accuracy-gated length objective.” Where is the accuracy gate reflected in the method?
- The Figure 3 caption explicitly states that the suffix after the cliff “gets no distillation gradient,” yet Eq. 4 sums over the entire sequence. How do the authors explain this inconsistency?
2 comments
Thank you to the authors for their detailed response. The authors' response has addressed my concerns regarding the clarity of the paper, but some concerns remain. In particular, the authors have not addressed Q2. Furthermore, although reducing output length may improve inference speed, it may also lead to performance degradation on more complex reasoning tasks, as these tasks typically require longer chains of thought. Therefore, I have decided to maintain my current rating.
We thank the reviewer for the constructive feedback.
W1: Resource savings from shorter outputs
In high-resolution perception, every output token repeatedly attends to a large visual context. Visual Probe and HR-Bench 4K/8K contain on average 13.8K–16.2K visual tokens per image. On HR-Bench 8K, one decoding step reads approximately 509 MB of KV cache, about 16× the volume for a 1K-token text prompt. Shortening post-decision generation therefore reduces repeated computation and KV-cache traffic.
Measurement. We count decoding FLOPs with Qwen3.5-9B on 50 randomly sampled questions per benchmark, reproducing the evaluation prompts and image preprocessing. The counted per-step cost is linear in context length: 15.87 GFLOP + 131,072 FLOP per context token. We apply this calibrated cost model to the saved output lengths of all questions and estimate KV-cache reads from the cache geometry. As in Table 2, outputs of ≥4096 tokens are excluded.
Decoding cost per question, Vision-OPD → VeGo. Avg. denotes the mean over these three benchmarks; reductions are computed from unrounded values.
VeGo reduces decoding FLOPs by 32.2% and KV-cache reads by 33.1%, saving approximately 21 GB of KV-cache traffic per question on these high-resolution benchmarks. This reduces bandwidth demand during decoding; shorter continuations also allow each request's cache to be released earlier, which is relevant to batched serving.
The efficiency gain accompanies higher five-benchmark accuracy (75.77% → 77.35%; Table 1). Short outputs are not inherent to these tasks: on the three benchmarks above, DeepEyes and SenseNova-MARS average 185 and 300 tokens, respectively, versus VeGo's 96.9 (Table 2). VeGo targets redundant post-decision text while retaining the evidence-writing prefix.
W2: Manuscript consistency
We apologize for the typographical and editorial errors and the confusion they caused. We will carefully proofread the entire manuscript and correct these issues in the revised version, including the captions, cross-references, and bibliography.
The actual coefficient is λ = 0.5; Appendix A's “default λ = 1.0” is a typo. The correct Len direction in Table 4 is ↓. Figure 8's question should read “Is the volleyball on the left or right side of the man with white cap?” Table 3's caption should read “Bold values indicate the best results.” The redundant-tail distribution is Figure 2(b), not Figure 2(c); the sensitivity analysis is Figure 6, not Figure 5. Reference notes and stray characters should be removed.
Q1: Definition of the decision cliff
Following Section 3.3, a fixed decision probe φ (e.g.,
The correct answer is: **) is appended to each generated prefix. Let P⁺ₜ and P⁻ₜ denote the EMA teacher's answer distributions given the question and this probed prefix, under the clean and degraded crops, respectively. Residual evidence dependence is:Iₜ = JSD₀.₅(P⁺ₜ ‖ P⁻ₜ).
The decision cliff
t_cliffis the position of steepest sustained descent of the Iₜ profile, after which Iₜ remains near zero. It is a trajectory-specific token index, not a universal divergence value. To preserve the complete evidence-writing sentence, compression starts after the sentence containing the cliff ends:Eᵢ = {t : t > sent-end(t_cliff) in yᵢ}.
Q3: Distillation on the post-cliff suffix
Eq. 4 describes evidence-weighted distillation over the full response, ∑ₜ gₜ dₜ; it does not impose a hard suffix mask. Figure 3's “no distillation gradient” should instead read: “Evidence dependence weights distillation; the length objective acts only on the post-cliff suffix.” The two signals differ: gₜ concerns the next generated token, whereas Iₜ probes the answer decision. Thus, Iₜ ≈ 0 does not imply gₜ = 0. Suffix tokens can still receive weighted distillation gradients; tokens outside Eᵢ have no direct length-loss term.