- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Cluster Intervention Gain (CIG), a training-free visual token selector for vision-language models. It obtains a reference answer distribution from the full model, clusters visual tokens, and evaluates candidate subsets through additional prefills. A greedy procedure selects subsets that minimize KL divergence from the reference while satisfying a token budget. Importantly, the implemented objective concerns a normalized first-answer-token distribution over a limited support, rather than complete generated answers. Evaluation includes four backbones under a fixed 576-token protocol, LLaVA comparisons, held-out checks, short free-form generation, ablations, and latency measurements. The strongest finding is improved accuracy under aggressive compression; efficiency and high-resolution transfer are more limited.
Critical strengths
- Accuracy gains at very small token budgets. Table 1 reports positive paired intervals against the strongest eligible heuristic for every backbone at . At , the mean advantage is 11.57 percentage points. The partial held-out checks and short free-form generation results provide additional support.
Critical weaknesses
1. Compression requires executing the full-reference prefill first, and the practical benefit remains inadequately established. Algorithm 1 requires a full-reference prefill, a blind prefill, and multiple candidate prefills. For the short-answer tasks dominating the accuracy evaluation, the reference already supplies the next-token information used to answer the question. Reducing the final KV cache therefore does not establish reduced work for the complete request. The paper's own estimator gives about 14.1 TFLOPs for CIG at , versus about 8.1 TFLOPs for full, as shown in Appendix O. End-to-end latency is 0.6519s versus 0.1888s for Full and 0.1599s for CDpruner, as shown in Table 30, which is approximately 3.45 and 4.08 times slower. The 128-token caption workload is also approximately 24% slower than Full in Table 31. I am skeptical that these results demonstrate an optimal trade-off between performance and efficiency. The serving experiments leave important questions unresolved. They use eager attention and do not remeasure quality on the timed workloads. Table 28 reports identical selection costs at batch sizes 4 and 16 without explaining whether independent selection for every request is included. Furthermore, Section 5.4 and Table 5 categorically state that Qwen3-VL-4B shows no crossover and batch 4 does not cross, although Table 28 contains positive speedups in both regimes. These accounting and interpretation issues directly affect the efficiency claim. Despite detailed empirical reporting, I am not convinced that the paper meets the standard required for acceptance at ICLR.
2. The objective preserves a resticted readout, while important experiments substantially narrow the claimed advantage. Matching a normalized first-token distribution cannot constrain probability mass outside its support or preserve later generated tokens. The unspecified construction of the open-ended support further prevents independent reproduction. Short-answer generation checks are useful, but do not establish fidelity for the long-generation workloads used to motivate the probing cost. The unfavorable compressions are consequential. On LLaVA-1.5, the seven-task advantage over CDPruner is only 0.163 precentage points, with a 95% interval of [-0.305, 0.636]. At high resolution, its macro is 62.59 versus 65.68 for CDPruner, and TextVQA is 40.36 versus 50.13 in Table 25. On held-out TextVQA at , MMTok exceeds CIG on every backbone, as shown in Table 17. The oracle interpretation is additionally unsupported. Append Q infers that at most approximately one percentage point of the oracle gap comes from candidate-space and search limitations. Achieved scores under different objectives and candidate spaces cannot establish this bound: changing the objective changes the greedy trajectory, and no exact optimum or search guarantee is provided. To substantiate the claimed attribution of the oracle gap, the authors should provide quantitative comparisons of the greedy search trajectories and final scores under different objectives, holding the candidate space and search budget fixed.
Questions for authors
1. Does Table 28 include independent subset selection for every request in each batch? Why is selection cost identical at batch sizes 4 and 16? Please specify the execution and accounting, including reuse of the reference prefill's output or KV cache.
2. What is the exact open-ended answer-support construction, including its size, treatment of multi-token forms, and whether it remains fixed across candidate subsets? What is used in , and how is defined for the highest-density point?
3. Is there existing evidence that the serving advantage persists with optimized attention and preserves quality on the actual timed long-generation workloads? Appendix N uses eager attention and borrows accuracy from the short-answer grid.
4. Please reconcile the crossover statements with Table 28. Does Appendix Q's approximately one-percentage-point search-gap claim rely on exact optimization over a common candidate space? If it uses heuristic achieved scores, what justifies treating it as a bound?