- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DUETTO, a learnable visual token selection method that combines visual information with the relevance to text instructions. Its visual branch models relationships between visual tokens using trainable self-attention, while the text branch estimates task relevance based on the cosine similarity between visual and text embeddings. The scores from both branches are fused and optimized via a differentiable Top-K mechanism and a language modeling loss. The authors conducted experiments across multiple VLM architectures and tasks involving both image and video understanding, achieving competitive performance even at high compression rates.
Critical strengths
-
Coherent framework with measurable inference benefits. The framework integrates learned visual scoring and instruction relevance into a single early selection step with an exact token budget, providing a unified mechanism for incorporating both signals. Its practical benefit is supported by measured runtime savings: on LLaVA-NeXT, DUETTO achieves approximately 3.2× prefill and 1.3× end-to-end speedups over the uncompressed model, with 4.5 ms selection overhead. This demonstrates actual inference acceleration rather than only theoretical reductions in token count or computation.
-
Strong performance under aggressive compression. DUETTO retains 97.2% of full-model performance with 64 visual tokens on LLaVA-1.5 and 96.5% with 160 tokens on LLaVA-NeXT, achieving the highest average relative accuracy among the evaluated baselines at these budgets. These results provide meaningful evidence that the method preserves downstream performance even when most visual tokens are removed, supporting its value under restrictive token budgets.
Critical weaknesses
-
Insufficient validation of dual-channel coupling. The vision-only ablation does not distinguish the benefits of training-time coupling from those of simply adding instruction-relevance scores at inference. A critical missing comparison is an independently trained vision-only selector combined with the same relevance scores only during inference.
-
Gap between training and inference. For discarded tokens, the selector receives no direct feedback on whether keeping them would improve the answer. Moreover, retained token features are scaled down during training but used at their original scale during inference. These are different inputs to the subsequent model layers. Appendix A.1.3 reports that 37.7% of retained tokens still have scaling weights below 0.9 after training, so the difference does not disappear through training.
-
Limited modeling of complex instruction relevance. The text channel uses input-embedding similarity with order-invariant pooling, without explicitly modeling contextual relationships, negation, or attribute binding. This raises concerns about selecting evidence for relational queries, similar instances, and object absence. The reported grounding failures and weaker POPE results reinforce these concerns, but targeted analyses are missing to clarify the design’s applicability.
-
Incomplete accuracy–efficiency comparison. At 160 tokens on LLaVA-NeXT, DUETTO achieves higher accuracy than VisionZip‡ but requires 2.06 versus 1.07 TFLOPs and incurs higher prefill and end-to-end latency. Equal retained-token budgets therefore do not imply equivalent computational costs. Comparisons at matched FLOPs or latency are needed to establish whether its accuracy advantage persists under comparable resource constraints.
-
Insufficient differentiation from and comparison with prior work. LearnPruner and DUET-VLM already combine visual redundancy reduction with text-guided pruning; DUET-VLM specifically uses vision-side merging followed by progressive attention-based pruning. However, direct comparisons with these frameworks are absent, leaving the substantive methodological and practical advantages of this formulation insufficiently established.
Questions for authors
-
Dual-channel coupling. Does DUETTO outperform an independently trained vision-only selector combined with the same instruction-relevance scores only at inference?
-
Complex instruction relevance. Does the instruction-relevance channel improve over the vision-only variant on questions involving spatial relationships, negation, attribute binding, and similar objects?
-
Distinction from prior work. What substantive methodological advances distinguish DUETTO from LearnPruner and DUET-VLM beyond the number and placement of selection stages?
Other questions please to weakness.
Discussion period
Dear reviewers,
We are checking in to confirm that our previous response resolved your concerns. We are keen to continue the review process during the remaining days of discussion.
Thank you very much for your time!