acceptodds
Withdrawn from ICLR 2027

DUETTO: Dual-Channel Efficient Token Selection for Large Vision-Language Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Vision-Language Models introduce significant computational challenges, as encoding images and videos into thousands of vision tokens imposes substantial computational costs. Prior work addresses this challenge by proposing methods to identify and select the most informative vision tokens while preserving model performance. Determining which vision tokens are important requires considering two signals: the information a token contributes relative to the rest of the vision token sequence and its relevance to the instruction. Yet, existing methods fail to jointly optimize both for the downstream task: heuristic approaches rely on fixed ranking rules that cannot adapt to the downstream effects of token selection, while learnable approaches either base selection solely on visual content or defer instruction relevance to a later stage. To address this, we propose DUETTO (Dual-channel Efficient Token selection), a vision token selector trained end-to-end that fuses a vision channel and an instruction-relevance channel into a single selection score. DUETTO achieves state-of-the-art performance, with the advantage over prior methods increasing as the token budget becomes more restrictive, maintaining 99.1% of full-model accuracy on 33.3% of the vision tokens and, at high resolution, 96.5% using 5.6% of the vision sequence, accelerating prefill by 3.2x while the selection costs only 6% of prefill time. Finally, experimental results show that DUETTO generalizes to diverse backbones with both image and video inputs, achieving state-of-the-art results.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision Token Compression, Efficient Large Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Trb2OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DUETTO, a learnable visual token selection method that combines visual information with the relevance to text instructions. Its visual branch models relationships between visual tokens using trainable self-attention, while the text branch estimates task relevance based on the cosine similarity between visual and text embeddings. The scores from both branches are fused and optimized via a differentiable Top-K mechanism and a language modeling loss. The authors conducted experiments across multiple VLM architectures and tasks involving both image and video understanding, achieving competitive performance even at high compression rates.

Critical strengths
  • Coherent framework with measurable inference benefits. The framework integrates learned visual scoring and instruction relevance into a single early selection step with an exact token budget, providing a unified mechanism for incorporating both signals. Its practical benefit is supported by measured runtime savings: on LLaVA-NeXT, DUETTO achieves approximately 3.2× prefill and 1.3× end-to-end speedups over the uncompressed model, with 4.5 ms selection overhead. This demonstrates actual inference acceleration rather than only theoretical reductions in token count or computation.

  • Strong performance under aggressive compression. DUETTO retains 97.2% of full-model performance with 64 visual tokens on LLaVA-1.5 and 96.5% with 160 tokens on LLaVA-NeXT, achieving the highest average relative accuracy among the evaluated baselines at these budgets. These results provide meaningful evidence that the method preserves downstream performance even when most visual tokens are removed, supporting its value under restrictive token budgets.

Critical weaknesses
  • Insufficient validation of dual-channel coupling. The vision-only ablation does not distinguish the benefits of training-time coupling from those of simply adding instruction-relevance scores at inference. A critical missing comparison is an independently trained vision-only selector combined with the same relevance scores only during inference.

  • Gap between training and inference. For discarded tokens, the selector receives no direct feedback on whether keeping them would improve the answer. Moreover, retained token features are scaled down during training but used at their original scale during inference. These are different inputs to the subsequent model layers. Appendix A.1.3 reports that 37.7% of retained tokens still have scaling weights below 0.9 after training, so the difference does not disappear through training.

  • Limited modeling of complex instruction relevance. The text channel uses input-embedding similarity with order-invariant pooling, without explicitly modeling contextual relationships, negation, or attribute binding. This raises concerns about selecting evidence for relational queries, similar instances, and object absence. The reported grounding failures and weaker POPE results reinforce these concerns, but targeted analyses are missing to clarify the design’s applicability.

  • Incomplete accuracy–efficiency comparison. At 160 tokens on LLaVA-NeXT, DUETTO achieves higher accuracy than VisionZip‡ but requires 2.06 versus 1.07 TFLOPs and incurs higher prefill and end-to-end latency. Equal retained-token budgets therefore do not imply equivalent computational costs. Comparisons at matched FLOPs or latency are needed to establish whether its accuracy advantage persists under comparable resource constraints.

  • Insufficient differentiation from and comparison with prior work. LearnPruner and DUET-VLM already combine visual redundancy reduction with text-guided pruning; DUET-VLM specifically uses vision-side merging followed by progressive attention-based pruning. However, direct comparisons with these frameworks are absent, leaving the substantive methodological and practical advantages of this formulation insufficiently established.

Questions for authors
  1. Dual-channel coupling. Does DUETTO outperform an independently trained vision-only selector combined with the same instruction-relevance scores only at inference?

  2. Complex instruction relevance. Does the instruction-relevance channel improve over the vision-only variant on questions involving spatial relationships, negation, attribute binding, and similar objects?

  3. Distinction from prior work. What substantive methodological advances distinguish DUETTO from LearnPruner and DUET-VLM beyond the number and placement of selection stages?

Other questions please to weakness.

Reviewer_iFoROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes DUETTO, a learnable visual-token pruning method for large vision-language models. It combines two complementary signals: a learned vision channel that models token redundancy and a parameter-free instruction-relevance channel that measures alignment between visual and text tokens. The fused scores drive token selection at an early decoder layer, allowing subsequent layers to process a shorter sequence. Experiments across six backbones show strong accuracy–efficiency trade-offs.

Critical strengths

S1 - well-motivated design and useful ablations: : The paper identifies that visual-token pruning depends on both information redundancy and instruction relevance. It's more principled than relying on visual saliency or text similarity alone. In ablations, removing either the instruction-relevance channel or the learned vision channel degrades performance, supporting the complementary roles of the two signals.

S2 - effective and efficient integration: DUETTO performs selection at an early layer, balancing pruning quality against selector overhead. The reported speedup and selection cost suggest practical inference benefits, not just reduced token counts.

S3 - strong experimental coverage: Evaluation spans multiple backbones, image and video benchmarks, and token budgets. The reported advantage becomes more pronounced under aggressive compression, an important regime for high-resolution multimodal inference.

Critical weaknesses

W1 - limited methodological novelty: Combining visual redundancy modeling with instruction-conditioned token selection is intuitive, and related methods already explore both signals. The paper should distinguish its contribution from prior learnable selectors and multi-criteria pruning methods.

W2 - fixed fusion may be suboptimal: The two channels are combined with a fixed weight across inputs and backbones, despite the relative importance of instruction relevance varying substantially between targeted questions and generic image understanding. An adaptive fusion or token budget could improve robustness.

W3 - potential generalization limitations: The selector is trained using a subset of LLaVA-1.5 instruction-tuning data and applied across different backbones and tasks. More analysis is needed to establish how performance changes under instruction-distribution shifts, domain-specific tasks, and substantially different visual encoders or language models.

W4 - efficiency claims need broader context: The reported prefill speedup is promising, but practical end-to-end gains depend on image resolution, sequence length, decoding cost, and selector overhead. The benefits may be smaller for workloads where generation rather than prefill dominates latency.

Questions for authors

Q1 - how does DUETTO compare with the closest prior methods that jointly exploit visual diversity and instruction relevance, under strictly matched token budgets and inference settings?

Q2 - could the fusion weight or retained-token budget be adapted per input, particularly for multi-object queries, and tasks requiring fine-grained spatial details?

Q3 - how robust is the selector when transferred to backbones with substantially different vision encoders, connectors, or language-model representations, without retraining?

Q4 - can the authors provide end-to-end latency and memory measurements across different image resolutions and generation lengths, in addition to prefill speedup?

Reviewer_BgyEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes DUETTO, a learnable visual token selection method that combines self-attention-based visual scoring with instruction relevance to reduce visual tokens while largely preserving the performance of vision-language models.

Critical strengths
  1. The paper is easy to follow. The motivation and methodology are clearly presented.

  2. The paper tries to address an important problem. Reducing visual token redundancy is crucial for improving the inference efficiency of large vision-language models.

Critical weaknesses
  1. The proposed method mainly combines standard self-attention, cosine-similarity-based instruction relevance, and score fusion (through a simple alpha fusion). These are well-established techniques, and related ideas have already been explored in works such as CDPruner, PruneSID, and VisionSelector. The technical novelty seems rather limited.

  2. The authors claim that the vision channel captures token redundancy. However, it is essentially a self-attention layer followed by a linear scorer, with no explicit mechanism encouraging diversity or reducing redundancy. The ablation in Table 6 only shows that self-attention helps, not that it actually models redundancy (the paper does not prove it). The improvement could simply come from better feature representations.

  3. It is unclear whether including instruction relevance during training actually improves the learned selector. The authors should compare their joint training approach against a baseline where the vision channel is trained alone, and the instruction-relevance scores are added only at inference time. Both settings would use the same architecture and score fusion at inference, isolating the actual benefit of instruction-aware training.

  4. The vision channel contains 84M trainable parameters, which is not negligible for a token selector. Could the gains be partly due to the additional model capacity? A parameter-matched comparison would help clarify this.

  • CDPruner. Maximizing Conditional Diversity for Token Pruning in MLLMs

  • PruneSID. Vision Token Compression in VLMs via Synergistic Importance-Diversity

  • VisionSelector. VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs

Questions for authors

The authors should address the points listed in the weakness section.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Discussion period

Dear reviewers,

We are checking in to confirm that our previous response resolved your concerns. We are keen to continue the review process during the remaining days of discussion.

Thank you very much for your time!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.