- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Dynamic Token Merging (DTM), a method for dynamically compressing visual tokens in vision-language models by learning which tokens to retain and which to discard. The authors adapt DTM to both encoder–decoder and decoder-only architectures and introduce a PID-based controller to improve the accuracy of the token deletion rate. Experiments across ten benchmarks show that DTM achieves competitive performance compared with existing token reduction baselines, particularly at high deletion rates, while substantially reducing language-model FLOPs.
Critical strengths
-
The writing is clear and easy to follow.
-
The method is deployment-friendly.
Critical weaknesses
-
The novelty of this paper is limited. Im my view, this is more like an engineering application report. The authors adapt an existing algorithm to a new setting without making contributions to the algorithm itself. While this constitute a meaningful engineering contribution, it is insufficient for a research paper. The authors conduct many experiments, but offer little new scientific insight since MrT5 is already an effective algorithm. More important questions are why it works in vision settings and what underlying mechanisms account for its effectiveness. But these experiments primarily demonstrate that the method can achieve good performance across benchmarks.
-
The authors mentioned many related works in the related work section, but the experiments only compare with few relatively naive baselines. Is this a fair comparison? Does DTM still demonstrate advantages over existing advanced methods for token compression?
-
Could you provide ablation studies for components, like soft-deletion training, the learnable gate, and etc, to better quantify the contribution of each component to vision token compression.
Questions for authors
See weaknesses.
2 comments
Response to Lu2h (#2)
Following up on our last response, we present additional results and analysis in our “Overall Response to Reviewers (#2)” comment: cross-task results on a third architecture, Qwen2.5-Omni-3B, including video and multi-image document tasks (under “Qwen2.5-Omni Cross-Task Results and Long-Context”); an analysis of the tokens each method keeps (under “Insights into Token Pruning Mechanisms”); and DTM applied to multimodal embeddings (under “Extending DTM to Multimodal Embeddings”). We have also uploaded a revised paper.
Weakness 1. You asked why DTM works in vision settings and what mechanisms account for its effectiveness. As promised in our first response, the section of that comment under “Insights into Token Pruning Mechanisms” (Appendix F of the revised paper) analyzes the keep decisions and per-token scores of DTM, LIS-DTM, VisionSelector, FastV, and φ-R at 80% deletion:
- DTM and LIS-DTM keep tokens where the answers are. On Gemma 3 4B and Qwen2.5-Omni-3B, their keep maps correlate with where the answers are at Pearson r = 0.61–0.74, against r ≤ 0.23 for FastV and φ-R (Figure 16).
- Their gates make near-binary decisions. 69–77% of DTM’s gate values (95–96% for LIS-DTM) fall below 0.05 or above 0.95, so the switch from soft deletion in training to hard deletion at inference leaves few tokens in doubt (Figure 17).
- DTM’s gains come from which tokens it keeps. When T5Gemma 2 1B–1B DocVQA models are re-evaluated with another method’s keep sets, DTM’s sets raise FastV’s and φ-R’s models by 11.5 and 12.9 points, and theirs lower DTM’s by about 20 points; on Gemma 3 4B RefCOCO+, DTM’s lead over FastV and φ-R grows from 1.4–1.5 points on central referents to 7.3–8.2 on small ones (Figures 18 and 19).
Weakness 2. Beyond the Average Pooling, FEATHER, and VisionSelector baselines from our first response, we now compare all methods on Qwen2.5-Omni-3B and on multimodal embeddings. While DTM and LIS-DTM do not beat VisionSelector, our strongest baseline, everywhere, they beat it more often than they trail it:
- At 60% and 80% deletion: LIS-DTM leads in 116 of 142 benchmark comparisons and DTM in 89 of 142, and on the 14 multimodal embedding sets, LIS-DTM leads in 24 of 28 and DTM in 17 of 28.
- Where our methods lead, the average improvement over VisionSelector is 2.4 points for LIS-DTM and 2.2 for DTM.
- Where they trail, the average deficit is 1.1 points for LIS-DTM and 1.5 for DTM.
- Parameters: DTM’s gate has 2.3k–5.1k parameters, about 580–1,280× fewer than VisionSelector’s scorer (1.3M–6.6M), and LIS-DTM, which adds the same parameters as VisionSelector, leads it most often.
As for novelty, we have extended DTM from MrT5’s byte-level text to visual tokens, video, multi-image documents, and multimodal embeddings, and shown that its training recipe improves another method’s learned scorer (LIS-DTM). To our knowledge, we are the first paper to apply the DTM framework for vision-language tasks with different types of vision encoders, to the decoder-only architecture, and to embedding-related tasks. These adaptations required significant implementation work, e.g., for decoder-only LLMs and for contrastive embedding training. In addition, we employ unique analyses such as the retention of task-relevant spatial evidence (Section 6.3), keep maps compared with where the answers are, the distributions of gate scores, accuracy by referent size and location, and swapping keep sets between trained models, and directly comparing our method’s mechanism of training a learned token selector with that of another (i.e., LIS-DTM vs. VisionSelector). These analyses explain why the learned gate selects better tokens, which vision token pruning papers typically do not examine. Finally, we believe “DTM as a general framework” is sufficient for novelty because our successful applications were not immediately obvious: it was not clear that a gate designed for byte-level text would transfer across vision encoders, architectures, and tasks, or that its training recipe would improve another method’s scorer.
While our paper is not necessarily novel in its mechanism (the authors of MrT5 introduce DTM), we believe our work’s contributions lies in showing whether, when, and why an established mechanism works in new modalities, tasks, deletion gates, and architectures, which our experiments and analyses do for DTM in vision-language models.
We hope you’ll consider raising your score, and we sincerely thank you again for your time to review our work.
Response to Lu2h (#1)
We appreciate the points raised in your review and your suggestions for improving the paper. We’ve addressed shared concerns across reviewers in our overall comment, “Overall Response to Reviewers (#1).” We take your critiques seriously and respond to each below:
Weakness 1
We understand your concern that our work adapts an existing algorithm to a new setting. While DTM’s core mechanism comes from MrT5, applying it to VLMs requires changes to the algorithm itself (Section 4). We extended DTM to decoder-only models, which deletes visual tokens during prompt prefill, so that later layers compute and cache only the compacted prompt, and the gate must score visual states that cannot see the question due to the nature of causal attention. We add a derivative term to MrT5’s controller (Appendix A.3), shift the gate biases seen by answer queries, and calibrate the initial deletion rate (Appendix A.2). Moreover, we show that a single checkpoint trained at one deletion rate remains effective across inference budgets (Section 6.2) and provide insights into the effectiveness and mechanics of DTM, as described below. In the “Overall Response to Reviewers (#1)” comment under “Additional Baselines” and “Cross-Task Performance,” we have further shown that the DTM framework is not tied to its own gate: applied to the learned scorer of VisionSelector, the resulting LIS-DTM performs on par with or better than VisionSelector on nearly all benchmarks. We have added results for a single DTM model trained across tasks, and we plan on doing more experiments and analysis during the remainder of the rebuttal.
On why DTM works in vision settings, the submitted paper analyzes evidence retention (Appendix E). DTM retains the most task-relevant spatial evidence in 22 of 24 model–dataset comparisons at 80% deletion (Section 6.3), and where it keeps more evidence than FastV, it alone answers correctly on 13.5% of questions, against 6.2% for FastV alone (Appendix E.2). Because the gate reads hidden states rather than attention weights, it stays nearly flat on T5Gemma 2 when the preceding layer switches from local to global attention, where the attention-based selectors drop (Appendix D.7), and in Gemma 3 it succeeds without seeing the question. We are also running further analyses of how DTM's gate selects tokens compared with other selectors, which we will share in a follow-up comment.
Weakness 2
We’ve added three baselines: Average Pooling (LLaVolta's Visual Context Compressor, fixed stride), FEATHER (a non-learned two-stage selector), and VisionSelector (a learned selector). We also created a new method, “LIS-DTM,” which replaces DTM's linear gate with VisionSelector’s Learned Importance Scorer (LIS, analogous to the deletion gate in DTM). In short, despite a gate hundreds of times smaller, DTM is competitive with VisionSelector in the single-task setting, though not uniformly better: VisionSelector is stronger on DocVQA, while DTM is stronger at aggressive deletion on RefCOCO+, POPE, and VQAv2. In the cross-task setting, DTM leads VisionSelector on Gemma 3 4B, while the results on T5Gemma 2 1B–1B are mixed. With the same scorer, LIS-DTM is on par with or better than VisionSelector on nearly all tasks. More details are described in the “Overall Response to Reviewers (#1)” comment under “Additional Baselines” and “Cross-Task Performance.”
Weakness 3
We’ve completed ablations for softmax1 on baselines and for adding DTM’s gate bias during inference, and detail the results in our comment “Response #1 to Reviewer Wj4X” under “Weakness 3, Softmax1 Experiments” and “Weakness 3, DTM Gate Biases at Inference: Ranking vs. reweighting.” Our “LIS-DTM” is the same method as DTM but with VisionSelector’s Learned Importance Scorer (LIS) instead of DTM’s linear gate, so it acts as an ablation for the design of the deletion gate (see Weakness 2). The submitted paper also ablates several of DTM’s components: PID versus PI control (Appendix A.3), the initial deletion-rate calibration (Appendix A.2), MrT5’s attention-score regularizer (Appendix A.4), natural thresholds versus fixed per-example budgets (Appendix A.5), and the deletion layer (Appendices D.7–D.8), while random deletion serves as a control. If compute permits, we plan to try performing more ablations.