acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Token Merging for Efficient Vision-Language Models

Abstract

Vision-language models (VLMs) represent each image with hundreds of visual tokens, which can make inference costly even when a prompt depends on only part of the image. Dynamic Token Merging (DTM), introduced in MrT5 for byte-level text encoders, learns which tokens to remove: a lightweight delete gate scores contextualized hidden states, acts as a differentiable attention bias during training, and physically removes tokens at inference. We argue that DTM is a general token deletion framework rather than a byte-level technique, and extend it to the vision modality and to decoder-only models. Applying DTM to T5Gemma 2 and Gemma 3 yields V-MrT5Gemma2, which deletes visual tokens inside a bidirectional multimodal encoder, and V-MrGemma3, which compacts the prompt and its key–value caches during decoder-only prefill. Across ten benchmarks, DTM’s advantage over attention-based and random deletion grows with the deletion rate and is largest on document, chart, and scene-text reading. At 80% visual token deletion (≈60% fewer FLOPs), DTM raises accuracy over FastV from 51.8 to 63.0 on ChartQA and from 63.3 to 67.8 on TextVQA with V-MrT5Gemma2, as well as from 62.2 to 68.8 and from 71.2 to 75.1 with V-MrGemma3, retaining 90–96% of unpruned performance on both tasks with significant efficiency gains. These results support DTM as a general framework for efficient transformers.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
vision language models, token pruning, efficient architecture, adaptive compression, encoder-decoder, autoregressive, fine-tuning, dynamic token merging

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Lu2hOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Dynamic Token Merging (DTM), a method for dynamically compressing visual tokens in vision-language models by learning which tokens to retain and which to discard. The authors adapt DTM to both encoder–decoder and decoder-only architectures and introduce a PID-based controller to improve the accuracy of the token deletion rate. Experiments across ten benchmarks show that DTM achieves competitive performance compared with existing token reduction baselines, particularly at high deletion rates, while substantially reducing language-model FLOPs.

Critical strengths
  1. The writing is clear and easy to follow.

  2. The method is deployment-friendly.

Critical weaknesses
  1. The novelty of this paper is limited. Im my view, this is more like an engineering application report. The authors adapt an existing algorithm to a new setting without making contributions to the algorithm itself. While this constitute a meaningful engineering contribution, it is insufficient for a research paper. The authors conduct many experiments, but offer little new scientific insight since MrT5 is already an effective algorithm. More important questions are why it works in vision settings and what underlying mechanisms account for its effectiveness. But these experiments primarily demonstrate that the method can achieve good performance across benchmarks.

  2. The authors mentioned many related works in the related work section, but the experiments only compare with few relatively naive baselines. Is this a fair comparison? Does DTM still demonstrate advantages over existing advanced methods for token compression?

  3. Could you provide ablation studies for components, like soft-deletion training, the learnable gate, and etc, to better quantify the contribution of each component to vision token compression.

Questions for authors

See weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Lu2h (#1)

We appreciate the points raised in your review and your suggestions for improving the paper. We’ve addressed shared concerns across reviewers in our overall comment, “Overall Response to Reviewers (#1).” We take your critiques seriously and respond to each below:

Weakness 1

The novelty of this paper is limited. Im my view, this is more like an engineering application report. The authors adapt an existing algorithm to a new setting without making contributions to the algorithm itself. While this constitute a meaningful engineering contribution, it is insufficient for a research paper. The authors conduct many experiments, but offer little new scientific insight since MrT5 is already an effective algorithm.

We understand your concern that our work adapts an existing algorithm to a new setting. While DTM’s core mechanism comes from MrT5, applying it to VLMs requires changes to the algorithm itself (Section 4). We extended DTM to decoder-only models, which deletes visual tokens during prompt prefill, so that later layers compute and cache only the compacted prompt, and the gate must score visual states that cannot see the question due to the nature of causal attention. We add a derivative term to MrT5’s controller (Appendix A.3), shift the gate biases seen by answer queries, and calibrate the initial deletion rate (Appendix A.2). Moreover, we show that a single checkpoint trained at one deletion rate remains effective across inference budgets (Section 6.2) and provide insights into the effectiveness and mechanics of DTM, as described below. In the “Overall Response to Reviewers (#1)” comment under “Additional Baselines” and “Cross-Task Performance,” we have further shown that the DTM framework is not tied to its own gate: applied to the learned scorer of VisionSelector, the resulting LIS-DTM performs on par with or better than VisionSelector on nearly all benchmarks. We have added results for a single DTM model trained across tasks, and we plan on doing more experiments and analysis during the remainder of the rebuttal.

More important questions are why it works in vision settings and what underlying mechanisms account for its effectiveness. But these experiments primarily demonstrate that the method can achieve good performance across benchmarks.

On why DTM works in vision settings, the submitted paper analyzes evidence retention (Appendix E). DTM retains the most task-relevant spatial evidence in 22 of 24 model–dataset comparisons at 80% deletion (Section 6.3), and where it keeps more evidence than FastV, it alone answers correctly on 13.5% of questions, against 6.2% for FastV alone (Appendix E.2). Because the gate reads hidden states rather than attention weights, it stays nearly flat on T5Gemma 2 when the preceding layer switches from local to global attention, where the attention-based selectors drop (Appendix D.7), and in Gemma 3 it succeeds without seeing the question. We are also running further analyses of how DTM's gate selects tokens compared with other selectors, which we will share in a follow-up comment.

Weakness 2

We’ve added three baselines: Average Pooling (LLaVolta's Visual Context Compressor, fixed stride), FEATHER (a non-learned two-stage selector), and VisionSelector (a learned selector). We also created a new method, “LIS-DTM,” which replaces DTM's linear gate with VisionSelector’s Learned Importance Scorer (LIS, analogous to the deletion gate in DTM). In short, despite a gate hundreds of times smaller, DTM is competitive with VisionSelector in the single-task setting, though not uniformly better: VisionSelector is stronger on DocVQA, while DTM is stronger at aggressive deletion on RefCOCO+, POPE, and VQAv2. In the cross-task setting, DTM leads VisionSelector on Gemma 3 4B, while the results on T5Gemma 2 1B–1B are mixed. With the same scorer, LIS-DTM is on par with or better than VisionSelector on nearly all tasks. More details are described in the “Overall Response to Reviewers (#1)” comment under “Additional Baselines” and “Cross-Task Performance.”

Weakness 3

We’ve completed ablations for softmax1 on baselines and for adding DTM’s gate bias during inference, and detail the results in our comment “Response #1 to Reviewer Wj4X” under “Weakness 3, Softmax1 Experiments” and “Weakness 3, DTM Gate Biases at Inference: Ranking vs. reweighting.” Our “LIS-DTM” is the same method as DTM but with VisionSelector’s Learned Importance Scorer (LIS) instead of DTM’s linear gate, so it acts as an ablation for the design of the deletion gate (see Weakness 2). The submitted paper also ablates several of DTM’s components: PID versus PI control (Appendix A.3), the initial deletion-rate calibration (Appendix A.2), MrT5’s attention-score regularizer (Appendix A.4), natural thresholds versus fixed per-example budgets (Appendix A.5), and the deletion layer (Appendices D.7–D.8), while random deletion serves as a control. If compute permits, we plan to try performing more ablations.

AuthorsOpenReview ↗

Response to Lu2h (#2)

Following up on our last response, we present additional results and analysis in our “Overall Response to Reviewers (#2)” comment: cross-task results on a third architecture, Qwen2.5-Omni-3B, including video and multi-image document tasks (under “Qwen2.5-Omni Cross-Task Results and Long-Context”); an analysis of the tokens each method keeps (under “Insights into Token Pruning Mechanisms”); and DTM applied to multimodal embeddings (under “Extending DTM to Multimodal Embeddings”). We have also uploaded a revised paper.

Weakness 1. You asked why DTM works in vision settings and what mechanisms account for its effectiveness. As promised in our first response, the section of that comment under “Insights into Token Pruning Mechanisms” (Appendix F of the revised paper) analyzes the keep decisions and per-token scores of DTM, LIS-DTM, VisionSelector, FastV, and φ-R at 80% deletion:

  • DTM and LIS-DTM keep tokens where the answers are. On Gemma 3 4B and Qwen2.5-Omni-3B, their keep maps correlate with where the answers are at Pearson r = 0.61–0.74, against r ≤ 0.23 for FastV and φ-R (Figure 16).
  • Their gates make near-binary decisions. 69–77% of DTM’s gate values (95–96% for LIS-DTM) fall below 0.05 or above 0.95, so the switch from soft deletion in training to hard deletion at inference leaves few tokens in doubt (Figure 17).
  • DTM’s gains come from which tokens it keeps. When T5Gemma 2 1B–1B DocVQA models are re-evaluated with another method’s keep sets, DTM’s sets raise FastV’s and φ-R’s models by 11.5 and 12.9 points, and theirs lower DTM’s by about 20 points; on Gemma 3 4B RefCOCO+, DTM’s lead over FastV and φ-R grows from 1.4–1.5 points on central referents to 7.3–8.2 on small ones (Figures 18 and 19).

Weakness 2. Beyond the Average Pooling, FEATHER, and VisionSelector baselines from our first response, we now compare all methods on Qwen2.5-Omni-3B and on multimodal embeddings. While DTM and LIS-DTM do not beat VisionSelector, our strongest baseline, everywhere, they beat it more often than they trail it:

  • At 60% and 80% deletion: LIS-DTM leads in 116 of 142 benchmark comparisons and DTM in 89 of 142, and on the 14 multimodal embedding sets, LIS-DTM leads in 24 of 28 and DTM in 17 of 28.
  • Where our methods lead, the average improvement over VisionSelector is 2.4 points for LIS-DTM and 2.2 for DTM.
  • Where they trail, the average deficit is 1.1 points for LIS-DTM and 1.5 for DTM.
  • Parameters: DTM’s gate has 2.3k–5.1k parameters, about 580–1,280× fewer than VisionSelector’s scorer (1.3M–6.6M), and LIS-DTM, which adds the same parameters as VisionSelector, leads it most often.

As for novelty, we have extended DTM from MrT5’s byte-level text to visual tokens, video, multi-image documents, and multimodal embeddings, and shown that its training recipe improves another method’s learned scorer (LIS-DTM). To our knowledge, we are the first paper to apply the DTM framework for vision-language tasks with different types of vision encoders, to the decoder-only architecture, and to embedding-related tasks. These adaptations required significant implementation work, e.g., for decoder-only LLMs and for contrastive embedding training. In addition, we employ unique analyses such as the retention of task-relevant spatial evidence (Section 6.3), keep maps compared with where the answers are, the distributions of gate scores, accuracy by referent size and location, and swapping keep sets between trained models, and directly comparing our method’s mechanism of training a learned token selector with that of another (i.e., LIS-DTM vs. VisionSelector). These analyses explain why the learned gate selects better tokens, which vision token pruning papers typically do not examine. Finally, we believe “DTM as a general framework” is sufficient for novelty because our successful applications were not immediately obvious: it was not clear that a gate designed for byte-level text would transfer across vision encoders, architectures, and tasks, or that its training recipe would improve another method’s scorer.

While our paper is not necessarily novel in its mechanism (the authors of MrT5 introduce DTM), we believe our work’s contributions lies in showing whether, when, and why an established mechanism works in new modalities, tasks, deletion gates, and architectures, which our experiments and analyses do for DTM in vision-language models.

We hope you’ll consider raising your score, and we sincerely thank you again for your time to review our work.

Reviewer_Wj4XOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper extends MrT5’s Dynamic Token Merging to visual tokens in encoder–decoder and decoder-only VLMs. A lightweight gate combines soft deletion during training with hard deletion at inference and deletion-rate control. Experiments on three backbone configurations and ten benchmarks show its clearest advantages over the evaluated attention-based and random selectors at aggressive compression.

Critical strengths

I am glad to see every selector receive task fine-tuning rather than comparing a trained method only against unadapted models. The main experiments also match the deletion depth and retained-token count per example. The query-aggregation and depth sweeps in Appendices D.6–D.7 strengthen the comparison. Random deletion is a useful additional control for whether the selection strategy contributes beyond adaptation to shorter sequences.

Critical weaknesses

I have presented the weaknesses together with the relevant details to make it easier for the authors to address them during the rebuttal period. If my concerns are adequately addressed, I would be willing to raise my score.

  1. In my view, the experiments establish that DTM improves on the tested fixed selection rules, but not yet why its particular learning recipe is preferable to other learned selectors. The gate and soft-to-hard paradigm come from MrT5, while the Related Work already discusses DynamicViT, VisionSelector, and LearnPruner. The evaluated φ-R is also only FEATHER’s scoring component, not its complete system. I would like at least one learned-selector adaptation under a matched budget, or a comparison of differentiable top-k/gating recipes within the same backbone. This would separate the benefit of learning a selector from the benefit of DTM specifically.

  2. Efficiency is a central motivation, but Section 5 and Appendices F–G primarily report analytical language-backbone FLOPs. The authors explicitly exclude the vision encoder and projector and state that these are neither wall-clock measurements nor detailed cached-decoding costs. That qualification is appropriate, but it means the roughly 60% savings do not establish an end-to-end benefit, particularly when image processing still starts from 4,096 patches. Please report time to first token including image processing, fixed-output-length decoding speed, peak memory, and the attention backend actually used. Token gathering, gate biases, and cache-compaction overhead should be included.

  3. Section 4.3 states that retained tokens still carry gate biases in some attention operations after hard deletion; under fixed-budget inference, some may also remain downweighted during answering. DTM therefore changes both which tokens survive and how surviving tokens are read. The soft–hard analysis in Appendix A.4 is useful, but its DocVQA experiment still shows an 8.03-point gap. I would like an ablation that holds the retained indices fixed and removes biases on surviving tokens, along with clarification of whether all baselines use the same softmax1 convention. This would identify how much of the high-compression gain comes from ranking versus additional attention reweighting.

  4. The single-checkpoint result means one task-specific checkpoint serving multiple budgets, not one model serving all tasks. Apart from POPE reusing VQAv2 checkpoints, the evaluation relies on task-specific adaptation. The three configurations also share SigLIP and a 256-token image interface, and T5Gemma 2 and Gemma 3 have related architectural origins. These results support a valuable architectural extension, but do not establish reuse on unseen tasks or substantially different visual sequence structures. A jointly trained multitask model, cross-task transfer, or a different visual interface would strengthen the generality claim. At minimum, the abstract and conclusion should make the current scope explicit.

Minor Weaknesses: Several high-compression gaps are large, but differences at mild compression and for controller modifications are sometimes below one point. Please add repeated training runs or paired bootstrap intervals and specify whether checkpoint selection and final reporting use the same validation set. The documented training budgets and splits are helpful, but do not establish the stability of these smaller differences.

Questions for authors

For clarity and to facilitate the rebuttal process, I have already outlined the key strengths and weaknesses that informed my rating in the sections above. Please refer to the Strengths and Weaknesses sections for the detailed rationale. I would be willing to reconsider my rating if the authors adequately address my main concerns during the rebuttal.

5 comments
AuthorsOpenReview ↗

Response to Wj4X (#1)

We appreciate immensely the points raised in your review and your suggestions for improving the paper. We’ve addressed shared concerns across reviewers in our overall comment, “Overall Response to Reviewers (#1).” We take your critiques seriously and respond to each below:

Weakness 1

We’ve added three baselines: Average Pooling (LLaVolta's Visual Context Compressor, fixed stride), FEATHER (a non-learned two-stage selector; its full system, not only the φ-R criterion), and VisionSelector (a learned selector), each under matched budgets. We also created a new method, “LIS-DTM,” which is DTM with its linear gate replaced by VisionSelector’s Learned Importance Scorer (LIS), its “scorer” (analogous to the deletion gate in DTM). Because LIS-DTM and VisionSelector share the same scorer, comparing them separates the benefit of learning a selector from the benefit of DTM’s Soft-Hard Additive Gate Bias + PID controller recipe. In short, despite a gate hundreds of times smaller, DTM is competitive with VisionSelector in the single-task setting, though not uniformly better: VisionSelector is stronger on DocVQA, while DTM is stronger at aggressive deletion on RefCOCO+, POPE, and VQAv2. With the same scorer, LIS-DTM is on par with or better than VisionSelector on nearly all tasks, which shows that DTM’s recipe substantially improves performance. More details are described in the “Overall Response to Reviewers (#1)” comment under “Additional Baselines.”

Weakness 2:

We report end-to-end wall-clock measurements for DTM and every baseline on T5Gemma 2 1B–1B and Gemma 3 4B; the tables are in the “Overall Response to Reviewers (#1)” comment under “Systems Analysis.” In brief:

  • Time to first token, including image processing: the vision encoder takes about 21 ms per image for every method, since all of them delete inside the language model. With one image at batch 1, no method shortens the first token by more than 2 ms; on inputs with 16 images (about 4,200 tokens), DTM shortens it by 14% at batch 1 and by 30–37% at batch 8 (Gemma 3 4B).
  • Fixed-length decoding (32 output tokens) and peak memory: at batch 256, DTM decodes 11% (Gemma 3 4B) and 27% (T5Gemma 2) faster than no deletion. Peak memory falls from 26.9 to 18.6 GB on Gemma 3 4B; on T5Gemma 2 it is 12.2 GB with or without deletion, because at batch 256 the peak comes while the images are encoded, before any token is deleted.
  • Attention backend: with each library’s stock PyTorch attention, softmax1 makes DTM decode 3–9% slower than VisionSelector; on one shared custom-written Triton kernel used by every method, DTM is within 1.1% of VisionSelector.
  • Overheads: every timing includes the gate (0.4 ms), token gathering and the gate biases. DTM drops tokens during prefill, so there is no cache-compaction step.

Weakness 4

We’ve trained our T5Gemma2 1B–1B and Gemma3 4B baselines on ChartQA, TextVQA, VQAv2, and RefCOCO+ and subsequently evaluated them on all ten of our benchmarks. Details and results are in the “Overall Response to Reviewers (#1)” comment under “Cross-Task Performance.” In our updated paper, we will ensure the results and analysis of single-task and cross-task settings are clearly indicated.

Minor Weaknesses

Hyperparameter Tuning and Checkpoint Selection: For each benchmark, we held out approximately 3,000 randomly selected examples from the training set to serve as our “development set,” on which we tuned hyperparameters, and made sure no images overlap in the training and development sets. We saved each token-deletion baseline’s checkpoint based on its best development-set performance over the training run with the final set of hyperparameters. We then evaluated each saved checkpoint on the benchmarks’ validation sets, from which we report the results presented in this paper, except for TallyQA and POPE, which only publicly provide test sets. All benchmarks had at least 20,000 examples in each of their original training sets, except for POPE, which is eval-only.

Additional Runs: Computing resources permitting, we plan to conduct additional runs with different seeds for baselines whose comparisons with other baselines show small performance deltas at specific settings. We will also run paired bootstrap evaluations. Once we have obtained these results, we will report them in a follow-up comment. We agree that these small differences may be substantially attributed to stochasticity, and we will be careful in characterizing performance differences and their interpretation, including in our updated paper.

Weakness 3, Softmax1 Experiments

We retrain each method’s 80% deletion single-task checkpoint with only the softmax1 setting flipped. Only DTM and LIS-DTM use softmax1; FastV, φ-R, VisionSelector, FEATHER, and Average Pooling do not (as in their official implementations), nor do Random and the no token-pruning baseline. We evaluate at 80% deletion (matched-rate evaluation) on ChartQA, TextVQA, VQAv2 and RefCOCO+; Δ = softmax1 on − off.

T5Gemma 2 1B–1B (gate at layer 5)

Methodsoftmax1ChartQATextVQAVQAv2RefCOCO+
DTMoff62.667.275.874.6
on (paper)63.067.875.675.3
Δ+0.4+0.6-0.2+0.7
FastVoff (paper)51.863.372.473.3
on52.062.971.868.8
Δ+0.2−0.4−0.6−4.5
φ-Roff (paper)55.558.471.671.5
on52.958.171.771.7
Δ−2.6−0.3+0.1+0.2
Randomoff (paper)51.055.172.470.0
on50.956.372.070.0
Δ−0.1+1.2−0.40.0
VisionSelectoroff (paper)62.869.074.267.7
on63.268.774.363.4
Δ+0.4−0.3+0.1−4.3
LIS-DTMoff64.867.875.373.7
on (paper)63.867.975.475.3
Δ−1.0+0.1+0.1+1.6

Gemma 3 4B (gate at layer 5)

Methodsoftmax1ChartQATextVQAVQAv2RefCOCO+
DTMoff67.774.876.471.5
on (paper)68.875.176.872.7
Δ+1.1+0.3+0.4+1.2
FastVoff (paper)62.271.273.869.5
on62.971.273.769.5
Δ+0.70.0−0.10.0
φ-Roff (paper)61.969.374.069.4
on60.670.074.368.0
Δ−1.3+0.7+0.3−1.4
Randomoff (paper)58.465.373.669.6
on57.065.073.568.8
Δ−1.4−0.3−0.1−0.8
VisionSelectoroff (paper)68.073.772.771.3
on66.371.875.267.2
Δ−1.7−1.9+2.5−4.1
LIS-DTMoff71.476.076.973.2
on (paper)72.175.776.973.7
Δ+0.7−0.30.0+0.5

These results suggest that DTM and the baselines generally perform worse when softmax1 is flipped from the setting used in the paper. Most changes are within about one point. Even with softmax1 off, and setting LIS-DTM aside, DTM matches or exceeds every baseline in the table, under either setting, on VQAv2 and RefCOCO+ for both T5Gemma 2 1B–1B and Gemma 3 4B.

Weakness 3, DTM Gate Biases at Inference: Ranking vs. reweighting

We evaluate DTM with the gate biases on the remaining vision tokens (consistent with the paper) and without the gate biases on the remaining vision tokens for T5Gemma 2 1B–1B model and Gemma 3 4B. Applying the gate biases occurs after deletion, so these comparisons between with vs. without adding the biases hold the retained indices fixed (i.e., the set of deleted tokens are identical). We evaluate our trained-at-80% single-task checkpoints at 80% deletion (matched-rate evaluation), with Δ = biases off − on:

T5Gemma 2 1B–1B (gate at layer 5)

BenchmarkBiases onBiases offΔ
DocVQA57.554.9−2.6
ChartQA63.062.7−0.3
InfoVQA34.135.8+1.7
TextVQA67.868.5+0.7
VQAv275.675.60.0
VizWiz69.669.60.0
GQA70.069.9−0.1
POPE84.084.3+0.3
TallyQA72.072.1+0.1
RefCOCO+75.367.6−7.7

Gemma 3 4B (gate at layer 5)

BenchmarkBiases onBiases offΔ
DocVQA58.852.0−6.8
ChartQA68.866.8−2.0
InfoVQA40.039.3−0.7
TextVQA75.174.0−1.1
VQAv276.876.80.0
VizWiz70.867.0−3.8
GQA68.067.5−0.5
POPE84.084.3+0.3
TallyQA71.668.9−2.7
RefCOCO+72.715.4−57.3

On T5Gemma 2 1B–1B, ranking would seem to account for most of the gain. Without the biases, seven of ten benchmarks change by at most 0.7 points. DTM still exceeds the strongest of FastV, φ-R and Random on nine of ten benchmarks, by 2.0–7.2 points. The biases matter most for grounding and documents (RefCOCO+ falls from 75.3 to 67.6, below FastV’s 73.3, and DocVQA falls from 57.5 to 54.9). On Gemma 3 4B, reweighting would seem to contribute more: removing it lowers six of ten benchmarks by more than one point, and the other four change by less than one point. The largest drops are RefCOCO+ (72.7 to 15.4), DocVQA (58.8 to 52.0), VizWiz (70.8 to 67.0) and TallyQA (71.6 to 68.9). With ranking alone, DTM still exceeds the strongest of FastV, φ-R and Random by 2.6–4.6 points on seven of ten benchmarks: DocVQA, ChartQA, InfoVQA, TextVQA, VQAv2, GQA and POPE.

LIS-DTM. The same ablation on our trained-at-80% LIS-DTM single-task checkpoints, again with identical retained tokens on every example:

T5Gemma 2 1B–1B, LIS-DTM (gate at layer 5)

BenchmarkBiases onBiases offΔ
DocVQA62.262.8+0.6
ChartQA63.863.9+0.1
InfoVQA35.736.3+0.6
TextVQA67.968.3+0.4
VQAv275.575.6+0.1
VizWiz69.869.6−0.2
GQA69.669.5−0.1
POPE84.084.1+0.1
TallyQA72.072.2+0.2
RefCOCO+75.375.5+0.2

Gemma 3 4B, LIS-DTM (gate at layer 5)

BenchmarkBiases onBiases offΔ
DocVQA65.866.8+1.0
ChartQA72.172.2+0.1
InfoVQA44.444.5+0.1
TextVQA75.775.70.0
VQAv276.977.0+0.1
VizWiz72.372.5+0.2
GQA68.468.5+0.1
POPE84.684.60.0
TallyQA72.172.3+0.2
RefCOCO+73.774.1+0.4

Unlike DTM, LIS-DTM seems to usually perform better with biases off; having biases on attains slightly higher performance only on two benchmarks for 1B–1B and none for 4B. We have reported all of our LIS-DTM results with the gate bias added at inference and will continue to do so; keeping the gate bias at inference is consistent with our implementation of DTM.

Reviewer_KSH8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper applies Dynamic Token Merging (DTM), originally introduced in MrT5 for byte-level text models, to visual token deletion in vision-language models. DTM places a lightweight learned delete gate at a fixed layer that scores contextualized hidden states. During training, the gate scores are added to the attention logits as soft biases, and at inference, low-scoring tokens are directly removed. A controller pushes the deletion rate toward a target. The authors restrict deletion to visual tokens and experiment the method with both encoder-decoder model and decoder-only model. The method is compared against baselines like FastV where the baseline is also trained with lora to ensure fairness. DTM shows its largest gains under aggressive deletion on document, chart and scene-text tasks.

Critical strengths
  • The experimental design is fair. Unlike many pruning papers, all baselines receive the same LoRA fine-tuning, so the comparison isolates the effect of the proposed method.
  • The method achieves clear gains over the baselines for both types of model, compared to Lora tuned FastV and φ-R. The matched-rate and fixed-checkpoint settings are intuitive design that fits different use cases.
Critical weaknesses
  • The novelty is limited. The core mechanism (learned delete gate, soft-to-hard deletion and rate controller) comes directly from MrT5. The main contribution is applying it to visual tokens. Positioning DTM as "a general framework for token deletion in transformers" overstates what is mainly a transfer of an existing method to a new modality.

  • The experimental setting is far from how VLMs are actually used and deployed. Every method is fine-tuned separately on each benchmark, meaning a separate model for every task. In practice, people use a single general instruction-tuned model across many tasks, so it's unclear whether DTM's advantage holds there. This setup also rules out the plug-and-play usage that makes training-free pruning methods attractive.

  • The paper experimented with T5Gemma 2 (270M and 1B) and Gemma 3 4B, and all models use a fixed 256 visual tokens per image. Visual token pruning matters most for models with high or dynamic resolution, such as the Qwen-VL and InternVL series, which produce thousands of visual tokens per image, and for video inputs. None of these are evaluated. So it’s unclear whether the method generalize to scenarios where the visual token amount is truly a bottleneck.

  • The paper only compared against FastV and φ-R in lora fine-tuned settings, but there are other learned visual token pruning methods like VisionSelector and LearnPruner, those are discussed but not directly compared.

Questions for authors
  1. Can you evaluate DTM on VLMs with many visual tokens per image (e.g., Qwen2.5-VL or Qwen3-VL at native resolution), and on video inputs?
  2. Can you report results with a single multitask or instruction-tuned checkpoint evaluated across all benchmarks, instead of per-task fine-tuning?
  3. Can you report wall-clock prefill latency, decoding throughput and KV-cache memory?
  4. How does DTM compare with learned selectors such as VisionSelector or the full LearnPruner?
3 comments
AuthorsOpenReview ↗

Response to KSH8 (#1)

We appreciate immensely the points raised in your review and your suggestions for improving the paper. We’ve addressed shared concerns across reviewers in our overall comment, “Overall Response to Reviewers (#1).” We take your critiques seriously and respond to each below:

Weakness 1:

We recognize your concern about the novelty of our work on DTM, and we agree that its core mechanism (the learned delete gate, soft-to-hard deletion, and the rate controller) comes from MrT5. Where MrT5 only evaluated on byte-level language tasks, we have shown that DTM works effectively for visual token deletion in both an encoder-decoder model (T5Gemma 2) and a decoder-only model (Gemma 3), and that a single trained checkpoint can be evaluated across deletion budgets. Furthermore, we provide insights into what DTM’s learned gate keeps and how it differs from attention-based selection. In Section 6.3, DTM achieves the highest mean retention of task-relevant spatial evidence in 22 of 24 model–dataset comparisons at 80% deletion, concentrating its budget on answer-bearing content such as chart titles, tick labels, and bar ends. On positioning DTM as a general framework, we understand your point; we’ve added a VisionSelector (a learned selector) baseline and applied the framework to VisionSelector’s learned scorer (LIS-DTM), showing DTM’s methodological versatility. More details are described in the “Overall Response to Reviewers (#1)” comment under “Additional Baselines.”

We will continue to report new experiments, results, and analysis. We will summarize the full set of contributions made during the rebuttal in a follow-up comment.

Weakness 2:

Your point that training-free pruning methods offer plug-and-play convenience is valid. However, we believe that fine-tuned token deletion serves a different and practical need: many deployments focus on a narrow, high-volume task where both accuracy and efficiency matter, and many such tasks, such as document parsing, often require fine-tuning to perform well.

For example, a company handling legal or financial work may want an efficient yet highly accurate vision-language model specifically to parse documents with visuals, and a robotics company may need a model specialized for referring-expression grounding to label vast quantities of images. Such users could lightly fine-tune their model with LoRA and DTM, which does not require immense compute relative to full fine-tuning or continued pretraining. To address the single-model setting you describe, we also trained each method once per backbone on a mixture of ChartQA, TextVQA, VQAv2, and RefCOCO+ and evaluated it on all ten benchmarks (see our response to Q2).

Q1 for Authors / Weakness 3:

Yes, we will train and evaluate our baselines and DTM on Qwen2.5-Omni (“Qwen2.5-Omni Technical Report,” https://arxiv.org/abs/2503.20215), at native resolution and with their dynamic tiling image processing. We will also test on video benchmark(s). You can expect from us the relevant results in a follow-up comment.

Q2 for Authors:

Yes, we trained our T5Gemma2 1B–1B and Gemma3 4B baselines on ChartQA, TextVQA, VQAv2, and RefCOCO+ and subsequently evaluated them on all ten of our benchmarks to test task transfer capabilities. Details and results are in the “Overall Response to Reviewers (#1)” comment under “Cross-Task Performance.” In brief, on 4B, DTM outperforms VisionSelector on most benchmarks, including DocVQA, which is not a training task, while the results on 1B–1B are mixed. With the same scorer, LIS-DTM leads VisionSelector more often than it trails on both models.

Q3 for Authors:

Yes, we measured wall-clock time to first token (including the vision encoder), decoding throughput, and KV-cache and peak memory for DTM and every baseline. The details are in the “Overall Response to Reviewers (#1)” comment under “Systems Analysis.” In brief, on the same attention kernel, DTM is about as fast as the other baselines, and its gate takes 0.4 ms. At 80% deletion and batch 256, DTM decodes 11% (Gemma 3 4B) and 27% (T5Gemma 2 1B–1B) faster than the model without deletion and shrinks the KV cache by 53% and 62%. On inputs with 16 images (~4,200 prompt tokens), it shortens Gemma 3 4B’s time to first token by 14% at batch 1 and by 30–37% at batch 8.

Q4 for Authors / Weakness 4:

We’ve added results for Average Pooling (LLaVolta's Visual Context Compressor, fixed stride), FEATHER (a non-learned two-stage selector), VisionSelector (a learned selector), and “LIS-DTM” baselines, and compare them with DTM. The details and results are in the “Overall Response to Reviewers (#1)” comment under “Additional Baselines.” In brief, DTM is competitive with VisionSelector despite a much simpler gate, though not uniformly better: VisionSelector is stronger on DocVQA, while DTM is stronger at aggressive deletion on RefCOCO+, POPE, and VQAv2. With the same scorer, LIS-DTM is on par with or better than VisionSelector on nearly all tasks.

Reviewer_KSH8OpenReview ↗

Response to authors

Thank you providing the additional results. While it is encouraging to see LIS-DTM achieves better results in some of the benchmarks, the overall results still seem mixed and VisionSelector is getting better results on the other benchmarks. Moreoever, I'm still not convinced that framing the DTM as a general framework is sufficient for the novelty argument, and I think this concern is shared by other reviewers too. So I maintain my overall score.

AuthorsOpenReview ↗

Response to KSH8 (#2)

Thank you for taking the time to look through our responses. As requested, we’ve trained and evaluated DTM and all baselines on Qwen2.5-Omni-3B at native resolution (about 256–1,280 visual tokens per image), as one model trained on four tasks and evaluated on all ten benchmarks, and on video and multi-image documents (up to 5,100 visual tokens per video clip). We have some additional results to share for your two remaining concerns; details are in our “Overall Response to Reviewers (#2)” comment.

You mention that the overall results “still seem mixed.” While DTM and LIS-DTM do not beat VisionSelector everywhere, they beat it more often than they trail it.

  • At 60% and 80% deletion: LIS-DTM leads in 116 of 142 benchmark comparisons and DTM in 89 of 142, and on the 14 multimodal embedding sets, LIS-DTM leads in 24 of 28 and DTM in 17 of 28.
  • Where our methods lead, the average improvement over VisionSelector is 2.4 points for LIS-DTM and 2.2 for DTM.
  • Where they trail, the average deficit is 1.1 points for LIS-DTM and 1.5 for DTM.
  • Parameters: DTM’s gate has 2.3k–5.1k parameters, about 580–1,280× fewer than VisionSelector’s scorer (1.3M–6.6M), and LIS-DTM, which adds the same parameters as VisionSelector, leads it most often.

Therefore, we believe that our DTM methods are better overall, especially in a fair parameter count comparison.

As for novelty, we have extended DTM from MrT5’s byte-level text to visual tokens, video, multi-image documents, and multimodal embeddings, and shown that its training recipe improves another method’s learned scorer (LIS-DTM). To our knowledge, we are the first paper to apply the DTM framework for vision-language tasks with different types of vision encoders, to the decoder-only architecture, and to embedding-related tasks. These adaptations required significant implementation work, e.g., for decoder-only LLMs and for contrastive embedding training. In addition, we employ unique analyses such as the retention of task-relevant spatial evidence (Section 6.3), keep maps compared with where the answers are, the distributions of gate scores, accuracy by referent size and location, and swapping keep sets between trained models, and directly comparing our method’s mechanism of training a learned token selector with that of another (i.e., LIS-DTM vs. VisionSelector). These analyses explain why the learned gate selects better tokens, which vision token pruning papers typically do not examine. Finally, we believe “DTM as a general framework” is sufficient for novelty because our successful applications were not immediately obvious: it was not clear that a gate designed for byte-level text would transfer across vision encoders, architectures, and tasks, or that its training recipe would improve another method’s scorer.

While our paper is not necessarily novel in its mechanism (the authors of MrT5 introduce DTM), we believe our work’s contributions lies in showing whether, when, and why an established mechanism works in new modalities, tasks, deletion gates, and architectures, which our experiments and analyses do for DTM in vision-language models.

We hope you’ll consider raising your score, and we sincerely thank you again for your time to review our work.

Comments on the paper

8 comments
AuthorsOpenReview ↗

Overall Response to Reviewers (#1)

We sincerely thank each reviewer for their careful review and detailed feedback of our paper! Here, we would like to address concerns that were shared among reviewers. We will respond directly to reviewers for individually raised points.

Additional Baselines

We ran additional experiments and added Average Pooling (LLaVolta's Visual Context Compressor, fixed stride), FEATHER (a non-learned two-stage selector), and VisionSelector (a learned selector), which has three main components:

  1. Learned Importance Scorer (LIS): a self-attention-inspired scorer, si=1N∑jqi⊤kj/ds_i=\frac{1}{N}\sum_j\mathbf q_i^\top\mathbf k_j/\sqrt{d} over learned query/key projections of hidden states; DTM’s linear gate scores each token alone, gi=c σ[−(w⊤RMSNorm(hi)+β)]g_i=c\,\sigma[-(\mathbf w^\top\mathrm{RMSNorm}(\mathbf h_i)+\beta)].
  2. Differentiable Mask: a soft top-KK mask mi=σ(si+t)m_i=\sigma(s_i+t), ∑imi=K\sum_i m_i=K, multiplies the tokens, whereas DTM directly adds gjg_j to key jj’s attention logits.
  3. Soft-Hard Masking: VisionSelector adds λtBCE(m,mhard)\lambda_t\mathrm{BCE}(m,m^{\mathrm{hard}}) to the loss, pulling the soft mask toward the hard one; DTM uses a gate regularizer αt1∣V∣∑jgj\alpha_t\frac{1}{|\mathcal V|}\sum_j g_j, with αt\alpha_t set by a PID controller to reach the target rate.

To illustrate DTM’s framework argument and to obtain a learned-gate ablation, we design “LIS-DTM.” In essence, LIS-DTM is DTM with its linear gate replaced by the LIS from VisionSelector. The LIS has 1.3M–6.6M parameters, about 580–1,280× more than DTM’s gate (2.3k–5.1k). LIS-DTM and VisionSelector add the same parameters and FLOPs, so comparing them isolates DTM's Soft-Hard Additive Gate Bias + PID controller paradigm against VisionSelector’s Soft-Hard Differentiable Masking + Binary Cross-Entropy (BCE) paradigm.

We train these new baselines at 60% and 80% deletion with the same hyperparameters as the other baselines (inserted at layer 5, same number of training steps, etc.), using their reference values for method-specific hyperparameters. Since FEATHER is a two-stage method, we adjust its gates’ deletion rates to match the FLOPs reductions of the other baselines for each benchmark and model, and insert its gates at layers 5 and 10. For Average Pooling, we only trained and tested with stride length 5, which removes 80.1% of tokens; no stride length corresponds to 60% deletion. In the tables below, each benchmark has four columns: the first three show the 60%-trained checkpoint at 40%, 60%, and 80% deletion, and the fourth shows the 80%-trained checkpoint at 80% deletion.

Without LIS-DTM:

With LIS-DTM:

DTM vs. VisionSelector: despite a much simpler gate, DTM is competitive with VisionSelector, though not uniformly better. Counting gaps of ≥1 point, DTM leads/trails in 10/12 of 40 columns on 1B–1B and 17/11 on Gemma 3 4B. VisionSelector is stronger on DocVQA (by 2.1–4.6 at every rate); DTM is stronger at aggressive deletion, e.g., with the 60%-trained checkpoint at 80% on RefCOCO+ (by 10.8 on 1B–1B and 5.3 on 4B), POPE (2.2, 4.8), and VQAv2 (1.4, 2.1).

LIS-DTM vs. VisionSelector: with the same scorer, LIS-DTM is on par or better on nearly all tasks: it trails by ≥1 point in only 5 of 40 columns on 1B–1B and 1 on Gemma 3 4B (mostly at 40%), and leads in 13 and 30 columns respectively. Since both use the same scorer, this shows that DTM’s Soft-Hard Additive Gate Bias + PID controller paradigm substantially improves performance.

Cross-Task Performance

We train T5Gemma2 1B–1B and Gemma3 4B for 3000 steps on ChartQA, TextVQA, VQAv2, and RefCOCO+. Each step trained on the same number of examples of each dataset (equal 25% ratio in the training mixture), and used the same hyperparameter values as we did with the single-task runs.

We evaluate each checkpoint on all ten benchmarks; six of them (DocVQA, InfoVQA, VizWiz, GQA, TallyQA, and POPE) are not among the training tasks. The tables below are presented in a similar manner to the tables mentioned for “Additional Baselines.”

Without LIS-DTM:

With LIS-DTM:

DTM vs. VisionSelector: on Gemma 3 4B, DTM beats VisionSelector across most benchmarks in the cross-task setting. Counting gaps of ≥1 point, it leads in 24 of 40 columns and trails in 1. With the 60%-trained checkpoint at 80%, it leads on all ten benchmarks (by 1.2–7.2 points), and on DocVQA, which is not a training task and where VisionSelector led in the single-task setting, it leads at every rate (by 5.7–9.7). On T5Gemma 2 1B–1B the result is mixed: DTM leads in 8 columns and trails in 17. It leads on DocVQA and InfoVQA (by up to 3.3), but VisionSelector leads on VQAv2, VizWiz, GQA, TallyQA, and POPE, most clearly among models trained at 80% (by 1.4–5.3).

LIS-DTM vs. VisionSelector: with the same scorer, LIS-DTM leads in 14 and trails in 10 columns on 1B–1B, and leads in 27 and trails in 2 on Gemma 3 4B. With the 60%-trained checkpoint at 80%, it leads on eight benchmarks on 1B–1B (by 1.1–5.7) and nine on Gemma 3 4B (by 1.1–11.4), and is within one point on the rest.

Collectively, the new single-task and cross-task results show that:

  • DTM is competitive with a learned selector hundreds of times larger for single-task settings.
  • DTM leads in cross-task settings, with even greater gains for LIS-DTM.
  • LIS-DTM contributes to our argument that DTM is a general framework: its training paradigm applies to other scorers.

Systems Analysis

In short: on the same attention kernel DTM runs as fast as the other selectors (its gate takes 0.4 ms), and like them it saves latency, throughput and memory once inputs are long or batches are large. With one image at batch 1, deletion barely changes the first token for any method.

Setup. We measured wall-clock time and memory end to end on one NVIDIA GH200 GPU: images from raw pixels through the vision encoder, prefill, and 32 forced output tokens, with every method deleting 80% of the image tokens after layer 5. Every timing includes the gate, token gathering and the gate biases. DTM drops tokens during prefill, so later layers do not write them to the KV cache, and there is no separate compaction step.

Attention backend. With each library's stock PyTorch attention, DTM's softmax1 runs as an extra zero key, which makes its decoding 3–9% slower than VisionSelector's. We therefore run every method on the same FlashAttention-2-style Triton kernel, written by us, with CUDA-graph decoding, using softmax1 for DTM and LIS-DTM and ordinary softmax for every other method. The kernel is 2.8–6.5× faster than stock for every method. All numbers below use this shared backend unless noted. For training and for the accuracy results in the paper, the softmax1 methods (DTM, LIS-DTM) use PyTorch’s eager attention and all other methods use SDPA.

Batch 1, one image (about 280 prompt tokens; mean of 4 sessions with the method order alternated). TTFT includes the vision encoder, which takes about 21 ms.

MethodGemma 3 4B TTFT (ms)Gemma 3 4B decode (ms/token)T5Gemma 2 1B–1B TTFT (ms)T5Gemma 2 1B–1B decode (ms/token)
No deletion79.79.33131.66.46
FastV83.69.34133.86.47
φ-R82.59.45131.06.47
Avg. Pooling85.19.37132.56.47
FEATHER93.99.37140.96.19
VisionSelector82.59.42131.46.46
Random82.79.42131.36.52
DTM83.29.46132.06.41
LIS-DTM85.49.24135.46.37

At one image, no method shortens the first token by more than 2 ms (Gemma 3 4B: +2.8 to +14.2 ms over no deletion; T5Gemma 2: −0.6 to +9.4 ms): at this length the GPU is bound by kernel launches, not by sequence length. DTM's gate takes 0.4 ms (VisionSelector's scorer 0.5 ms, LIS-DTM's 0.8 ms), and DTM's TTFT is within 0.7 ms (Gemma 3 4B) and 0.6 ms (T5Gemma 2) of VisionSelector's. The KV cache shrinks with deletion: 40.4 to 17.3 MB per image on Gemma 3 4B, and 7.8 to 2.6 MB on T5Gemma 2.

Batch 256: decoding throughput, KV cache and peak GPU memory.

MethodGemma 3 4B tokens/sGemma 3 4B KV (MB)Gemma 3 4B peak (GB)T5Gemma 2 1B–1B tokens/sT5Gemma 2 1B–1B KV (MB)T5Gemma 2 1B–1B peak (GB)
No deletion13,42411,15226.915,1182,15212.2
FastV14,9695,23618.318,73282612.2
φ-R14,8775,23618.319,28982612.2
Avg. Pooling14,5885,20718.319,20481912.2
FEATHER14,6705,20118.219,71466312.2
VisionSelector15,0925,23618.319,23882612.2
Random14,6865,23618.619,42382612.2
DTM14,9245,23618.619,24582612.2
LIS-DTM15,0295,23618.619,26882612.2

Against no deletion, every 80% method decodes 9–12% faster on Gemma 3 4B and 24–30% faster on T5Gemma 2 at batch 256; the gain grows with the batch as the KV cache becomes a larger share of each decoding step (batch 64: +2 to +5% and +9 to +11%; batch 16: within 4%). Peak memory falls from 26.9 to 18.2–18.6 GB on Gemma 3 4B; on T5Gemma 2 it is 12.2 GB for every method, no deletion included, because at batch 256 the peak comes while the images are encoded, before any token is deleted. At batch 16, 64 and 256, every method is within 3.3% of VisionSelector’s throughput (DTM within 1.1%), about the run-to-run variation.

Long inputs. With 16 images as input (about 4,200 prompt tokens), deletion is also effective at batch 1. On Gemma 3 4B, every 80% method shortens TTFT by 11–14% (DTM 448 to 387 ms; VisionSelector 385 ms): the vision encoder takes 306 ms with or without deletion, and DTM shortens the rest of the first-token time, mostly the language model’s prefill, from 142 to 80 ms (−44%). At batch 8 (SDPA attention), DTM shortens Gemma 3 4B's TTFT by 30–37% and its decoding step by 12–13%, and cuts the KV cache by 37–38% and peak memory by 19–21%. FastV's peak memory instead rises by 32–33%, because it materializes attention weights at its scoring layer. On T5Gemma 2, where the vision encoder is 73% of TTFT at batch 1, DTM shortens TTFT by 13–17% and decoding by 35–36% at batch 8, and cuts the KV cache by 77–78%. Note that both models use sliding-window attention in their first five layers.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.