acceptodds
Withdrawn from ICLR 2027

AdaVTP: Prediction Guided Adaptive Visual Token Pruning for Efficient Diffusion MLLM Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion multimodal large language models (dMLLMs) enable strong multimodal understanding through iterative parallel decoding and bidirectional context modeling, but repeatedly process all visual tokens at every denoising step, causing substantial redundant computation. Existing methods mitigate this cost by pruning visual tokens using relevance estimated from early prediction steps, but apply a fixed retention ratio to every input. This overlooks varying visual evidence requirements: simple questions may require only a few local cues, whereas complex ones require information distributed across multiple regions. To address this, we propose AdaVTP, a training-free framework for adaptive visual token pruning in dMLLMs. After the first denoising step, AdaVTP uses prediction confidence to adaptively balance visual attention from the question and masked response positions, yielding task-conditioned relevance scores. Guided by these scores, AdaVTP first retains a sample-specific candidate set covering a prescribed fraction of the cumulative relevance, and then progressively selects complementary tokens by relevance-weighted semantic coverage gain until the target coverage is reached. The resulting subset is reused for all remaining denoising steps, avoiding repeated pruning overhead. Experiments on three dMLLMs and 13 image/video benchmarks show consistent gains over existing training-free pruning methods under matched dataset level token budgets. With only 10% of visual tokens retained overall, AdaVTP preserves 92.1% of the full token performance on LLaDA-V, exceeding RedVTP by 4.6 points. At comparable task performance, it delivers 1.39–1.92 end-to-end inference speedups over RedVTP.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
diffusion multimodal large language models, adaptive visual token pruning, training-free inference acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_voPpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaVTP, a training-free adaptive visual token pruning framework for efficient inference in diffusion-based multimodal large language models (dMLLMs). After the first denoising step, AdaVTP estimates visual token relevance using prediction confidence and attention from question tokens and masked response positions. It then combines cumulative relevance screening with semantic coverage-based selection to determine a sample-specific visual token budget.

The authors evaluate AdaVTP on three backbones—LLaDA-V, LaViDA-LLaDA, and LaViDA-Dream—across 13 image and video benchmarks. They report improved task performance and inference efficiency compared with existing visual token pruning methods.

Critical strengths
  1. A practical training-free approach to visual token pruning.

Adaptively adjusting the visual token budget without additional model training is a practical approach to improving inference efficiency. In particular, performing pruning only once and reusing the selected visual tokens throughout subsequent denoising steps is well suited to reducing repeated visual token processing in dMLLMs.

  1. Promising performance and efficiency improvements over existing methods.

AdaVTP demonstrates improvements over directly relevant diffusion MLLM pruning methods, including RedVTP and D³ToM, across multiple benchmarks. Its ability to preserve relatively strong performance under aggressive visual token compression, together with measured end-to-end latency reductions, is encouraging.

Critical weaknesses
  1. The limited diffusion decoding configurations in the main experiments make it difficult to assess the general effectiveness of the proposed method.

The central motivation of this paper is to reduce the computational overhead caused by repeatedly processing visual tokens throughout the denoising process of dMLLMs. However, according to Table 7 in the appendix, most image understanding benchmarks use generation length, block length, and denoising steps of only 2/2/2 or 4/4/4.

Since AdaVTP performs pruning after the first denoising step, under the 2-step configuration, the pruned visual tokens are used for only one subsequent denoising step. It is questionable how representative such a setting is of general multi-step diffusion inference.

Although additional experiments with 16–32 denoising steps are provided on DocVQA, InfoVQA, and MathVerse, these experiments are restricted to LLaDA-V. More importantly, the paper lacks a systematic analysis of how performance and inference efficiency change when the number of denoising steps and block size are varied on the same backbone and benchmark.

Given that the effectiveness of visual token pruning can depend substantially on decoding configurations, the current results do not sufficiently establish whether AdaVTP provides consistent advantages across different dMLLM inference settings.

  1. The independent effectiveness and necessity of adaptive token allocation, the central contribution of the paper, are not sufficiently established.

AdaVTP combines confidence-guided relevance estimation, relevance mass screening, and semantic coverage selection to determine sample-specific visual token budgets. However, the independent improvements provided by these components are relatively modest compared with the overall performance gains.

For example, in Table 4, adaptive allocation improves performance over fixed allocation using identical relevance scoring and token selection by only 1.2 points on InfoVQA and 0.6 points on POPE. Although the appendix reports larger improvements under more aggressive compression, the paper does not directly demonstrate that the allocated token budgets accurately reflect the visual evidence requirements or question complexity of individual inputs.

Similarly, the improvement from confidence-guided relevance fusion over a simple combination of question and masked-response attention is relatively small. It therefore remains unclear whether the proposed adaptive mechanism genuinely captures input-specific visual complexity and allocates tokens accordingly.

Overall, the experiments demonstrate improvements from the complete framework, but provide insufficient evidence explaining why adaptive visual evidence allocation is necessary or how it fundamentally differs from existing relevance-based pruning approaches.

Questions for authors

1.Why do most of the main benchmarks use only 2–4 denoising steps? Can the authors compare AdaVTP and RedVTP under different denoising step counts and block sizes on the same backbone, reporting both task performance and inference efficiency?

2.Why are the longer-generation experiments restricted to LLaDA-V? Can the authors evaluate the proposed method on LaViDA-LLaDA and LaViDA-Dream under longer denoising trajectories?

3.How can the authors demonstrate that adaptive token allocation actually reflects question difficulty or the amount of visual evidence required by individual inputs? Can they analyze the relationship between the number of retained visual tokens and input characteristics such as visual complexity or question type?

4.What fundamental advantages does confidence-guided relevance fusion provide over simpler pruning strategies based on question attention or masked-response attention? How should the relatively modest performance improvements be interpreted?

Reviewer_KirsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaVTP, which selects the visual tokens to retain for each sample after the first denoising step of a diffusion multimodal model, using the question, answer predictions, and visual information. The selected tokens are reused in subsequent denoising steps to reduce inference cost.

Critical strengths
  1. The method allows different samples to retain different numbers of tokens, offering greater flexibility than a fixed retention ratio.
  2. Experiments include multiple models, component ablations, and actual latency measurements, providing some support for the method’s effectiveness.
Critical weaknesses
  1. The coverage threshold can be adjusted for each dataset, leaving its performance on unseen data with a fixed threshold unclear. Moreover, matching the total number of tokens does not necessarily match computational cost.
  2. The practical overhead of attention extraction and token selection is insufficiently documented. Inconsistent generation-step settings in the appendix also affect reproducibility.
Questions for authors
  1. How does the method perform without adjusting the threshold for each test dataset? Does it retain its advantage under matched FLOPs or latency?
  2. How much time is spent on attention extraction and token selection, respectively? Do all methods use the same optimized backend?
  3. Do the two LaViDA models use two or four denoising steps? How does changing the number of denoising steps affect the speedup?
Reviewer_ry19OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes AdaVTP, a training-free, adaptive visual token pruning framework for diffusion multimodal LLMs. After the first denoising step, it builds confidence-guided relevance by fusing question-to-vision and masked-response-to-vision attentions, screens a candidate set by cumulative relevance mass, and then greedily selects tokens to meet a semantic-coverage target, yielding an adaptive retained set reused for all subsequent steps. Experiments on three diffusion backbones and 13 benchmarks show consistent gains over training-free baselines, preserving 92.1% of full-token accuracy at 10% retained tokens on LLaDA-V and delivering 1.39-1.92x end-to-end speedups at comparable task performance.

Critical strengths
  1. The paper identifies a limitation in existing dMLLM pruning methods that they use fixed retention ratios regardless of the visual evidence required by each input. To mitigate this, they introduce an adaptive query-dependent approach for token pruning in dMLLMs.
  2. The confidence-guided fusion of question and masked-response attention and retention of tokens based on semantic coverage is well-motivated.
  3. The proposed method improves accuracy over existing training-free token pruning methods with fixed retention ratios.

Questions:

  1. What is the selector overhead latency breakdown (attention aggregation, RMS sorting, SCS similarity computations), and how does it scale with context length?
  2. Did the authors compare against step-dependent token selection? How does it compare with static selection?

Overall Assessment: Weak Reject

Critical weaknesses
  1. The authors state that AdaVTP achieves the lowest computational cost and latency while retaining the highest task performance. However, the authors do not explain why AdaVTP achieves lower computational cost compared to other training-free methods at the same retention ratio. A detailed breakdown of the computational overhead for the proposed method will help the reader appreciate why this has lower latency.
  2. The authors present efficiency results for a single model and benchmark. They must present latency results for different models and for different tasks like long-form generation and video understanding.
  3. The reuse of a single subset for all subsequent steps may be suboptimal for tasks where visual evidence shifts during the denoising process, particularly for reasoning tasks. A comparison with step-dependent selection will justify the choice.
  4. A comparison with dynamic token pruning approaches [1,2] is missing.

[1] VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning, ICML 2026. [2] SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference, ICCV 2025.

Questions for authors
  1. What is the selector overhead latency breakdown (attention aggregation, RMS sorting, SCS similarity computations), and how does it scale with context length?
  2. Did the authors compare against step-dependent token selection? How does it compare with static selection?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.