acceptodds
Under review as a conference paper at ICLR 2027

PACE: Prune Aggressively and Commit Early for Diffusion VLM Inference Acceleration

Abstract

Diffusion vision-language models (DVLMs) have emerged as an alternative to autoregressive VLMs for their capacity for parallel decoding. In practice, however, each denoising step runs full bidirectional attention over the entire sequence, and the resulting inference cost scales as the product of the step count and the per-step cost of a full-sequence pass. Since visual tokens make up most of the sequence, compressing them has become an active direction. We show that the step count is a design factor as important as compression for DVLM inference acceleration, and that the two are not separable. Compression is applied after the first denoising step, which commits part of the answer on the full visual context, and the step count determines the size of that part. Under the default schedule, the core content of the answer is determined on the pruned context, where aggressive compression loses its accuracy. With fewer steps, more of the answer is committed before pruning and less of it is exposed to the loss from compression, so penalty shrinks. We formalize this regularity as a monotone comparative statics result. Based on these observations, we propose PACE, a training-free inference acceleration framework for DVLMs that scores visual tokens from the first forward pass and selects the step count jointly with the retention ratio under a task-level rule. Reducing the step count or pruning alone degrades accuracy, whereas combining them removes most of the penalty, and the temporal lever improves existing DVLM token pruning methods as well. PACE achieves 97.05% mean relative performance across nine benchmarks, while reducing generation FLOPs to 13.6% of vanilla at the 10% token budget.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deep learning, diffusion vision-language model, inference acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_6DkPOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper aims to accelerate inference in diffusion vision language models by jointly reducing the number of visual tokens and denoising iterations. It is motivated by the observation that the number of visual tokens are much bigger than that of prompt tokens and when visual context is compressed, using fewer generation steps can help preserve model accuracy. Based on these observations, this paper proposes PACE, which combines step-count adjustments with token pruning. PACE retains visual tokens that receive high attention scores from the prompt tokens and from responses tokens that are unlikely to become filler tokens. The remaining visual tokens are pruned, while if there is a remaining token that has sufficiently similar direction with the pruned one, it will be merged to the remaining token (spatial lever). PACE uses a different number of generation steps depending on the task. Experiments are conducted on benchmarks with output lengths of 16 tokens or fewer, and the results show that PACE preserves accuracy better than the baselines, reducing computational cost.

Critical strengths
  • PACE preserves model accuracy better than other strong baselines even under aggressive token pruning, while also achieving fast inference.
Critical weaknesses
  • Non-consistent setups for experiments and the results are only limited to short responses (16 or fewer).
    • There are only r=0.75 results for LaViDa 8B model.
    • No experiment with longer response tasks.
  • Incomplete related work discussions.
    • In terms of token pruning, the only difference between DVLMs and DLLMs is that visual tokens are concatenated in the front. State of the art DLLM inference acceleration methods should be considered as related works.
    • accuracy score might improve with semi autoregressive decoding on K=16.
    • when multiple tokens are generated per step, confidence aware parallel decoding (Wu et al. ICLR 2026) should be considered.
    • Token pruning methods like Elastic-Cache (Nguyen-Tri et al. ICLR 2026) or DyLLM (Lee et al. ICML 2026) should also be studied.
Questions for authors
  • The accuracy with lower K might be because there are less steps to be affected by the error caused by token pruning. Does PACE also work for tasks that require hundreds of output tokens?
  • Is the trend shown in Figure 2 also valid in longer response generations?
  • How could be the latency result of PACE-S and PACE-FULL same on generation tasks with r=0.5?
  • Could you provide the accuracy results for LLaDA-V 8B with r=0.75 and LaViDa 8B with r = 0.5, 0.25, and 0.1?
  • More breakdown of PACE's acceleration would be helpful to understand the results. For example, the lowest TFLOPs options were not always the fastest. It can happen because some operations are memory bounded, not computation bounded.
  • Is PACE's temporal lever more efficient than confidence aware parallel decoding or other related methods that reduces the number of generation steps?
  • Is PACE's spatial lever more efficient than DyLLM's salient token selection setting the cosine similarity threshold τ\tau differently between text tokens and visual tokens? How about Elastic-Cache?
Reviewer_uorrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PACE to study the interaction between visual token compression and denosing steps in diffusion vision-language models. It's a training-free framework that scores visual tokens from the first full-context forward pass, and prunes or selectively merges visual tokens for subsequent passes. The central observation is that fewer steps can reduce pruning-induced accuracy loss by committing more answer positions before compression. The paper provides a conditional formalization of this interaction and works on LLaVA-V-8B, with additional spatial-pruning results on LaViDa-8B.

Critical strengths

The interaction between pruning and denosing step reduction is an useful emprical finding. At 10% visual rention, Table 3 shows that reducing the step count from 16 to 2 improves ChartQA accuracy from 57.52 to 73.24. This suggests that compression and decoding schedules should be tuned jointly. The corresponding compute and latency measurements provide evidence of substantial savings within the evaluated uncached, short-output setting.

Critical weaknesses
  1. The increment over existing pruning methods is limited. PACE retains RedVTP's first-pass attention-based selection framework and introduces content weighting, prompt attention and gated merging. The contribution of these additions is small under the reduced-step configuration used by PACE-Full. In table 3, with 10% retention and two denoising steps, RedVTP and PACE obtain 72.96 and 73.24 on ChartQA, repectively, and 51.26 and 51.08 on GQA, at nearly identical FLOPs. On multiple-choice tasks, PACE-full uses a single full-context pass without pruning. These results indicate that most of the accuracy recovery follows from the decoding schedule, while the additional selection components contribute relatively little. The interaction analysis is informative, but the methodological advance remains modest.
  2. The evaluation doest not establish the acceleration benefit under cached decoding or longer generation. All reported latency measurements use uncached decoding, the additional speedup over a cached baseline remains unmeasured. Furthermore, all evaluated outputs are limited to at most 32 tokens. This directly constrains the evidence for the proposed mechanism, whose effectiveness depends on commiting a substantial fraction of the answer during the initial full-context pass. The joint pruning-scheduling result is also confined to LLaDA-V; LaViDa is evaluated with spatial pruning alone.
Questions for authors
  1. What accuracy and latency differences remain between PACE and RedVTP when both the denosing schedule and the visual token budget schedule are matched across all four generation benchmarks, including the use of more visual tokens in early steps and fewer in later steps on DocVQA and InfoVQA?
  2. What is the additional end-to-end speedup of PACE over Fast-dLLM caching alone at comparable accuracy? please include a cached baseline with a comparably tuned denosing schedule.
  3. Does the improvement persist for outputs substantially longer than 32 tokens? A document transcription or table parsing task would directly test this dependence. OCRBench/OCRBench v2 results would additionally help assess the effect of aggressive pruning on fine-grained visual recognition.
Reviewer_ypcXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In this paper, the authors study how to speed up diffusion-based vision-language models (DVLMs). A common strategy in the literature for increasing efficiency is to drop most image tokens after the first step. The authors observe that the first step still sees the full image and already fixes part of the answers. The number of steps then decides how large this part is. Using the default number of the steps, the first step fixes almost none of the answer, so the whole answer depends on the reduced image leading to lower accuracy. On the other hand, using very few steps leading to a fixed answer by having access to the the full image. Based on these observations, the authors propose PACE (Prune Aggressively and Commit Early) which has two parts. The first one chooses which image tokens to keep by looking at the attention signals of the first diffusion step. The second one sets the total numbers of steps depending on the answer type (i.e., one step and no pruning for multiple choce, two steps for short answers and eight steps for long answers).

The experiments show that using LLaDA-V-8B and keeping 10% of the total image tokens, PACE reaches 93.5% of the original accuracy on four tasks at 13.6% of the compute. Moreover, the paper shows improvements when applying the same rule for the number of steps to existing pruning methods .

Critical strengths

[S1] The paper shows a particularly surprising result. Using fewer diffusion steps on a DVLM when most images tokens are dropped increases performance increase. This is something that of course is not happening without any image token pruning, where with fewer steps we expect a lower accuracy.

[S2] The proposed approach is simple and easy to use.

[S3] The paper shows that this trick can also help existing pruning methods.

Critical weaknesses

[W1] It seems that most of the gain comes from using fewer steps, which is something that any existing method could do. RedVTP nearly matches the proposed PACE when the same number of steps applied at the same compute. This means that the proposed way of selecting image tokens adds very little. This can be observed by comparing the random selection of tokens with PACE on ChartQA (only 1.1p difference). I would expect a more detailed analysis and discussion in the paper about this and further explanation about the selection of image tokens which is one of the core contributions of the paper.

[W2] The experiments were done only on benchmarks with short answers (i.e., at most 32 tokens). The authors clearly state that they expect the benefit of fewer steps to shrink in longer answers (e.g., chain-of-thought reasoning, longer image captioning, etc). It would be important to see this in practice in the paper.

[W3] The core idea of using fewer steps only works with one model (LLaDA-V-8B). The paper presents experiments with another model (LaViDa-8B) but this idea does not help there and only the image token pruning part of the proposed method is used there.

[W4] Pruning image tokens will not save compute when KV caching is applied. The paper notes that when reusing computation across steps with KV-caching leads to no dependency between compute and the numbers of image tokens kept. The results in the paper show that the cache with two steps and no pruning matches the proposed PACE method, so when we are in this setting the proposed method is boiled down to selecting only the number of steps.

[W5] Deciding the number of steps requires predicting the question type (multiple-choice, short or long answer), which was tuned on test data. It seems that this task-level rule (L. 334-339) was chosen using samples from ChartQA and DocVQA, which are also part of the evaluation. It's unclear how this can be applied in real-word since one needs to know each question's type beforehand.

Questions for authors

[Q1] What happens when you give existing methods the same number of steps as PACE on all tasks (i.e., 1-step for multiple-choice, etc)? In this setup which is more fair, how much does the proposed visual token pruning add over other methods and RedVTP and over choosing tokens at random?

[Q2] Could you add at least one task with long answers, such as detailed long image descriptions or step-by-step reasoning?

[Q3] Using a KV caching, what does the proposed method add over simply using the cache with fewer steps and no pruning? Could you report the accuracy for both including compute and latency?

[Q4] How would you choose the number of steps when the question type is not known in advance?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.