acceptodds
Under review as a conference paper at ICLR 2027

Temporal Aware Pruning for Efficient Video Generation

Abstract

Video generative models have recently enabled high-quality video generation with Diffusion Transformer (DiT) architectures, but remain computationally intensive because generation requires attention computation over long spatiotemporal sequences. Token pruning has proven effective for ViTs and VLMs. However, most prior pruning methods are attention-based and operate per frame, failing to ensure the vital temporal coherence across frames in video generation tasks. In practice, naively adopting attention-only pruning causes noticeable degradation due to worsened background consistency, flickering, and reduced image quality. To address this, we propose TAPE, a training-free Temporal Aware Pruning for Efficient video generation. TAPE (i) applies temporal smoothing to align token importance across adjacent frames and suppress selection jitter; (ii) performs token reselection in selected layers to align token pruning with layers' diverse semantic focus and avoid error accumulation in specific areas; and (iii) adopts a timestep-level budget scheduling that prunes aggressively at early noisy steps and relaxes pruning during fidelity-critical refinement. The experimental results show that TAPE delivers significant speedups while preserving high visual fidelity, outperforming prior token reduction approaches.

Primary area:
generative models
Keywords:
Token Pruning, Video Generation, Efficiency

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mTK2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TAPE, a training-free token-pruning framework for accelerating video diffusion transformers. Rather than permanently removing tokens, it skips computation for selected tokens while retaining their representations for reconstruction. The method combines temporal averaging of attention-based importance scores, periodic token reselection across layers, and a denoising schedule that gradually reduces pruning. Experiments on HunyuanVideo and Wan2.1 compare the speed–quality trade-off against EViT, ToMe, and STA-ViT using VBench, with additional FVD comparisons on HunyuanVideo.

Critical strengths

The reported low-pruning operating point is practically useful. At 20% token reduction, TAPE achieves a reported 1.3× speedup while changing the VBench total score from 83.24 to 83.16 on HunyuanVideo (Table 4), with a similar trend on Wan2.1 (Table 8). Obtaining this trade-off without retraining is an advantage for applying the method to existing checkpoints. The dimension-level results in Table 10 also support improvements in background consistency and temporal flickering relative to the evaluated pruning baselines.

Critical weaknesses
  1. The methodological contribution is incremental, and the evaluation does not sufficiently establish its significance for video-generation acceleration. The method consists of a two-frame average of importance scores, periodic re-ranking, and a manually specified decreasing pruning schedule (§4). These are reasonable design choices, but the demonstrated advance is limited to improving three token-reduction baselines. There is no comparison with generation-specific sparse-attention or caching approaches, nor with simply reducing denoising steps at matched inference time. For a method whose main contribution is a better speed–quality trade-off, these comparisons are important: outperforming the selected pruning adaptations does not establish that TAPE offers a competitive allocation of inference computation. The second-model evaluation is useful, but it repeats the same restricted comparison.
  2. The evidence does not adequately isolate or validate the central temporal-coherence mechanism. Equation (1) averages scores at the same spatial index in adjacent frames, without establishing correspondence between their contents. Under object motion, camera motion, or occlusion, changing token importance may be appropriate rather than undesirable selection jitter. The paper explains why explicit motion estimation would be costly, but this does not establish when fixed-coordinate smoothing preserves the right computation. Figure 5 provides an illustrative mask comparison, and Table 6 shows a smoothing-weight trade-off, but neither resolves this distinction. Moreover, Table 5 evaluates each component individually and all three together, but omits reselection plus timestep scheduling without smoothing. It therefore does not establish the additional benefit of temporal smoothing within the complete framework. This leaves an important gap between the paper’s central explanation and the evidence supporting it.
Questions for authors
  1. At a matched inference-time budget, how does TAPE compare with reducing the number of denoising steps and with a relevant generation-specific accelerator? Please report absolute inference times and clarify whether importance scoring, token selection, and reselection overhead are included in the reported speedups.
  2. What happens when temporal smoothing is disabled while retaining both token reselection and timestep scheduling? Please report the temporal-quality dimensions as well as the total score at comparable computational cost. Within the evaluated videos, is its benefit maintained for substantial object or camera motion, where identical spatial indices need not correspond to the same content?
Reviewer_taXrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

TAPE is a training-free token pruning method for video DiT models. The authors reproduce EViT-style attention-based pruning on HunyuanVideo and observe degraded background consistency, temporal flickering, and lower imaging quality. Taking this as the starting point, they add three designs: averaging token importance scores across adjacent frames, re-selecting the tokens that participate in computation at intermediate layers, and lowering the pruning ratio toward the end of denoising. Experiments on HunyuanVideo and Wan2.1-1.3B show that TAPE achieves a better speed-quality trade-off than the paper's adapted versions of EViT, ToMe, and STA-ViT. The main result is a 1.5x speedup, with the VBench total score dropping from 83.24 to 82.01 (Table 4).

Critical strengths

The method is simple and requires no additional training. The problem analysis and motivation in Section 3 are explained clearly. It is validated on two base models, and the ranking relative to EViT, ToMe, and STA-ViT is consistent across them, so the results are self-consistent within this comparison set.

Critical weaknesses

The experimental comparisons in this work are somewhat outdated and disconnected from the current state of the field. The baselines are only EViT (image classification), ToMe/ToMeSD, and STA-ViT (video recognition). None of the training-free acceleration methods designed for video DiTs are cited or compared, such as AsymRnR, Sparse VideoGen, Sliding Tile Attention, SpargeAttn, and TeaCache. Several of these works also optimize HunyuanVideo. For example, AsymRnR likewise combines token reduction with a schedule that varies across timesteps and blocks; on HunyuanVideo it reduces runtime from 913 s to 703 s (1.30x) while VBench rises from 81.57 to 82.37. Sparse VideoGen achieves roughly 2x end-to-end speedup at 720p.

The novelty is also fairly limited, and the method design lacks details. On one hand, the smoothing is applied only at the same spatial location, without motion compensation. This may be reasonable for static regions, but its impact on fast-moving regions is neither considered nor discussed. In addition, the paper does not adequately explain how attention scores are aggregated into token scores, which computations are actually skipped, or how the states of previously excluded tokens are updated when they are re-selected. On the other hand, re-selecting tokens every few layers is a standard dynamic selection practice, and timestep-dependent schedules already exist.

Questions for authors
  1. Could you explain specifically how the attention scores are obtained and how the token importance scores are computed? Please present this in mathematical form.
  2. Could you explain whether the skipped tokens still serve as keys and values for the retained tokens? When they are re-selected, how are their states updated?
  3. Could you report the end-to-end runtime in seconds for the baseline, EViT, and TAPE, broken down into DiT, text encoder, and VAE?
  4. Could you provide a comparison against AsymRnR, Sparse VideoGen, and TeaCache on HunyuanVideo at matched wall-clock speedup? A clear win, or a clear gain from stacking TAPE on top of these methods, would significantly change my assessment.
  5. Could you analyze how TAPE performs under fast motion? Is simply averaging the scores with the previous frame sufficient?
Reviewer_XCfDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies token pruning for efficient video generation with Diffusion Transformers, observing that attention based pruning selects tokens independently per frame and thereby degrades temporal coherence and imaging quality. It proposes TAPE, a training free framework combining temporal smoothing of importance scores across adjacent frames, periodic token reselection across layers, and a timestep level schedule that prunes aggressively at early denoising steps and relaxes later. On HunyuanVideo 13B, TAPE reports 1.3 to 1.8 times speedups with minor VBench degradation, outperforming EViT, ToMe, and STA-ViT across pruning rates.

Critical strengths
  1. The paper identifies a clear and practically relevant failure mode of existing token pruning in video generation. The preliminary study convincingly attributes the quality degradation to frame-wise incoherent pruning masks, with the drop concentrated in background consistency, temporal flickering, and imaging quality.
  2. Each of the three components corresponds to a diagnosed failure mode, and the overall design is lightweight and training free. Temporal smoothing stabilizes selection across frames, reselection prevents errors from accumulating in fixed regions, and the timestep schedule matches pruning strength to the denoising dynamics.
  3. The empirical evaluation is consistent and reasonably comprehensive. TAPE is compared against three representative token reduction methods at multiple reduction rates, and the component ablations together with the sensitivity studies in the supplement support the role of each design choice.
Critical weaknesses
  1. The three components are simple heuristics with close precedents in the pruning and diffusion acceleration literature, so the conceptual contribution is modest. The ablation in Table 5 also shows the full combination improving over the best single component by only about 0.25 VBench points, which weakens the complementarity claim in the absence of variance estimates.
  2. Temporal smoothing blends importance scores at identical spatial indices across frames, which implicitly assumes static correspondence and may blur token selection under camera or fast object motion. No evidence is provided on highly dynamic videos, and lightweight alignment alternatives are dismissed on cost grounds without empirical comparison.
  3. The evaluation is limited to one base model and VBench in the main text, with the second model and additional metrics deferred to the supplement. The speedup figures do not specify the measured pipeline scope, and no comparison or combination with other acceleration families such as caching or sparse attention is provided, which limits the practical significance of the reported 1.3 to 1.8 times gains.
Questions for authors
  1. The speedup figures in Tables 4 and 5 do not specify whether they cover DiT denoising only or the full pipeline including text encoding and VAE decoding. Could the measurement scope and wall clock protocol be clarified, ideally with a component level latency breakdown?
  2. Temporal smoothing blends scores at identical spatial indices, which assumes static correspondence across frames. How does TAPE behave on highly dynamic content, for example a VBench dynamic degree breakdown or qualitative results on fast motion prompts?
  3. In Table 5 the full model exceeds the best single component by about 0.25 VBench points and no variance estimates are given. Are these differences stable across random seeds or prompt subsets?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.