acceptodds
Under review as a conference paper at ICLR 2027

Steering Video Diffusion Transformers with Massive Activations

Abstract

In this work, we study the role of Massive Activations (MAs), which are rare, high-magnitude spikes confined to a few fixed hidden dimensions in video diffusion transformers (DiTs). We uncover a structured positional hierarchy: MA magnitudes peak at first-frame tokens and recur at the spatial boundary tokens of latent frames, with this pattern being most pronounced during early denoising. We trace this organization to an encoding asymmetry of the video VAEs, whose causal temporal padding and zero spatial padding cause the first latent frame and frame borders to carry reduced content load. Elevated MAs consistently align with these lower-content structural positions. To understand their function, we analyze intermediate representations and find that MAs act as implicit rescalers of residual computation: enlarging MAs suppresses the corresponding self-attention and feed-forward updates, while erasing them amplifies these updates. Together, these observations suggest that MAs serve as a token-level rescaler of residual computation, which video DiTs deploy unevenly, placing the strongest damping at the encoding-asymmetric structural positions. Motivated by this native rescaling behavior, we propose Structured Activation Steering (STAS), a training-free technique that steers MAs at the observed structural positions toward a scaled, model-derived reference during early denoising. STAS requires no additional forward passes and consistently improves video quality and temporal coherence across text-to-video models with negligible overhead.

Primary area:
generative models
Keywords:
Video Diffusion Transformers, Massive Activations, Training-free

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_h7bhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper identifies a structured magnitude hierarchy of MAs unique to video DiTs. It further proposes STAS, a training-free method that specifies what to steer, where to steer, how to steer, and when to intervene to improve T2V video quality and temporal coherence. The discovery of position-dependent MAs, the proposed STAS intervention, and the reported quality improvement make it a useful contribution to inference-time control of generative models.

Critical strengths

The paper identifies a structured magnitude hierarchy of MAs unique to video DiTs. Proposed a lightweight training-free, self-guidance technique that steers MAs without rerunning the backbone model and improves video quality and temporal coherence. It modifies activations inline during the standard forward pass, yielding minimum computational overhead. Comprehensive performance analysis comparing proposed scheme with 4 popular baseline models conducted.

Critical weaknesses

The paper provides a math definition of STAS. It can be significantly strengthened if the steering coefficient is a quantity derived from an optimization objective. Certain method, when improving temporal consistency, may suppress valid motion and reduce motion dynamics. The authors explicitly pointed out in the paper that there’s a reduction in dynamic degree score. Since “temporal smoothing" is a known 'limitation', some discussions and justifications on the trade-off is encouraged.

Questions for authors

Would some equally strong but spatially or temporally randomized intervention produce similar gains and why? Is the proposed solution sensitive to different VAE architectures, different temporal attention mechanisms, ……? Does it work for I2V as well? What happens in the case of long video generation? How much of the improvement comes from the underlying principle and how much depends on model specific tuning?

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We thank the reviewer for the constructive feedback. Below we address each point and all revisions are highlighted in blue in the revised version.

1. Optimization-derived steering coefficient

On optimization-derived coefficients. Our goal is to leverage the MA findings to improve generation quality with negligible overhead. Deriving the coefficient from an optimization objective would typically require per-step gradients or extra forward passes, increasing inference cost. We agree it is a promising direction and leave it for our future work.

2. Trade-off between temporal smoothing and motion

Dynamic degree drop discussions. We agree that STAS may slightly attenuate subtle motion, and we provide more discussion of this trade-off in Appendix I of the revised paper. However, a Dynamic Degree drop does not necessarily indicate visible motion loss: the metric is threshold-based and can count artifacts such as flickering or object reappearance as motion. To quantify this, we run STAS on Dynamic-bench [1], a benchmark of high-motion prompts, and conduct a blind user study on the flip cases where Dynamic Degree changes from True (baseline) to False (STAS): 12 raters judged 33 pairs from Wan2.2-5B, Wan2.1-14B, and MiniMax-H3 (396 judgements). For each pair, raters answered two questions in order. Q1: Which video has more movement overall? Q2: Which video shows more natural and physically plausible motion?

Q1: Which video moves more?

STASAbout the sameBaseline
49 (12.4%)242 (61.1%)105 (26.5%)

Q2: Which video is more plausible? (grouped by the Q1 answer)

Perceived motionSTASTieBaseline
About the same4517522
STAS moves more4126
Baseline moves more242259

Despite the Dynamic Degree flips, raters perceive similar motion in most judgements (242/396), and in these cases STAS is judged more plausible about twice as often as the baseline (45 vs. 22). Visible motion attenuation occurs only in a minority of cases (105/396), where the baseline is preferred; we have added this discussion to the Limitations.

3. Randomized interventions

On randomized interventions. We include this control in Table 5, replacing boundary tokens with an equal number of random interior tokens under otherwise identical settings (Wan2.1-1.3B):

Target tokensSub. Consist.BG. Consist.AestheticImagingQuality Score
Vanilla94.3695.6863.2867.2182.01
First + random interior94.7295.7363.2668.0882.09
First + boundary (STAS)94.7495.8463.4768.0582.24

Random interior tokens yield only a marginal gain, well below boundary tokens. This is because the model itself assigns larger MAs to the first-frame and boundary tokens, which carry less information under the same token budget, and applies stronger rescaling of residual writing to them. STAS is inspired by this learned behavior and reinforces it, leading to better results.

4. Sensitivity to VAEs and temporal attention

Sensitivity to VAEs and temporal attention. STAS is not tied to a specific VAE or attention design. Across different backbones with diverse VAE architectures (Wan2.1, Wan2.2, CogVideoX, LTX-2, MiniMax-H3), elevated MAs consistently appear at single-frame latents and spatial boundaries. For example, while other VAEs encode only the first latent from a single frame, where MAs peak, MiniMax-H3 encodes every 17 frames into 5 latents (1-4-4-4-4), and MAs peak periodically at each single-frame latent. All use full spatiotemporal attention with 3D RoPE, and STAS improves performance on all of them.

AuthorsOpenReview ↗

Rebuttal Part 2

5. Image-to-video generation

Generalization to I2V. We apply STAS to Wan2.2-5B I2V on VBench-I2V, using the same hyperparameters as T2V:

MethodSub.BG.Smooth.Dyn.Aes.Img.I2V Sub.I2V BG.QualityI2VTotal
Wan2.2-5B96.0995.4297.6145.3559.3570.8397.5898.1879.0597.3588.20
+STAS96.2695.4997.6744.9059.4771.1097.7398.2279.1697.4788.32

STAS consistently improves temporal consistency and visual quality in the I2V setting.

6. Long video generation

Long video generation. We focus on standard short-video generation. Many long-video methods build on short-video models and convert them into causal autoregressive generators, which changes the token structure. We leave the effect of MAs in this setting to our future work.

7. Principle vs. model-specific tuning

Principle vs. model-specific tuning. The improvement mainly stems from the principle, which fixes the core design: MA dimensions are identified by a single profiling pass, target tokens follow the VAE structure (single-frame latents and spatial boundaries), and steering is applied during high-noise steps. Model-specific tuning only adjusts the magnitude and depth, following simple heuristics: steering at 1/4–1/3 of the network depth, with smaller α\alpha for larger models.

[1] Sangwon Jang, Taekyung Ki, Jaehyeong Jo, Saining Xie, Jaehong Yoon, and Sung Ju Hwang. Self-refining video sampling. In ICML, 2026

Thank the authors for taking the time to conduct the additional experiments and for addressing my concerns and questions. Appreciate your efforts. I have no additional concerns or questions.

Reviewer_wcG2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper identifies the phenomenon of "massive activations" in existing large-scale video models, explores its connection to VAE encoding, and proposes a training-free activation fine-tuning technique called "Structured Activation Steering." Overall, this is a well-motivated, thoroughly evaluated, and thought-provoking paper.

Critical strengths
  1. The paper provides a systematic analysis of the "structured positional hierarchy" phenomenon observed in massive activations across spatio-temporal token sequences. It includes thorough validation experiments to investigate the underlying causes, making the findings highly convincing.

  2. The paper proposes a training-free method that is effective across various baseline models while incurring minimal additional computational overhead.

  3. The approach demonstrates strong scalability; STAS can be integrated with other training-free methods to further enhance the quality of video generation.

Critical weaknesses

The method's hyperparameters appear somewhat complex and lack a universal selection criterion, likely relying on empirical tuning. It remains unclear whether applying this method to other open-source video models would require extensive trial and error.

The paper notes that, across all models, massive activation magnitudes peak at the tokens of the first latent frame. This is a relatively common phenomenon in video models, often referred to as a "sink," and many existing methods leverage it as an advantage to enhance long-sequence modeling capabilities. This empirical observation seems somewhat inconsistent with the conclusions drawn in this paper.

The authors provide a clear explanation of the origin of MA. However, they do not provide sufficient theoretical justification or empirical evidence to demonstrate why MA negatively affects video generation quality.

It would be valuable to evaluate the method on larger-scale model architectures, such as Wan2.1-14B or MiniMax-H3, to further verify its effectiveness.

Questions for authors

See Weaknesses.

1 comment

We thank the reviewer for the constructive feedback. Below we address each point and all revisions are highlighted in blue in the revised version.

1. Hyperparameter selection criterion

Hyperparameter selection. Applying STAS to a new model requires only lightweight tuning. Most design choices are fixed by the principle: MA dimensions are identified by a single profiling pass, target tokens follow the VAE structure (single-frame latents and spatial boundaries), and steering is applied during high-noise steps. The remaining hyperparameters follow simple heuristics that narrow the search: the layer is chosen at 1/4–1/3 of the network depth by visual inspection on a small subset, and α\alpha decreases with model size.

2. Relation to first-frame sinks

Sinks in attention vs. MAs in hidden states. We agree that first-frame attention sinks are common in video models (e.g., causal autoregressive long video generation). However, they are distinct from the MAs studied here.

  • Location: Attention sinks arise in attention weights, where the first frame receives more attention. In contrast, MAs arise in a few fixed hidden-state dimensions, taking larger values on less informative tokens, i.e., single-frame latents (the first latent frame in Wan, CogVideoX, and LTX-2; every single-frame latent in MiniMax-H3) and boundary tokens.
  • Functionality: Attention sinks let subsequent frames reference the first frame to maintain consistency. MAs rescale the token's own residual writing, damping updates at these less informative positions.

The reduced information on these tokens originates from the VAEs: temporal and spatial padding make them inherently less informative, and the DiT learns during training to place higher MAs on them. This placement mirrors LLMs, where MAs appear at weakly semantic starting and delimiter tokens [1], and ViTs, where high-norm tokens appear in low-informative background patches [2]. Therefore, the observation on attention sinks does not contradict our conclusions, as the two concern distinct mechanisms.

3. Evidence on how MAs affect generation quality

Why MAs matter for generation quality. We do not claim that MAs negatively affect generation. Rather, insufficient MAs would degrade quality. To verify this, we zero out the MA dimension at all visual tokens on Wan2.1-1.3B:

MethodSub. Consist.BG. Consist.AestheticImagingQuality Score
Vanilla94.3695.6863.2867.2182.01
Zero-out MAs94.2895.6061.5666.2581.13
STAS94.7495.8463.4768.0582.24

Removing MAs clearly degrades quality, especially aesthetic and imaging quality, whereas strengthening them at these positions improves it.

4. Larger-scale models

Results on Wan2.1-14B and MiniMax-H3. The MA hierarchy persists at this scale (Figure 1(d) for MiniMax-H3; Appendix A.2 for Wan2.1-14B). We further evaluate STAS on both models on Dynamic-bench, and STAS improves the quality score on both:

ModelMethodSub.BG.FlickerSmooth.DynamicAes.Img.Quality
Wan2.1-14BVanilla89.3993.1496.8098.0476.1150.2561.5779.02
+STAS89.5193.2496.8298.0777.2250.1561.4879.15
MiniMax-H3Vanilla88.2193.6997.4098.9288.3350.4062.1280.69
+STAS88.3493.7697.4898.9587.5050.1862.7780.77

STAS consistently improves temporal consistency and overall quality on both models. Full configurations are listed in Tables 7 and 8 of the revised paper.

[1] Mingjie Sun, Xinlei Chen, J. Zico Kolter, and Zhuang Liu. Massive activations in large language models. In COLM, 2024.

[2] Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision Transformers Need Registers. In ICLR 2024.

Reviewer_P36UOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies Massive Activations (MAs) in video diffusion transformers (DiTs). The authors observe an MA hierarchy that appears across several video DiT backbones and tie it to the first latent frame, explaining it through a first-frame content-load argument. Building on this observation, they propose STAS, a training-free activation-steering method that replaces activations at fixed steering positions using a maximum-based reference (Eq. (2)). STAS is evaluated on four backbones, including Wan2.1-1.3B and Wan2.2-5B, and is reported to reduce artifacts in the generated videos.

Critical strengths
  1. The cross-model observations are interesting. The MA phenomenon is examined on several video DiT backbones rather than a single model, which gives the analysis a broader basis.
  2. The intervention is low-cost. STAS is training-free and needs no additional denoiser evaluations, so it can be applied directly to existing models.
Critical weaknesses
  1. Generality of the first-frame explanation (main concern). The explanation of the MA hierarchy, and the fixed steering positions derived from it, depend on the first latent frame representing a single physical frame. In the non-overlapping 17-frame-to-5-latent encoding setup described for MiniMax H3, this is not established. As a result, the one-quarter content-load ratio and the first-frame MA hierarchy cannot be assumed without verification. Please examine this setting and clarify how general the proposed explanation and the fixed steering positions are.

  2. The gains may be a side effect of reduced motion. Dynamic Degree decreases on all four backbones in Table 10. It is therefore unclear whether the artifact reduction reflects a genuine benefit of STAS or is merely a consequence of suppressed motion. Please separate the two effects and add an evaluation on high-motion cases.

  3. Experimental coverage is limited.

    • The Wan experiments cover only Wan2.1-1.3B and Wan2.2-5B (Sec. 6, Table 1), without their 14B counterparts. Please verify whether the MA hierarchy and the STAS gains persist at this scale, and whether the steering configuration needs to be retuned.
    • The maximum-based reference in Eq. (2) may vary with token count or be affected by outliers. Please evaluate STAS at different resolutions and video lengths, and compare against a high-quantile reference to assess sensitivity.
  4. Minor issues.

    • The computational-cost statement in Sec. 6 should be made precise. STAS requires no additional denoiser evaluations, but computing maxima and replacing activations still incur a small overhead. Please replace "no additional forward computation" with "no additional denoiser evaluations."
Questions for authors

Please see the weaknesses above.

3 comments
AuthorsOpenReview ↗

Rebuttal Part 1

We thank the reviewer for the constructive feedback. Below we address each point and all revisions are highlighted in blue in the revised version.

1. Generality of the first-frame explanation

Generality beyond the first latent frame (MiniMax-H3). We thank the reviewer for raising this important point and have examined MiniMax-H3, which encodes the video in non-overlapping 17-frame chunks, each mapped to 5 latents with a 1-4-4-4-4 frame mapping. In the revised paper, Figure 1(d) (page 2) shows a two-level periodicity in its MAs: at the frame level, MAs are higher at every single-frame latent, recurring every 5 latents. Within each latent frame, MAs are further elevated at the boundary tokens. Figure 12 (page 19 in the revised paper) quantifies both levels: single-frame latents reach ~105–107.5% of baseline versus ~98–102.5% for 4-frame latents, and the spatial MA map confirms the elevation at frame borders.

This confirms that MAs follow the content load per latent rather than just the first position. In Wan, CogVideoX, and LTX-2, the two coincide. We have accordingly revised "the first latent frame" to "single-frame latents" throughout the paper, and generalized the steering positions to all single-frame latents plus boundary tokens. These positions are thus determined by the VAE structure without additional search.

2. Gains as a side effect of reduced motion

Disentangling quality gains from motion reduction. To evaluate high-motion cases, we run STAS on Dynamic-bench [1], a benchmark of high-motion prompts, and collect the flip cases where Dynamic Degree changes from True (baseline) to False (STAS). On these cases, we conducted a blind user study: 12 raters judged 33 pairs from Wan2.2-5B, Wan2.1-14B, and MiniMax-H3 (396 judgements, details in Appendix I). For each pair, raters answered two questions in order. Q1: Which video has more movement overall? Q2: Which video shows more natural and physically plausible motion?

Q1: Which video moves more?

STASAbout the sameBaseline
49 (12.4%)242 (61.1%)105 (26.5%)

Q2: Which video is more plausible? (grouped by the Q1 answer)

Perceived motionSTASTieBaseline
About the same4517522
STAS moves more4126
Baseline moves more242259

Despite the Dynamic Degree flips, raters perceive similar motion in most judgements (242/396). In these motion-matched cases, STAS is judged more plausible about twice as often as the baseline (45 vs. 22), so the gain does not rely on reduced motion. Plausibility drops only where motion is visibly suppressed.

[1] Sangwon Jang, Taekyung Ki, Jaehyeong Jo, Saining Xie, Jaehong Yoon, and Sung Ju Hwang. Self-refining video sampling. In ICML, 2026

AuthorsOpenReview ↗

Rebuttal Part 2

3. Limited experimental coverage

Results on Wan2.1-14B and MiniMax-H3. We evaluate STAS on both models on Dynamic-bench. The MA hierarchy persists at this scale, and STAS improves the quality score on both:

ModelMethodSub.BG.FlickerSmooth.DynamicAes.Img.Quality
Wan2.1-14BVanilla89.3993.1496.8098.0476.1150.2561.5779.02
+STAS89.5193.2496.8298.0777.2250.1561.4879.15
MiniMax-H3Vanilla88.2193.6997.4098.9288.3350.4062.1280.69
+STAS88.3493.7697.4898.9587.5050.1862.7780.77

STAS consistently improves temporal consistency and overall quality on both models. Since architectures differ across models, the steering configuration is re-profiled and tuned for each. Full configurations are listed in Tables 7 and 8 of the revised paper.

Sensitivity of the max reference. The max reference is robust to both token count and outliers.

Token count (resolutions and video lengths). STAS remains effective across resolutions and video lengths. On Wan2.2-5B, the main paper uses 81×704×1280 (Table 1). We further evaluate 81×480×832 (lower resolution) and 49×480×832 (lower resolution and shorter length) with the same hyperparameters:

Resolution (tokens)MethodSub.BG.FlickerSmooth.DynamicAes.Img.Quality
81×480×832 (8,190)Vanilla93.9494.8796.8398.1848.1359.2766.5680.30
+STAS94.0394.8696.8898.1847.8959.5967.0780.44
49×480×832 (5,070)Vanilla95.2395.4696.9198.2850.5259.5367.7781.14
+STAS95.2995.3996.9798.3249.5659.9368.1981.24

Outliers (high-quantile reference). The max is not outlier-driven. Across 100 prompts on Wan2.2-5B, we compare the max of the absolute MA activations over video tokens (used in Eq. 2) with their 99th percentile (q99). At each denoising step where STAS is applied, the max fluctuates across prompts as little as q99, and their ratio stays nearly fixed (mean ± std across prompts):

Stepmaxq99max / q99
074.4 ± 4.064.7 ± 2.91.15
361.1 ± 3.750.0 ± 2.81.22
656.1 ± 3.344.9 ± 2.81.25
952.9 ± 2.841.6 ± 2.61.27

Hence, a high quantile reference is approximately equivalent to rescaling α\alpha in Eq.(3). We further directly replace the max with q99 in Eq. (2), keeping all other settings fixed (Wan2.2-5B, 49×480×832, 5 seeds):

ReferenceSub.BG.FlickerSmooth.DynamicAes.Img.Quality
Vanilla95.2395.4696.9198.2850.5259.5367.7781.14
max (STAS)95.2995.3996.9798.3249.5659.9368.1981.24
q9995.2395.4296.9598.3149.8860.0268.0881.24

The q99 reference performs comparably to the max reference across dimensions, confirming that the max-based reference is not affected by outliers.

4. Minor issues

Computational cost. Revised to "no additional denoiser evaluations" throughout the paper.

Thank you for the additional experiments. The periodic MA pattern in MiniMax-H3 substantially addresses my main concern about the first-frame assumption, and the larger-model and reference ablations also alleviate my coverage and robustness concerns. The additional results provide evidence of positive gains from STAS. I therefore raise my rating to Weak Accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.