acceptodds
Under review as a conference paper at ICLR 2027

Temporal-Attention Head Specialization During Video Diffusion Training

Abstract

Video diffusion transformers depend on temporal attention to coordinate information across frames, yet existing analyses of this mechanism examine only trained models. Such final-checkpoint analyses show what temporal structure exists but not when or where it forms during training, which is the information needed to monitor training and to target head-level interventions such as pruning. Population averages can also hide this structure, because the increase in a few heads and the decrease in most other heads offset each other in the mean. We therefore conduct a checkpoint-resolved census of every temporal-attention head across nine Open-Sora STDiT training runs spanning three model scales (306M to 1.03B parameters). Each head is scored by its cross-frame attention concentration (CFAC), the entropy-normalized concentration of its attention over the 16 frames. CFAC is 0 when a head spreads attention evenly across all frames and approaches 1 when it routes each frame's attention to a few specific frames. Heads are then selected under a preregistered change-point and effect-size rule. Per-head analysis shows that a small minority of heads, roughly 4-13% in full-grid runs, becomes strongly concentrated during training, while aggregate CFAC stays flat or decreases in every run. Across seeds, selected heads repeatedly appear in the first temporal block, but the specific heads selected within that block differ across seeds. In the 760M runs, selected heads show two main recurring frame-to-frame attention patterns. In the first, a self-frame diagonal, each frame attends mostly to itself. In the second, an adjacent-frame band, each frame attends mostly to its neighbors. These patterns recur across runs even when they emerge in different heads (i.e., at different layer and head indices). Our correlation and ablation experiments do not establish that these heads affect generated video quality. The census nevertheless identifies when and where temporal specialization forms during training. The same checkpoint-resolved, per-head analysis under fixed selection rules can provide this information for other factorized video diffusion transformers and, with adapted routing metrics, for joint spatio-temporal architectures.

Primary area:
interpretability and explainable AI
Keywords:
video diffusion transformers, temporal attention, head specialization, training dynamics, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_z5QiOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper conducts a checkpoint resolved census of every temporal attention head across nine Open Sora STDiT training runs at three scales from 306M to 1.03B parameters. Each head is scored by cross frame attention concentration, an entropy normalized concentration of attention over the 16 frames, and heads are selected under a preregistered change point and effect size rule. The reported findings are descriptive: roughly 4 to 13 percent of heads become strongly concentrated during training while aggregate CFAC stays flat or decreases, selected heads repeatedly appear in the first temporal block but differ across seeds, and two attention motifs recur in the 760M runs, a self frame diagonal and an adjacent frame band. The authors state that their correlation and ablation experiments do not establish that these heads affect generated video quality.

Critical strengths

The honesty of this paper deserves real credit. The abstract states plainly that the correlation and ablation experiments do not establish that the selected heads affect generated video quality, and the body text lives up to that standard. The selection rule is preregistered, the CFAC metric is simple and interpretable, and checkpoint resolved measurement of how temporal attention structure forms during training is genuinely underexplored next to the large literature on final checkpoint analysis. Releasing the census from nine training runs is a service to the community.

Critical weaknesses

1、The main problem is that the contribution is a census whose bottom line is that a small fraction of heads specialize, they mostly live in the first temporal block, they show two recurring motifs, and we cannot tell whether any of this matters for the model's output. That is thin for a top tier venue. The two motifs in particular sit close to the path of least resistance for temporal attention: of course some heads learn to attend to the current frame, and of course some learn to attend to neighboring frames. Finding this inductive bias in a trained factorized video transformer is close to confirming the architecture works as designed. Moreover, the NLP head specialization literature spent years establishing that concentrated or specialized heads are frequently removable with no loss, so specialization is not even evidence of importance, and this paper never gets past step zero of that chain.

2、The promised applications are entirely speculative. The motivation is training diagnostics, pruning targets, and quality monitoring, yet there is no pruning experiment, no intervention of any kind, and no link from any measured quantity to any generation quality metric. Admitting this does not turn a description into a contribution. Generality is also weaker than the framing suggests: everything is one architecture family, one training recipe, 16 frames, three scales, and the offer that the analysis extends to joint spatio temporal architectures comes with the quiet caveat that the metric would need adaptation. Finally, the one claim with scientific content, that selected heads recur across seeds, dissolves on contact: reproducibility holds only at block level, with specific head indices differing across seeds. A positional regularity that does not survive to the level of individual heads is more consistent with a mechanism being allocated wherever training happens to land than with the census isolating a functional unit. None of this makes the work wrong; it makes it a measurement exercise in search of a use, and for the main track I would want at least one causal result, even a small one, such as pruning the selected heads and showing that something changes in training dynamics or sample quality.

Questions for authors
  1. If the selected heads were pruned or frozen at the checkpoint where they emerge, does anything measurable change in training dynamics or sample quality, even at one scale?
  2. Given that selected head indices differ across seeds, what exactly is the scientific claim a reader should take away: a mechanism, a property of the first temporal block, or a training artifact?
  3. Does CFAC correlate with any existing quantity of interest, such as loss curvature, gradient norms, or attention sink behavior, that would justify monitoring it during training in practice?
Reviewer_4XowOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tracks every temporal-attention head across checkpoints of nine Open-Sora STDiT runs (306M, 760M, 1.03B; three seeds each) trained from scratch on UCF-101 at 16 frames / 256x256. Each head is scored by cross-frame attention concentration (CFAC), one minus the normalized entropy of its 16x16 query-frame to key-frame attention. Heads are selected ("birth heads") by a preregistered rule combining a change-point gate (BH q=0.05) with an effect-size filter Δh>0.1\Delta_h \gt 0.1. Findings: the population-mean CFAC is flat or decreasing in every run while 4 to 13% of heads increase strongly; selected heads reproduce across seeds at the level of the first temporal block (L0) and population rank statistics, but not at specific (layer, head) coordinates; in the 760M runs the selected heads end in self-frame-diagonal or adjacent-frame-band motifs. A preregistered correlation between CFAC and warp error and a 10-head ablation are both null, and the authors show warp error is motion-confounded. The paper positions itself as a preregistered methodology plus a case study and explicitly makes no causal or quality claims.

Critical strengths

1. Unusual methodological honesty. Preregistered decision rules with a recorded timeline, a full claim-boundary table (Table 11), negative results reported in full, sensitivity analyses (BY correction, p-value flooring, threshold grid), and explicit acknowledgement that the change-point gate is not calibrated. The paper never overclaims.

2. A useful caution. The demonstration that a population-mean statistic can fall while a sparse subset of heads rises (Table 2) is a concrete reminder for anyone studying training dynamics in video DiTs to work at head resolution.

Critical weaknesses

1. Low informativeness relative to the machinery. After the statistical apparatus, the substantive findings are: some heads become more frame-concentrated; they are more often in the first temporal block; the resulting patterns are self-frame and adjacent-frame attention. These are the most generic temporal patterns one would predict a priori, their head identities are not stable across seeds, and the paper's own interventions find no detectable effect on generation. The introduction motivates the study by training monitoring and head-level interventions such as pruning, yet neither is demonstrated: no diagnostic signal is derived, no pruning or early-stopping experiment is run, and the framework for joint-attention architectures is described in one sentence. The deliverable is a careful description with no established consequence, which is below the bar for a main-track paper in my view.

2. A narrow and unvalidated regime. All runs are one factorized architecture on UCF-101 at 256x256, at most 50k to 95k steps at lr 2e-5 with per-process batch 4 to 8. No generation quality (e.g. FVD/IS on UCF-101) is reported for the analyzed checkpoints, so the reader cannot tell whether these are reasonably trained video models or an under-trained regime in which temporal attention is barely used; the monotonically decreasing mean CFAC is consistent with either. CFAC is confounded with logit scale (acknowledged), the census size is unstable across seeds (26 vs 75 heads at 1.03B), and the primary 760M rule was developed post hoc on the same runs. Modern video DiTs use joint spatio-temporal attention, so the generality question the paper raises repeatedly remains fully open.

Questions for authors

Q1. Please report sample quality over training (FVD or similar on UCF-101) for each run. Do selected-head onsets coincide with any change in quality, and are the final checkpoints competitive with published Open-Sora/Latte numbers on UCF-101?

Q2. Is the census robust to a scale-invariant routing statistic (e.g. rank-based concentration or argmax-frame consistency across spatial positions) rather than softmax entropy? Uniform logit sharpening would raise CFAC without any routing change.

Q3. Why L0? One hypothesis is that the first temporal block sees raw patch embeddings plus temporal position embeddings, where frame identity is most salient. Can this be tested (e.g. by altering the temporal position encoding)?

Q4. Can you demonstrate at least one of the motivating uses: prune the selected heads vs matched random heads and report FVD, or show that census onset provides an earlier or more sensitive training signal than the loss curve?

Q5. The ablation uses n=32 prompts. What effect size on warp error (or LPIPS) was it powered to detect?

Reviewer_VttxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Authors extend previous works on analysis of temporal attention, which was previously only done on final checkpoints, but in this paper they investigate when and where exactly those structures form during training. Paper tracks this across 3 sizes of STDiT trained with 3 seeds each. In transformer each head is scored using CFAC, which is equal to information redundency of their attention over 16 frames. Specific heads are selected by a rule established on runs with 760M model and applied unchanged for 306M and 1.03B models.

This paper reports 4 findings:

  • About 4-13% of heads become concentrated, while overall CFAC stays flat
  • Training on multiple seeds agree in the first temporal block (L0), but exact heads are matched only by chance. If L0 is excluded the cross seed doesn't pass the registered threshold
  • Selected 760M heads end as self-frame diagonals or adjacent-frame bands
  • Correlation and ablation tests against warp error are null, and the paper shows that metric is strongly motion confounded

Paper makes no claims about causality or quality and is presented as a methodology and a case study.

Critical strengths
  1. It adds a training-time view of video DiT attention. Prior analyses (Wen et al., 2025; Liu et al., 2025; Nam et al., 2025) study trained models. This paper tracks every temporal head across checkpoints, seeds and three configurations. The sparse-versus-mean decomposition (Table 2, Fig. 7) is clean. The L0-exclusion and within-L0 conditional tests correctly show that global coordinate overlap reflects block location, not stable head identity.

  2. It handles negative evidence responsibly. Null results, the temporal-before-spatial reversal at 1.03B Run B, census-size instability and the motion confound in warp error are all reported, not hidden. As a result, the descriptive claims that remain are credible.

Critical weaknesses
  1. The L0 localization result may have an architectural explanation the paper does not discuss. The paper treats the first-block preference as a training-dynamics phenomenon. In the public Open-Sora v1.0 STDiT code (opensora/models/stdit/stdit.py), the temporal position embedding is added only to the input of block 0's temporal attention (tpe is passed only when i == 0) and is not added to the residual stream. So L0 is the only temporal block whose queries and keys see explicit frame position.

    Self-frame diagonals and +/-1-frame bands are exactly the patterns explicit position makes easy. Several L0 heads are also already structured at the first checkpoint (L0 CFAC max 0.424 at 1k steps; L0H11 previous-frame mass 0.266 at 1k). The finding that L0 reproduces across seeds but individual heads do not may therefore mostly reflect where position is injected. This bears directly on the paper's main positional claim.

  2. The measurements do not yet establish when routing specialization forms, which is the stated contribution.

    • CFAC rises under uniform logit sharpening without any change in which frames are attended. For 16 frames with one favored key, scaling the logits by 4x raises CFAC from 0.02 to 0.60, far above the 0.1 threshold. Temporal attention in this code has no QK-norm, so growth in logit scale is a plausible driver (Zhai et al., ICML 2023). The fixed-timestep re-probe shows stable rankings, but not stable selected sets, onsets, or routing under controlled logit scale.
    • The paper acknowledges the gate is anti-conservative on the log-spaced checkpoint grid, and Table 3 shows that the net-rise filter, not the gate, produces most of the sparsity.
    • The most reproducible heads (L0) are largely structured before the first probed checkpoint, so the census does not observe their onset.

    Taken together, the onset and "birth" interpretation is not yet validated.

Questions for authors
  1. (Affects W1) Do your three configurations inject the temporal position embedding only at block 0, as in the original STDiT class (unchanged from v1.0 through the opensora/v1.3 branch)? The newer STDiT2 and STDiT3 classes instead apply RoPE in every temporal block, and STDiT3 enables QK-norm by default, so knowing which class you trained matters for both weaknesses. On existing checkpoints, do the L0 diagonal and band motifs survive if the position embedding is zeroed at probe time? A small retraining run at 306M with the embedding added at every block would be stronger, but the probe-time check alone would be informative.

  2. (Affects W2) On a fixed held-out probe set with repeated noise draws, do census membership, L0 enrichment and onsets persist under a logit-scale control, such as rank- or offset-based routing statistics? Please report uncertainty across clips, not across correlated spatial positions. If possible, also show that the gate recovers known onsets on simulated trajectories using your actual checkpoint grid.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.