- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper conducts a checkpoint resolved census of every temporal attention head across nine Open Sora STDiT training runs at three scales from 306M to 1.03B parameters. Each head is scored by cross frame attention concentration, an entropy normalized concentration of attention over the 16 frames, and heads are selected under a preregistered change point and effect size rule. The reported findings are descriptive: roughly 4 to 13 percent of heads become strongly concentrated during training while aggregate CFAC stays flat or decreases, selected heads repeatedly appear in the first temporal block but differ across seeds, and two attention motifs recur in the 760M runs, a self frame diagonal and an adjacent frame band. The authors state that their correlation and ablation experiments do not establish that these heads affect generated video quality.
Critical strengths
The honesty of this paper deserves real credit. The abstract states plainly that the correlation and ablation experiments do not establish that the selected heads affect generated video quality, and the body text lives up to that standard. The selection rule is preregistered, the CFAC metric is simple and interpretable, and checkpoint resolved measurement of how temporal attention structure forms during training is genuinely underexplored next to the large literature on final checkpoint analysis. Releasing the census from nine training runs is a service to the community.
Critical weaknesses
1、The main problem is that the contribution is a census whose bottom line is that a small fraction of heads specialize, they mostly live in the first temporal block, they show two recurring motifs, and we cannot tell whether any of this matters for the model's output. That is thin for a top tier venue. The two motifs in particular sit close to the path of least resistance for temporal attention: of course some heads learn to attend to the current frame, and of course some learn to attend to neighboring frames. Finding this inductive bias in a trained factorized video transformer is close to confirming the architecture works as designed. Moreover, the NLP head specialization literature spent years establishing that concentrated or specialized heads are frequently removable with no loss, so specialization is not even evidence of importance, and this paper never gets past step zero of that chain.
2、The promised applications are entirely speculative. The motivation is training diagnostics, pruning targets, and quality monitoring, yet there is no pruning experiment, no intervention of any kind, and no link from any measured quantity to any generation quality metric. Admitting this does not turn a description into a contribution. Generality is also weaker than the framing suggests: everything is one architecture family, one training recipe, 16 frames, three scales, and the offer that the analysis extends to joint spatio temporal architectures comes with the quiet caveat that the metric would need adaptation. Finally, the one claim with scientific content, that selected heads recur across seeds, dissolves on contact: reproducibility holds only at block level, with specific head indices differing across seeds. A positional regularity that does not survive to the level of individual heads is more consistent with a mechanism being allocated wherever training happens to land than with the census isolating a functional unit. None of this makes the work wrong; it makes it a measurement exercise in search of a use, and for the main track I would want at least one causal result, even a small one, such as pruning the selected heads and showing that something changes in training dynamics or sample quality.
Questions for authors
- If the selected heads were pruned or frozen at the checkpoint where they emerge, does anything measurable change in training dynamics or sample quality, even at one scale?
- Given that selected head indices differ across seeds, what exactly is the scientific claim a reader should take away: a mechanism, a property of the first temporal block, or a training artifact?
- Does CFAC correlate with any existing quantity of interest, such as loss curvature, gradient norms, or attention sink behavior, that would justify monitoring it during training in practice?