acceptodds
Under review as a conference paper at ICLR 2027

SANA-Video 2.0: Hybrid Linear Attention with Attention Residuals for Efficient Video Generation

10 views

Abstract

We introduce SANA-Video 2.0, a hybrid video diffusion transformer instantiated at 5B scale under a unified architecture. Designed to generate high-quality video up to 720p on a single GPU, SANA-Video 2.0 matches full-softmax video DiTs in quality while retaining the favorable long-sequence scaling of linear attention. To avoid quadratic attention throughout, Hybrid Linear–Softmax Attention combines gated linear attention for𝑂(𝑁 )-dominated mixing with periodic gated-softmax anchors at a 3:1 ratio, restoring the full-rank token interactions that pure linear attention lacks. To propagate these refreshed representations across depth, Block Attention Residuals (AttnRes) route completed block summaries into later linear layers, enabling anchor-feature reuse and boosting deep-layer effective rank by ∼12%. Through from-scratch training, SANA-Video 2.0 learns the complete hybrid directly rather than linearizing pretrained models, with reduced-resolution proxy studies establishing 25% softmax as the optimal quality–efficiency trade-off. With 40-step sampling, SANA-Video 2.0 achieves a VBench score of84.30 in 13.2s at 480p on a single H100, remaining competitive with far larger softmax video DiTs at a fraction of the latency. Its compiled DiT forward pass is 3.2× faster than a matched full-softmax baseline at 720p/60s, a gap that expands with video duration. Furthermore, full-stack Sol-Engine optimization (kernel fusion, caching, and sparse attention) accelerates this hardware-friendly backbone by a further 3.58×, bringing the 5B pipeline to 13.06s at 720p/5s and making it 120× faster than Wan 2.2-A14B on one H100. Overall, our hybrid design recovers softmax-level expressiveness at substantially reduced cost, unlocking scalable long, high-resolution video generation.

Primary area:
generative models
Keywords:
Video Diffusion Model

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_en2ZOpenReview ↗
Rating
3
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents SANA-Video 2.0, an efficient video generation model that combines linear attention, full attention, and attention residuals. The backbone uses mostly linear-attention layers, with one full-attention layer in every four layers. The paper also includes several training and inference optimizations. The results show competitive video quality and fast inference.

Critical strengths
  1. The speed improvement is impressive. Generating a 720p video in about 13 seconds on a single H100 is a strong practical result.

  2. The paper is very detailed thus solid. It covers the model design, data processing, training recipe, and inference optimizations, giving readers a clear picture of the full system.

  3. The work has clear value to the community. An efficient video generation model, together with these detailed recipes, could be useful for both further research and practical applications.

Critical weaknesses
  1. Limited novelty compared with SANA-Video 1.0 and existing work. Most main components follow existing ideas, including linear attention, hybrid attention, and attention residuals. The main contribution is combining and adapting these components for video generation.

  2. The ablations could better show the benefits for final video quality. The paper includes useful attention-ratio experiments and detailed AttnRes studies in the appendix. However, the ratio comparison mainly uses validation loss after short training runs at 256p. The late-stage AttnRes comparison also gives nearly the same loss with and without the module, as the authors acknowledge. These results help explain the design choices, but leave their effect on final video quality unclear. A controlled comparison using video-quality metrics or human evaluation would make the benefits of hybrid attention and AttnRes more convincing.

  3. The quality performance is not as strong as claimed. In the 81-frame comparison, the total VBench score improves only slightly over SANA-Video 1.0, from 84.17 to 84.30, while the semantic score drops from 81.46 to 79.05. The total gain is driven by the higher Dynamic Degree score, with the other dimensions together contributing negatively. More motion does not necessarily mean better video quality. These results do not show a clear overall improvement over the first version.

Questions for authors
  1. Which improvements over SANA-Video 1.0 remain when model size, training budget, and inference settings are controlled?

  2. How does the comparison change when Dynamic Degree is held fixed or removed from the total score? A human evaluation of motion quality and prompt following would also help explain the results.

3 comments

Thank you for recognizing the practical speed and the value of the detailed training recipe. We have clarified the contribution beyond SANA-Video 1.0 and added a direct perceptual comparison to better characterize the quality differences.

Q1. What improves over SANA-Video 1.0, and which factors are controlled?

SANA-Video 2.0 advances the complete generator through model scaling, architecture and training. We scale the backbone from 2B to 5B, introduce periodic softmax anchors and cross-depth feature reuse, and upgrade shot segmentation, captioning, quality/motion filtering, the training curriculum and post-training. Our contribution is the resulting efficient video generator together with the video-specific design findings and documented training recipe. These findings include the proxy-based softmax allocation and a shared-query routing design for reusing softmax-containing block features in bidirectional video diffusion.

The architectural choices draw on the controlled proxy sweeps and early visual and feature analyses. AttnRes gives later linear layers access to softmax-enriched block summaries and combines shallow with deeper representations. Its S=8 design probe adds 3.1% latency and 2.1% memory. These observations and the cost of pretraining on 64 H100s guided the selected design, as clarified in Section 5.3.2.

Model size and training recipe are part of the upgraded system. The architecture-search studies control backbone dimensions and training settings to examine the softmax allocation, while the production comparison evaluates the combined upgrades. For the complete-system comparison, we now control the inference conditions and report direct human preferences:

We evaluated 30 prompts fixed before generation, with ten prompts each for detail/stability, motion/anatomy and semantics/composition. One author rated model-anonymized A/B video pairs, with randomized side assignment and ties allowed. Both models use matched prompts, seeds and sampling settings at 704 x 1280, 81 frames, 16 fps and 50 steps, with their native VAEs. SANA-Video 2.0 uses the ReFL-trained model.

CriterionSANA-Video 2.0 winsSANA-Video 1.0 winsTieSANA-Video 2.0 preferred (%)
Overall preference291096.67
Local detail291096.67
Temporal stability271290.00
Fewer artifacts271290.00
Anatomical naturalness261386.67
Prompt alignment1621253.33

Percentages use all 30 pairs, including ties in the denominator. The 180 criterion judgments come from one evaluator on 30 video pairs. These preferences describe the combined improvements to the complete system on the evaluated prompts.

The observed preference is substantial for overall quality (29/30, 96.67%), local detail (29/30), stability and fewer artifacts (27/30 each), and anatomical naturalness (26/30). This assesses the practical benefit of the full upgrade, including model size, architecture, data and training. Table 13 in the revised manuscript reports the same evidence.

Q2. What happens when Dynamic Degree is removed, and what does human evaluation show?

We directly assess motion quality through human judgments of stability and artifacts. We also recomputed the original rounded dimension scores after removing Dynamic Degree and renormalizing the remaining Quality weights, keeping the standard 4:1 Quality-Semantic weighting for Total:

Metric / aggregationSANA-Video 1.0SANA-Video 2.0
Quality, original84.8585.61
Quality, excluding Dynamic Degree86.1586.17
Total, original84.1784.30
Total, excluding Dynamic Degree85.2184.74

The reweighted Quality scores are close. This sensitivity calculation and the human preferences provide complementary views: the latter directly assess the observed gains in detail and visual stability, together with prompt alignment and ties.

In the model-anonymized 30-pair assessment above, the author evaluator prefers 2.0 in 29 pairs overall, with one preference for 1.0 and no overall ties. For temporal stability and fewer artifacts, the counts are 27 preferences for 2.0, one for 1.0 and two ties each. Prompt alignment is more balanced: 16 preferences for 2.0, two for 1.0 and 12 ties. This distinction explains why a small VBench aggregate difference can coexist with a large observed preference for detail and visual stability in this assessment. We report all ties and the evaluator count, and include the Dynamic Degree sensitivity calculation in Appendix L.2.

Thank you for the clarification and additional evaluation. I acknowledge that model size is part of the upgrade and effectiveness of the dynamic degree. However, my concerns about the novelty compared with SANA 1.0 is not fully addressed. Given the increase from 2B to 5B parameters, some performance improvement is expected from model scaling alone, making it difficult to assess the contribution of the architectural changes. I therefore maintain my rating of 3 (weak accept).

Thank you for the follow-up and your support. We would like to highlight the controlled pure-linear versus hybrid comparison in Section 5.3.1 and Figure 4, which helps distinguish the attention-design effect from the 2B-to-5B model upgrade.

All variants use 28 layers and a hidden width of 3,072, and are trained from scratch at 256p/81 frames for 10,000 steps with the same data, optimizer settings, seed and training recipe. Validation uses 200 held-out videos. Selected configurations from this sweep are:

Attention layoutSoftmax fractionValidation MSE
Pure linear (SANA-Video 1.0)0%0.955
Hybrid (Ours)25%0.905

At these fixed backbone dimensions, the 25% hybrid reduces validation MSE by approximately 5.2% relative to pure linear attention. This provides evidence for the attention-layout choice at the proxy scale, independently of the 2B-to-5B production-model comparison. We select 25% as a practical quality-efficiency tradeoff in the full ratio sweep.

Then, we try to use AttnRes to help later linear-attention layers reuse features enriched by the softmax anchors. It retains block summaries and learns how to combine them across depth. A later layer can therefore access shallow block features together with more recent, deeper representations, including softmax-anchor updates. Our early qualitative visualizations and feature analyses motivated this feature-reuse design.

We have not yet completed a full end-to-end comparison of the architectural variants at matched model scale and training budget. Our production pretraining uses 64 H100 GPUs, and the additional training runs exceed our current compute budget. We leave this controlled comparison, using video-quality metrics and human preference evaluation under matched inference settings, to future work.

Our architectural contribution concerns how to allocate softmax computation and reuse its features within a predominantly linear video-diffusion backbone. The controlled ratio study, early visualization results and state analysis support the allocation choice, while the final 5B model demonstrates the resulting complete system. We hope this makes the architectural evidence and its role in the contribution clearer.

Reviewer_zumCOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper brings hybrid linear-softmax attention with attention residuals to bidirectional video diffusion, trains the resulting 5B model from scratch, and re-selects the softmax ratio for the video gen. The final design places a gated global softmax anchor every four layers, routes completed block summaries across depth with shared queries, and reaches VBench Total 84.30 at 13.2s for 480p/81f on one H100, with compiled forward speedups over a matched full-softmax backbone up to 3.17x at 720p/60s.

The paper's own related work section states the contribution precisely: port the hybrid layout and AttnRes to video, train from scratch, re-pick the ratio. Everything else is solid engineering around that core.

Critical strengths

S1. The measurement work is careful, and the reporting is honest. Several things here that I wish more efficiency papers did. The authors notice that validation loss varies about 3x across the noise axis, so they read every proxy comparison per timestep instead of collapsing to a scalar mean. Both attention variants get their best kernels under torch.compile, and eager diagnostics are never mixed with compiled numbers. The AttnRes continuation probe comes out tied (0.4851 vs 0.4855), and they say so rather than spinning it. The timestep-stratified checkpoint monitor, which cuts the variance of the early-to-final estimate by 2.27x, is a useful tool on its own.

S2. A complete, well-documented system at a useful operating point. To my knowledge this is the first from-scratch study of hybrid linear attention with cross-depth routing for bidirectional video diffusion. The loop is closed: proxy-based architecture search, a documented training recipe, DPO and online ReFL, and QAT that is lossless on VBench (83.25 vs 83.22 for BF16) while cutting storage by 67.9%. The mechanistic analyses, especially the block-entry interventions that drop representation rank by 81 to 88%, give some real insight into why 25% softmax suffices. A 5B model that runs in 13.2s on a single H100 matters to anyone without a cluster.

Critical weaknesses

W1. AttnRes is a title contribution, but there is no evidence it improves generation quality. Everything supporting it is mechanistic: effective rank up 19.7%, routing interventions, entropy probes. The one direct quality check ties, as the authors admit. Nobody shows that higher effective rank leads to better videos, and a router could raise rank while leaving VBench unchanged. What I need is an end-to-end comparison, w/ and w/o AttnRes, on VBench or human preference, at matched compute. Small scale is fine. Right now the paper shows that AttnRes changes representations. Whether it helps generation is open.

W2. Several headline claims outrun the evidence. All quality conclusions rest on VBench, with no human preference study, and the two most natural efficiency controls are absent: a sliding-window hybrid with periodic global layers, and training-free sparse attention on a full-softmax backbone, both at equal latency. Without them, "hybrid is the right efficiency tradeoff" is asserted rather than tested against the alternatives a practitioner would actually reach for. Two framing choices make this worse. The abstract pairs the 120x figure, measured with the accelerated Sol-Engine stack, with a VBench score measured on the unaccelerated model. The Physical AI section claims a win over Cosmos3-Edge and parity with a 30B model from a single qualitative figure with one prompt.

Questions for authors
  1. Can you train a no-AttnRes control at matched compute, from scratch or with a longer continuation budget, and report VBench or a small preference study? This is the single most important addition for me.

  2. What is VBench Total for the fully accelerated Sol-Engine pipeline, w/ caching and sparse anchors on?

  3. At equal latency, how does the 25% hybrid compare against a sliding-window plus periodic-global variant, and against training-free sparse attention on a full-softmax backbone?

  4. Semantic is your weakest aggregate. Do you have a sense of where the gap comes from: the linear state bottleneck, the Gemma-2-2B text encoder, or the data? I am also curious whether the drift at 193 frames survives matched sampler settings.

3 comments

Thank you for recognizing the practical operating point and the training and measurement methodology. We have added a joint quality-speed evaluation and a model-anonymized human comparison, and revised the scope of the architectural and Physical AI claims. The key results are included here.

Q1. What motivates AttnRes, and what does the evidence establish?

We use AttnRes to help later linear-attention layers reuse features enriched by the softmax anchors. It retains block summaries and learns how to combine them across depth. A later layer can therefore access shallow block features together with more recent, deeper representations, including softmax-anchor updates. Our early qualitative visualizations and feature analyses motivated this feature-reuse design.

The overhead is modest: the S=8 block-span probe measures 3.1% latency and 2.1% memory overhead. In the query-sharing probe, shared queries give loss 0.495 with 4% memory overhead, compared with 0.496 and 16% for per-layer queries. These are early router-design measurements. The routing analysis additionally assigns 52.9%/46.9% of deep-layer attention/FFN aggregation mass to completed blocks, supporting the intended cross-depth reuse. Given the cost of pretraining on 64 H100s, these early observations, explicit feature reuse and modest overhead guided our choice. Section 5.3.2 now states this design rationale explicitly. The rank and routing analyses quantify the feature-reuse mechanism, and the production results evaluate the complete generator.

In future work, we plan to train larger models and evaluate how AttnRes contributes to cross-depth feature reuse and generation quality as model size increases. We have included this direction in the revised Future Work section.

Q2. What is the quality with caching and sparse anchors enabled?

We added a joint measurement of the ReFL-trained 5B pipeline with AttnRes and cache plus PISA.

Both conditions use the complete ReFL-trained 5B model with AttnRes enabled at 704 x 1280, 81 frames and 16 fps, with 50-step Flow DPM-Solver++, CFG 6, flow shift 6 and motion score 30. We pair weights, prompts, conditioning, noise and decoding. The acceleration uses cache threshold 0.0348, seven initial and one final dense steps, and 10% PISA block density (block size 64) on the last four softmax anchors.

MetricBaseCache + PISAChange95% paired interval
Mean resident E2E latency (s)33.6922.66-11.02-
Quality83.1082.94-0.17[-0.65, +0.30]
Semantic77.7378.04+0.31[-0.73, +1.39]
Total82.0381.96-0.07[-0.51, +0.36]

Quality evaluation covers all 944 distinct official prompts, one seed per prompt, for 1,888 videos across the two conditions. We measure latency on one H200 using six diagnostic prompts and three repetitions per condition. The timing covers text encoding, denoising and VAE decoding. Model loading, warmup and file writing are excluded. The geometric mean of paired per-prompt median speedups is 1.49x, with a 95% interval of [1.42, 1.57]. Score intervals use 20,000 paired prompt resamples at the evaluated seed. Changes are computed before rounding.

Total is nearly unchanged at this operating point, with scores of 82.03 for the base and 81.96 after acceleration. The decrease is 0.07 points, while E2E inference is 1.49x faster. The revised Tables 11-12 report this joint H200 operating point, including all 16 dimensions. Table 9 documents the B200 deployment measurements.

Q3. How does the hybrid compare with sliding-window/global or sparse full-softmax alternatives at equal latency?

Our central question is how to raise the capability ceiling of a predominantly linear-attention video backbone. We therefore study where to place softmax anchors and how much softmax computation to allocate. The controlled ratio sweep and sequence-scaling profiles directly examine this design space, and support 25% as a practical operating point within it.

Sparse attention is complementary: our joint deployment result above applies PISA to the remaining softmax anchors. A fair comparison with a sliding-window/global backbone requires comparable training. Training-free sparsity on full softmax should be evaluated on the corresponding full-softmax backbone at matched latency. Within a production-training setup of 64 H100s, we concentrate on the selected hybrid backbone and use the controlled proxy sweeps to study softmax allocation. Appendix A.2 now makes this predominantly linear design space explicit.

Q4. What explains the Semantic gap and the 193-frame decline?

SANA-Video 1.0 starts from an image model, which provides extensive semantic alignment before video training. SANA-Video 2.0 trains its video backbone from scratch. This initialization difference is relevant to interpreting the Semantic comparison. At 79.05, our 81-frame Semantic score is still above the reported LTX-2.3 (74.35), Wan 2.1 1.3B/14B (75.65/76.11), HunyuanVideo (76.88) and Cosmos-3 Nano (78.50) results.

The frame-count results are:

ModelFramesTotalQualitySemantic
SANA-Video 1.08184.1784.8581.46
SANA-Video 2.08184.3085.6179.05
SANA-Video 2.012185.2986.9878.51
SANA-Video 2.019384.4886.5176.37

The 121- and 193-frame results use the same checkpoint and sampler settings: 40 steps, CFG 8, flow shift 12, motion bucket 30 and 24 fps. Thus, Semantic is lower at 193 frames under matched sampler settings. Our training data are concentrated around 121-frame clips. Limited longer-clip coverage is a likely contributor. The 121-frame setting gives the strongest Total score. Appendix D.1 now includes this training-distribution explanation.

To complement aggregate metrics with direct perceptual evidence, we also performed the following human assessment.

We evaluated 30 prompts fixed before generation, with ten prompts each for detail/stability, motion/anatomy and semantics/composition. One author rated model-anonymized A/B video pairs, with randomized side assignment and ties allowed. Both models use matched prompts, seeds and sampling settings at 704 x 1280, 81 frames, 16 fps and 50 steps, with their native VAEs. SANA-Video 2.0 uses the ReFL-trained model.

CriterionSANA-Video 2.0 winsSANA-Video 1.0 winsTieSANA-Video 2.0 preferred (%)
Overall preference291096.67
Local detail291096.67
Temporal stability271290.00
Fewer artifacts271290.00
Anatomical naturalness261386.67
Prompt alignment1621253.33

Percentages use all 30 pairs, including ties in the denominator. The 180 criterion judgments come from one evaluator on 30 video pairs. These preferences describe the combined improvements to the complete system on the evaluated prompts.

In this assessment, the complete 2.0 system is preferred in 29/30 pairs overall and for detail, 27/30 for stability and fewer artifacts, and 26/30 for anatomical naturalness. Prompt alignment has more ties, as reported explicitly above. The revised Table 13 contains the same counts.

Physical AI claim

We have revised the Physical AI paragraph and figure caption to describe the illustrated robot-manipulation outputs with precise qualitative wording.

Thank you for the rebuttal, I'll maintain my current score since no additional cues appear during the rebuttal phase.

Reviewer_oXRgOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

SANA-Video 2.0 is a ~5B video diffusion transformer trained from scratch, in which three of every four layers use gated bilinear linear attention and every fourth layer is a gated global-softmax anchor (25% softmax), combined with Block Attention Residuals (AttnRes) that route completed 8-layer block summaries to later layers via a depth-shared query. The hybrid ratio is selected by reduced-resolution proxy studies, and the model is trained with a documented multi-stage recipe (quality/motion data funnel, low-to-high resolution/duration curriculum, Self-Flow/TQD, then Diffusion-DPO and online ReFL). The model reports VBench Total 84.30 at 480p/81f in 13.2s on one H100, compiled DiT-forward speedups over a matched full-softmax backbone that grow with sequence length (1.16× at 480p/121f to 3.17× at a 720p/60s tensor shape), and a separate Sol-Engine deployment giving 3.58× end-to-end speedup at 720p/8s on B200.

Critical strengths
  1. A complete, well-executed system with unusually careful efficiency profiling. The latency methodology is rigorous: the full-softmax control uses its own best implementation (FlashAttention, torch.compile) under the same H100/bf16/batch-1 protocol with CUDA-event timing (Appendix D.2), the paper reports full scaling curves over resolution, duration, and model size rather than a single favorable shape, and it includes an honest controlled decomposition of where the speedup dilutes (Appendix G.1), module-level composition showing the remaining softmax anchors dominating long shapes (Figure 12), and staged, per-device deployment attribution (Table 8). The timestep-stratified way of reading the proxy loss curves (Figure 4) is also a useful methodological contribution, and the paper is commendably transparent about its own limitations, including stating in Figure 1 that quality and latency are not jointly evaluated and that isolating AttnRes's end-task effect would require a second production run.

  2. A coherent and plausible architectural story, validated at the level the experiments target. The division of labor — linear layers for cheap sequence mixing, periodic softmax anchors for less rank-constrained interaction, AttnRes to carry anchor-containing block features across depth — is well motivated, and the mechanistic probes are matched to the claims made for them: enabling AttnRes raises deep linear-state effective rank by ~20% on the same checkpoint with paired prompts/seeds (Figure 6b), routing mass does concentrate on completed blocks, and removing completed sources selectively reduces rank at block entries (Appendix F.6). Reaching Quality-leading VBench performance at 5B scale with a mostly linear backbone, trained from scratch rather than linearized post hoc, is a meaningful practical result.

Critical weaknesses
  1. The architecture's contribution to generation quality is not isolated, and the two central design choices are never validated jointly. The 25% softmax ratio is selected on 256p proxy backbones (10K steps, 200 validation videos; Section 5.3.1), as described without the AttnRes router, and AttnRes is only studied afterwards at the locked 25% ratio (Section 5.3.2, Appendix I). This sequential selection assumes the two choices do not interact, although AttnRes is motivated precisely by propagating anchor features across depth: with effective routing a lower anchor fraction might suffice, and the router is never switched on at the pure-linear (0%) or pure-softmax (100%) endpoints. For AttnRes itself, the only training-level evidence is a late-stage continuation probe that is essentially tied (MSE 0.48506 vs. 0.48547; Appendix F.1); the effective-rank and routing probes establish mechanism, not generation quality. Finally, the production VBench 84.30 is obtained after a data funnel, Self-Flow/TQD, Diffusion-DPO, and ReFL that were "selected jointly, without component-wise ablations" (Section 4), with no stage-wise quality numbers — so how much of the final quality comes from the hybrid-plus-AttnRes backbone, as opposed to the recipe and post-training, cannot be determined from the paper.

  2. The headline efficiency numbers are not jointly measured with quality, and the main quality table mixes protocols. The Figure 5 forward speedups are measured with AttnRes disabled, and the larger factors (2.01× at 1080p, 3.17× at a 720p/60s tensor shape) are at shapes far beyond the 8s maximum training duration, with no corresponding quality results; at the 480p operating point where quality is reported, the speedup is 1.16×. The Sol-Engine figures (3.58×; "approximately 120× faster than Wan 2.2-A14B") come from an external stack using caching (33 of 50 steps evaluated) and sparse attention whose generation quality is not measured (Appendix K), applied only to the authors' model. Table 1 meanwhile mixes official leaderboard scores, author-reported scores, and the authors' own measurements at 81–129 frames; per Appendix D.1 the 81-frame and 121/193-frame results use different checkpoints and sampling settings, contrary to the "same checkpoint" phrasing in Section 5.2, and the BF16 VBench baseline in Table 9 (Total 83.22) does not match the headline 84.30. The claim of competitive quality at a fraction of the latency is therefore assembled from configurations that were never measured together.

Questions for authors
  1. On the proxy scale, how does AttnRes interact with the softmax ratio? Specifically, what are the from-scratch validation losses (and, if feasible, a small generation-quality metric) for AttnRes on/off at 0%, 25%, and 100% softmax under the Figure 4 protocol? Does the 25% knee move when the router is present, and does AttnRes help the pure-softmax endpoint, where there is no linear-state rank bottleneck to compensate?

  2. Is there any end-task evidence for AttnRes beyond the continuation probe and rank analysis — e.g., a from-scratch proxy run with vs. without AttnRes compared on generated samples, or a VBench-style evaluation of the late-stage continuation pair of Appendix F.1? If the intended claim is mechanism-level only, it would help to say so explicitly in the main text where the rank results are presented.

  3. Could you report one fully joint operating point: end-to-end latency and VBench for the complete production model with AttnRes enabled (no Sol-Engine), and, if available, any quality measurement (even on a small prompt subset) for the Sol-Engine accelerated configuration of Table 8? This would connect the quality and efficiency claims that are currently measured separately.

  4. Could you provide stage-wise VBench (or a subset) for the production recipe — after SFT, after Diffusion-DPO, and after ReFL? This would help readers gauge how much of the final quality comes from post-training relative to the backbone, and would make the recipe itself more useful to reproduce.

5 comments

Thank you for the detailed review and for recognizing the controlled efficiency measurements and mechanistic analyses. We have added a joint quality-latency evaluation and clarified the basis for the architecture choices. We summarize the relevant evidence below.

Q1. How does AttnRes interact with the softmax ratio?

Our search is sequential: we first select the softmax budget, then select the AttnRes design at that fixed budget. For completeness, the ratio sweep gives:

Softmax fractionValidation loss
0% (all linear)0.955
14.3% (4/28 layers)0.914
25%0.905
50%0.897
100% (all softmax)0.945

These models are trained from scratch using the Figure 4 proxy protocol: 256p, 81 frames, depth 28, width 3,072, and 10K training steps, with validation on 200 held-out videos. The 50% configuration has the lowest loss, while 25% is the practical quality-efficiency knee: moving to 50% increases profiled latency by 1.29x at 1080p/121 frames for less than 1% lower validation loss.

Introducing AttnRes may change the preferred softmax fraction. Our 25% configuration is the practical operating point selected before the router design, as now stated in Appendix I. AttnRes targets cross-depth reuse in this linear-heavy backbone. At a pure-softmax endpoint, the same router would instead provide cross-depth reuse without a linear-state bottleneck.

Q2. What evidence motivates AttnRes beyond MSE and rank?

We use AttnRes to help later linear-attention layers reuse features enriched by the softmax anchors. It retains block summaries and learns how to combine them across depth. A later layer can therefore access shallow block features together with more recent, deeper representations, including softmax-anchor updates. Our early qualitative visualizations and feature analyses motivated this feature-reuse design.

The overhead is modest: the S=8 block-span probe measures 3.1% latency and 2.1% memory overhead. In the query-sharing probe, shared queries give loss 0.495 with 4% memory overhead, compared with 0.496 and 16% for per-layer queries. These are early router-design measurements. The routing analysis additionally assigns 52.9%/46.9% of deep-layer attention/FFN aggregation mass to completed blocks, supporting the intended cross-depth reuse. Given the cost of pretraining on 64 H100s, these early observations, explicit feature reuse and modest overhead guided our choice. Section 5.3.2 now states this design rationale explicitly. The rank and routing analyses quantify the feature-reuse mechanism, and the production results evaluate the complete generator.

In future work, we plan to train larger models and evaluate how AttnRes contributes to cross-depth feature reuse and generation quality as model size increases. We have included this direction in the revised Future Work section.

Q3. Can generation quality and end-to-end latency be measured jointly?

Yes. We added the following paired quality and latency evaluation with AttnRes enabled in both conditions.

Both conditions use the complete ReFL-trained 5B model with AttnRes enabled at 704 x 1280, 81 frames and 16 fps, with 50-step Flow DPM-Solver++, CFG 6, flow shift 6 and motion score 30. We pair weights, prompts, conditioning, noise and decoding. The acceleration uses cache threshold 0.0348, seven initial and one final dense steps, and 10% PISA block density (block size 64) on the last four softmax anchors.

MetricBaseCache + PISAChange95% paired interval
Mean resident E2E latency (s)33.6922.66-11.02-
Quality83.1082.94-0.17[-0.65, +0.30]
Semantic77.7378.04+0.31[-0.73, +1.39]
Total82.0381.96-0.07[-0.51, +0.36]

Quality evaluation covers all 944 distinct official prompts, one seed per prompt, for 1,888 videos across the two conditions. We measure latency on one H200 using six diagnostic prompts and three repetitions per condition. The timing covers text encoding, denoising and VAE decoding. Model loading, warmup and file writing are excluded. The geometric mean of paired per-prompt median speedups is 1.49x, with a 95% interval of [1.42, 1.57]. Score intervals use 20,000 paired prompt resamples at the evaluated seed. Changes are computed before rounding.

Total is nearly unchanged at this operating point, with scores of 82.03 for the base and 81.96 after acceleration. The decrease is 0.07 points, while E2E inference is 1.49x faster. The revised Tables 11-12 report this joint H200 operating point, including all 16 dimensions. Table 9 documents the B200 deployment measurements.

We also reprofiled the complete 5B model, including AttnRes, against a width- and depth-matched full-softmax computational reference:

Profiled duration / framesHybrid + AttnRes (s)Full-softmax reference (s)Speedup
5s / 1210.3330.4661.40x
20s / 4811.8634.1892.25x
40s / 9615.29814.7092.78x
60s / 1,44110.46332.0063.06x

These are complete DiT-forward medians on one H200, at 736 x 1280 and 24 fps, with BF16, batch size one, matching compiled operators, eight warmups and 20 CUDA-event timed forwards. They exclude text encoding, VAE decoding, sampling and compilation. The durations denote the profiled tensor shapes. Figure 1(c) now includes these AttnRes-enabled measurements. The 60s shape gives 3.06x forward speedup.

Q4. What do the SFT, DPO and ReFL results show?

Our post-training recipe uses SFT, DPO and ReFL. The following table reports representative checkpoints from these three settings, evaluated under a common protocol: 720p, 121 frames at 24 fps, 50-step Flow DPM-Solver++, CFG 8, flow shift 12 and motion score 30. All three evaluations cover the full 16 VBench dimensions with five videos per prompt, using the same prompt lists.

CheckpointQualitySemanticTotal
SFT86.3280.1185.08
DPO86.6479.8285.28
ReFL86.6580.2985.38

ReFL has the highest Total among these checkpoints. Its aesthetic and imaging quality scores are 70.19 and 69.50, compared with 68.03 and 66.83 for SFT, and 68.34 and 67.74 for DPO. Table 8 in Appendix J now reports the checkpoint results alongside the qualitative evidence.

In our qualitative observations, SFT and DPO mainly improve motion stability, while ReFL further improves anatomical details and visual quality. Figure 14 provides the most direct visual comparison between SFT and the final RL checkpoint. It shows four prompts (a dancer, a convertible on a mountain road, an outdoor living-room scene and a running cheetah), at the same CFG scale of 8, with frames at 0, 2, 4, 6 and 8 seconds. The paired sequences illustrate refinements in limb structure, local appearance and visual quality across the post-training sequence. Figure 19 complements these examples with the HPSv3++, DeQA-Score and UniPercept reward trajectories, all of which increase over the reported 400 ReFL iterations. The revised Figure 14 caption makes the full SFT-DPO-ReFL sequence explicit.

Protocol clarifications

For Tables 1 and 5, the revised protocol specifies 16 fps, CFG 6 and flow shift 6 for 81 frames, and 24 fps, CFG 8 and flow shift 12 for 121/193 frames, all with 40 sampling steps. Table 1 identifies score sources: star denotes official leaderboard scores, dagger denotes our runs of official pipelines, and double dagger denotes author-reported results. For QAT, the quality and storage comparison is:

PrecisionVBench TotalStatic storage (GB)Peak memory (GB)
BF1683.228.9410.74
MXFP4 weights / MXFP8 activations (QAT)83.252.874.63

QAT gives VBench Total 83.25 against its BF16 reference of 83.22, with 67.9% lower static storage and 56.9% lower peak memory. Table 10 reports this paired comparison. The original 480p/81-frame quality result remains 84.30, with 13.2s end-to-end inference on one H100.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We thank the AC and all reviewers for the constructive discussion. All three reviewers maintain positive overall ratings (weak accept) and recognize complementary strengths:

  • oXRg highlights the rigorous efficiency profiling and well-motivated architecture, and notes that the additional experiments substantially strengthen the empirical evidence.
  • zumC recognizes the careful measurement, transparent reporting and value of a complete, documented system trained from scratch.
  • en2Z acknowledges the impressive practical speed, detailed technical recipe and value to the research community.

We have strengthened the paper with the following experimental evidence:

  • Architecture beyond scaling. At fixed backbone depth and width, the 25% hybrid reduces proxy validation MSE from 0.955 to 0.905 compared with pure linear attention, a 5.2% reduction.
  • Joint quality and speed. With AttnRes enabled, the complete 5B pipeline achieves 1.49× H200 acceleration using cache plus PISA, reducing E2E latency from 33.69s to 22.66s, with VBench Total scores of 82.03 and 81.96. Quality evaluation covers 944 official prompts with one seed each.
  • Perceptual quality. In a model-anonymized, 30-pair assessment by one author, 2.0 is preferred in 29/30 pairs overall and 27/30 each for stability and fewer artifacts, supporting the practical benefit of the complete system.
  • Post-training results. Representative SFT, DPO and ReFL checkpoints achieve VBench Total scores of 85.08, 85.28 and 85.38 under a shared 720p/121-frame, 50-step protocol. Figure 14 provides complementary visual comparisons.

The revised paper brings together controlled proxy studies, feature-reuse analyses, and joint quality and latency measurements of the complete system. We plan to extend this study to larger models and longer videos, examining how softmax allocation and cross-depth feature reuse evolve with scale.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.