- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
In this paper the authors study the initialization problem when replacing softmax attention in a pretrained Video DiT with Gated DeltaNet, a popular linear attention architecture. They propose LinMorph, a novel method that learns a shared transformation from the pretrained Q, K, V, and O weights to the GDN parameters. The transformation is shared across layers but produces layer-specific weights. These weights are then materialized as independent GDN modules and further trained through progressive distillation and joint fine-tuning.
Experiments on Wan2.1 and Wan2.2 show that LinMorph provides better initial functional alignment and faster recovery than random initialization and manual weight copying. The recovered hybrid models also achieve competitive video quality and improved inference speed for longer videos.
Critical strengths
-
The paper introduces an interesting and novel direction for post-hoc linearization. Instead of treating the quality loss only as a recovery problem, it learns how to transfer pretrained softmax weights to a different attention architecture. This is a useful and relatively unexplored perspective.
-
The controlled initialization results are convincing. Under the same target architecture and recovery setup, LinMorph reaches better functional alignment than random initialization and manual copying, often with fewer recovery updates.
-
The mapper is analyzed in good detail. The paper examines the learned source-to-target relationships and shows that the gains don't come only from copying the matching Q, K, V, and O projections. The ablations also show that the structured base transformation provides most of the improvement, with the residual correction giving a smaller additional gain.
-
The paper is technically detailed and transparent. The appendix provides useful information about parameter counts, recovery schedules, evaluation protocols, runtime settings, and the cost of mapper training.
Critical weaknesses
-
The recovery-efficiency comparison does not account for the cost of learning the weight transformation. LinMorph requires two mapper-fitting stages, reported to take about 23 A800 GPU hours. However, the comparison with random and manual initialization considers only recovery after materialization. The current results therefore show that LinMorph provides a better initialization and reduces subsequent recovery, but they don't establish that it reduces the total cost of converting a model. This is important because improved conversion efficiency is a central motivation of the paper. A shared mapper may be reused across conversion depths or recovery configurations, which could amortize its cost, but this benefit is not quantified.
-
The resulting GDN architecture has a considerably larger parameter count and memory footprint than the original model. The converted 21-GDN model is more than three times larger than the original denoiser and requires substantially more peak memory. It is also slower in the short-video setting and becomes faster only for longer sequences. Furthermore, the model retains nine softmax-attention blocks and therefore its total complexity remains quadratic. On the other hand, these concerns mainly reflect the chosen GDN architecture; even though it weakens the broader practical-efficiency argument of the obtained method, it doesn't fully invalidate the generic direction of the paper on learning weight transfers.
Questions for authors
-
Could the authors report the total GPU hours or wall clock cost of mapper fitting and recovery together for LinMorph, manual initialization, and random initialization? A comparison at matched functional fidelity would clarify whether LinMorph reduces the cost of a single conversion. I would consider raising my score if the authors include the weight-transformation cost in the comparison and showed that LinMorph reduces the total compute needed to reach comparable functional alignment or generation quality.
-
Does the learned weight-transformation approach depend strongly on GDN, or could the same formulation be applied to other linear-attention operators? Please clarify which parts of the mapper you expect to be general and which are specific to the GDN parameterization.
Minor remark: the bold-face number indication in the right table under table 2, seems incorrect for the Output NMSE and Output cos.
6 comments
Response to Reviewer 5ztz (Part 2/3)
W2: The hybrid is larger, retains nine softmax layers, and originally accelerates only longer videos.
Replacing 21 of 30 softmax layers substantially reduces the quadratic computation and yields a shallower measured runtime curve over the tested video lengths. Retaining a quadratic component does not negate the observed backbone and end-to-end speedups. Our optimized implementation also extends the benefit to 81-frame videos.
(1) Reduced quadratic computation and better measured scaling.
For fixed channel dimensions, the attention contribution changes schematically from to , where is the video-token count and collect fixed-width operator costs. The retained layers preserve a quadratic asymptotic term, but 21 layers no longer incur quadratic token interactions. Thus, the hybrid reduces the quadratic coefficient while introducing linear-cost replacement operations.
The original A800 measurements already show this benefit at the operator, backbone, and full-generation levels:
| Frames | GDN attention-module speedup | Full DiT speedup | End-to-end speedup |
|---|---|---|---|
| 241 | — | — | 1.27× |
| 321 | 2.02× | 1.48× | 1.47× |
| 481 | — | 1.76× | — |
| 641 | 3.91× | — | — |
“—” denotes a value not included in this summary. Here attention-module timing measures one complete replacement module, including projections and adapters; it differs from the all-layer H20 timing below.
| Frames | Video tokens | GDN attention-module latency |
|---|---|---|
| 321 | 126,360 | ≈231.7 ms |
| 641 | 251,160 | ≈464.7 ms |
Doubling the token count approximately doubles GDN latency. Despite retaining softmax layers, the hybrid achieves measured backbone and end-to-end speedups; we claim improved practical scaling, not an asymptotically linear backbone.
(2) Optimized H20 execution accelerates short videos as well.
| Frames | All-layer attention speedup | DiT speedup | End-to-end speedup |
|---|---|---|---|
| 81 | 1.273× | 1.205× | 1.195× |
| 161 | 1.771× | 1.612× | 1.610× |
| 241 | 2.068× | 1.897× | 1.850× |
| 321 | 2.272× | 2.097× | 2.068× |
| 401 | 2.418× | 2.265× | — |
| 481 | 2.509× | 2.348× | — |
| 561 | 2.615× | — | — |
| 641 | 2.665× | — | — |
Measurements use one H20 per run, BF16, batch size one, and 480×832 resolution, against Wan2.1 with FlashAttention-2. All-layer attention includes all 30 self-attention layers, including the nine retained softmax layers; DiT measures one full backbone forward. Both use medians over three timing rounds. End-to-end measurements use one timed generation after warmup, with 50 denoising steps and CFG 6, including text encoding and VAE decoding. “—” denotes settings outside the scheduled measurement range.
| Frames | Softmax generation (s) | LinMorph generation (s) | Speedup |
|---|---|---|---|
| 81 | 317.46 | 265.70 | 1.195× |
| 321 | 3,587.60 | 1,734.70 | 2.068× |
The original A800 experiments used Triton GDN kernels; the new H20 implementation enables Hopper-specific FlashQLA kernels. We apply three execution optimizations:
- Kernel adaptation: support the existing K=256/V=512 head dimensions.
- Intra-GPU sequence parallelism: use larger streaming chunks to improve parallel execution.
- Projection composition: combine input/output width adapters with adjacent linear projections at load time, reducing matrix multiplications and intermediate activations.
The checkpoint files and mapper design are unchanged. Full-DiT checks at 81 and 161 frames show relative output RMSE below 2% versus serial execution, reflecting BF16 rounding changes.
These measurements include the retained softmax layers and the expanded GDN projections. The end-to-end advantage is therefore a property of the executed hybrid system, not an inference from operator asymptotics alone. Longer-video measurements assess runtime scaling; generation-quality evaluation uses 81 frames.
(3) Parameter and memory expansion is operator-specific.
The original A800 measurements in Appendix C.7 report the following parameter and full-generation memory costs:
| Resource | Pretrained Wan2.1 | LinMorph, 21 GDN |
|---|---|---|
| Denoiser parameters | 1.42B | 4.59B |
| Peak memory, 81 frames | 15.50 GiB | 28.40 GiB |
| Peak memory, 241 frames | 18.42 GiB | 44.61 GiB |
| Peak memory, 321 frames | 19.87 GiB | 52.72 GiB |
Memory is peak PyTorch allocated memory under the original A800 implementation. These costs arise from the chosen GDN width expansion, additional projections and gates, and width bridges. The mapper itself is removed after materialization.
The GLA experiment below demonstrates that learned weight transfer does not require this expansion: its attention-module parameter increase is only 0.30%, while it retains substantial initialization-alignment gains.
Response to Reviewer 5ztz (Part 3/3)
Q2: Which parts of the formulation are general, and does it apply beyond GDN?
We have fitted a Softmax-to-GLA mapper and verified substantial alignment gains on source-ID-disjoint clips. This tests the shared weight-transfer formulation with a different linear-attention operator.
(1) General framework and operator-specific construction.
| Component | Reuse or adaptation |
|---|---|
| Source-conditioned cross-projection transformation | Reused |
| Sharing the mapper across layers | Reused |
| Local teacher alignment and residual refinement | Reused |
| Target tensor shapes and auxiliary-parameter constructors | Adapted to the operator |
| GDN-specific components | Update/decay parameters, short convolutions, width bridges |
| GLA-specific construction | Q/K/V/O, input-dependent key decay, output gate, normalization |
GLA does not require the GDN delta-rule parameters or our GDN width expansion. The mapper is fitted separately for the new target; the reusable element is the formulation and training procedure.
(2) GLA initialization results.
We use Gated Linear Attention and the same two-stage alignment procedure. Evaluation uses 37 QWorld clips with source-video IDs excluded from mapper training, 17 frames at 448×256, all 30 layers, and five timesteps: 5,550 paired local cases per method, with zero recovery updates. All methods use the same target architecture, and Manual retains nonzero gates.
| Initialization | Attention NMSE ↓ | Block NMSE ↓ | Block cosine ↑ |
|---|---|---|---|
| Random | 1.0187 | 0.09457 | 0.96135 |
| Manual copying | 2.0379 | 0.10842 | 0.95475 |
| LinMorph base | 1.1061 | 0.06025 | 0.97032 |
| LinMorph full | 0.5306 | 0.02320 | 0.98844 |
The full mapper reduces Block NMSE by 75.5% relative to Random and 78.6% relative to Manual, with improvement on each of the 37 clips. Results use one fitted mapper seed.
| Attention module | Parameters | Increase over source |
|---|---|---|
| Source softmax | 9.446M | — |
| GLA replacement | 9.475M | 0.30% |
These results support applicability beyond GDN and show that a large target-parameter expansion is not intrinsic to LinMorph.
Minor remark: Incorrect boldface in the right-hand subtable of Table 2.
Thank you for pointing this out. The lowest Output NMSE and highest Output cosine in the Wan2.2 subtable belong to Manual at 2,500 updates (0.0222 and 0.9889). We will correct the boldface accordingly; the numerical results are unchanged.
Response to Reviewer 5ztz (Part 1/3)
We thank the reviewer for recognizing the novelty of learned weight transfer, our controlled initialization results, and the mapper analysis.
LinMorph achieves better 7-GDN recovery alignment with approximately 30% less total compute, even when charged the entire mapper fit. The same fitted mapper is already reused across the 7-, 14-, and 21-GDN stages.
(1) Better functional fidelity at lower total cost.
All checkpoints use the same architecture and functional evaluation inputs: 64 videos and 5 diffusion timesteps. The totals include the original 23.48 GPU-hours of mapper fitting, using the actual GPU count for each stage: four A800-80GB GPUs for fitting and eight for recovery.
At approximately 120 GPU-hours, LinMorph yields lower Block and Output NMSE than either baseline. Compared with the 3,000-update baselines, it achieves better fidelity using 30.4% less compute than Manual and 30.1% less than Random. This saving does not rely on mapper reuse across configurations.
(2) Full stage accounting and reuse.
Entries are GPU-hours unless marked as updates; totals use unrounded measurements. Each column includes one full mapper fit. Mapper fitting accounts for 19.7%, 6.0%, and 1.8% of these totals, respectively.
The mapper receives each layer's pretrained weights and produces its layer-specific replacement. New replacement layers therefore use the same fitted transformation, while already converted modules inherit their recovered weights. After materialization, all replacement modules are optimized independently.
For K configurations with the same source backbone and target parameterization, the average mapper-fitting cost is Cmapper/K. Sharing the original fit between the 14- and 21-GDN branches, for example, allocates 11.74 GPU-hours to each. The single-conversion comparisons above nevertheless charge the full 23.48 GPU-hours. Updated QWorld fitting with improved data loading and caching takes 15.044 ± 0.072 GPU-hours across three seeds; those new fitting costs are not substituted into the original recovery totals.
(3) Existing evidence for sharing and cross-scale reuse.
The cross-layer experiment in Table 14 supports learning one transformation across the full depth:
Each one-/six-layer mapper is evaluated at all five anchor blocks (0, 6, 12, 18, 24), and results are averaged across mappers; the full-depth mapper uses the same anchors. Thus, the table evaluates cross-layer applicability, rather than only each mapper's fit to its own training layers.
The same residual-refined mapper also transfers from the fitting scale to the larger recovery scale without refitting (Table 13):
Both rows use the original OpenVid evaluation subset, five timesteps, and all 30 layers. These results explain why the one-time mapper fit can support different replacement subsets and larger video shapes.
(4) Complementary generated-video evidence.
We additionally evaluate all three 7-GDN initializations after 1,500 recovery updates with identical generation settings, all 18 VBench-2.0 dimensions, and 3,860 videos per method.
LinMorph improves the total by 6.78 and 6.24 points over Random and Manual, respectively, from unrounded scores. This comparison matches recovery updates and uses one recovery-training run per method; the earlier alignment comparison accounts for total compute directly.