acceptodds
Under review as a conference paper at ICLR 2027

LinMorph: Learning to Morph Softmax Weights for Efficient Video DiT Linearization

Abstract

Post-hoc linearization is an attractive way to accelerate pretrained Video Diffusion Transformers (Video DiTs), yet existing methods largely treat the quality loss induced by attention replacement as a recovery problem. We show that a more fundamental bottleneck lies upstream: initialization. Random initialization discards pretrained knowledge, whereas manual weight reuse transfers it through rigid and incomplete source–target correspondences. We introduce LinMorph, which recasts Video DiT linearization as a learnable softmax-to-linear weight transformation problem. Instead of copying predefined counterparts, LinMorph learns a cross-parameter morphism that lets each target Gated DeltaNet (GDN) parameter draw jointly from pretrained query, key, value, and output projections, including for parameters with no direct softmax counterpart. A single morphism is shared across layers and learned through local teacher alignment, while producing layer-specific initializations from their respective pretrained weights. The resulting weights are then materialized as independent GDN modules for progressive distillation and joint recovery. Since the morphism operates in weight space rather than token space, it can be learned on short, low-resolution clips and directly reused at larger video scales. Across pretrained Video DiTs, LinMorph enables markedly faster recovery and better final generation quality than random initialization and heuristic weight reuse. Results show that efficient linearization depends not only on how a converted model is recovered, but critically on how pretrained knowledge is transferred before recovery.

Primary area:
generative models
Keywords:
Video Generation, Diffusion Models, Attention Linearization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_5ztzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper the authors study the initialization problem when replacing softmax attention in a pretrained Video DiT with Gated DeltaNet, a popular linear attention architecture. They propose LinMorph, a novel method that learns a shared transformation from the pretrained Q, K, V, and O weights to the GDN parameters. The transformation is shared across layers but produces layer-specific weights. These weights are then materialized as independent GDN modules and further trained through progressive distillation and joint fine-tuning.

Experiments on Wan2.1 and Wan2.2 show that LinMorph provides better initial functional alignment and faster recovery than random initialization and manual weight copying. The recovered hybrid models also achieve competitive video quality and improved inference speed for longer videos.

Critical strengths
  • The paper introduces an interesting and novel direction for post-hoc linearization. Instead of treating the quality loss only as a recovery problem, it learns how to transfer pretrained softmax weights to a different attention architecture. This is a useful and relatively unexplored perspective.

  • The controlled initialization results are convincing. Under the same target architecture and recovery setup, LinMorph reaches better functional alignment than random initialization and manual copying, often with fewer recovery updates.

  • The mapper is analyzed in good detail. The paper examines the learned source-to-target relationships and shows that the gains don't come only from copying the matching Q, K, V, and O projections. The ablations also show that the structured base transformation provides most of the improvement, with the residual correction giving a smaller additional gain.

  • The paper is technically detailed and transparent. The appendix provides useful information about parameter counts, recovery schedules, evaluation protocols, runtime settings, and the cost of mapper training.

Critical weaknesses
  • The recovery-efficiency comparison does not account for the cost of learning the weight transformation. LinMorph requires two mapper-fitting stages, reported to take about 23 A800 GPU hours. However, the comparison with random and manual initialization considers only recovery after materialization. The current results therefore show that LinMorph provides a better initialization and reduces subsequent recovery, but they don't establish that it reduces the total cost of converting a model. This is important because improved conversion efficiency is a central motivation of the paper. A shared mapper may be reused across conversion depths or recovery configurations, which could amortize its cost, but this benefit is not quantified.

  • The resulting GDN architecture has a considerably larger parameter count and memory footprint than the original model. The converted 21-GDN model is more than three times larger than the original denoiser and requires substantially more peak memory. It is also slower in the short-video setting and becomes faster only for longer sequences. Furthermore, the model retains nine softmax-attention blocks and therefore its total complexity remains quadratic. On the other hand, these concerns mainly reflect the chosen GDN architecture; even though it weakens the broader practical-efficiency argument of the obtained method, it doesn't fully invalidate the generic direction of the paper on learning weight transfers.

Questions for authors
  1. Could the authors report the total GPU hours or wall clock cost of mapper fitting and recovery together for LinMorph, manual initialization, and random initialization? A comparison at matched functional fidelity would clarify whether LinMorph reduces the cost of a single conversion. I would consider raising my score if the authors include the weight-transformation cost in the comparison and showed that LinMorph reduces the total compute needed to reach comparable functional alignment or generation quality.

  2. Does the learned weight-transformation approach depend strongly on GDN, or could the same formulation be applied to other linear-attention operators? Please clarify which parts of the mapper you expect to be general and which are specific to the GDN parameterization.

Minor remark: the bold-face number indication in the right table under table 2, seems incorrect for the Output NMSE and Output cos.

6 comments
AuthorsOpenReview ↗

Response to Reviewer 5ztz (Part 1/3)

We thank the reviewer for recognizing the novelty of learned weight transfer, our controlled initialization results, and the mapper analysis.

W1 & Q1: Does LinMorph reduce total conversion cost, and can the mapper be reused across conversion depths?

LinMorph achieves better 7-GDN recovery alignment with approximately 30% less total compute, even when charged the entire mapper fit. The same fitted mapper is already reused across the 7-, 14-, and 21-GDN stages.

(1) Better functional fidelity at lower total cost.

InitializationRecovery updatesTotal GPU-hours ↓Block NMSE ↓Output NMSE ↓
LinMorph1,500119.370.0524510.010125
Manual2,000121.180.1146300.018132
Random2,000120.740.1361950.021124
Manual3,000171.580.0904560.013700
Random3,000170.800.1167540.016820

All checkpoints use the same architecture and functional evaluation inputs: 64 videos and 5 diffusion timesteps. The totals include the original 23.48 GPU-hours of mapper fitting, using the actual GPU count for each stage: four A800-80GB GPUs for fitting and eight for recovery.

At approximately 120 GPU-hours, LinMorph yields lower Block and Output NMSE than either baseline. Compared with the 3,000-update baselines, it achieves better fidelity using 30.4% less compute than Manual and 30.1% less than Random. This saving does not rely on mapper reuse across configurations.

(2) Full stage accounting and reuse.

Stage7-GDN recovery14-GDN final model21-GDN final model
Mapper base fitting12.2912.2912.29
Mapper residual fitting11.1911.1911.19
7-GDN distillation95.8895.8895.88
14-GDN distillation—80.2080.20
21-GDN distillation——84.77
Joint fine-tuning—193.281,022.97
Joint fine-tuning updates—4,00020,000
Total GPU-hours119.37392.841,307.31

Entries are GPU-hours unless marked as updates; totals use unrounded measurements. Each column includes one full mapper fit. Mapper fitting accounts for 19.7%, 6.0%, and 1.8% of these totals, respectively.

The mapper receives each layer's pretrained weights and produces its layer-specific replacement. New replacement layers therefore use the same fitted transformation, while already converted modules inherit their recovered weights. After materialization, all replacement modules are optimized independently.

For KK configurations with the same source backbone and target parameterization, the average mapper-fitting cost is Cmapper/KC_{\mathrm{mapper}}/K. Sharing the original fit between the 14- and 21-GDN branches, for example, allocates 11.74 GPU-hours to each. The single-conversion comparisons above nevertheless charge the full 23.48 GPU-hours. Updated QWorld fitting with improved data loading and caching takes 15.044 ± 0.072 GPU-hours across three seeds; those new fitting costs are not substituted into the original recovery totals.

(3) Existing evidence for sharing and cross-scale reuse.

The cross-layer experiment in Table 14 supports learning one transformation across the full depth:

Layers sharing a mapperBlock NMSE ↓Block cosine ↑
1 layer0.203880.94327
6 layers0.207340.94558
30 layers0.015600.99221

Each one-/six-layer mapper is evaluated at all five anchor blocks (0, 6, 12, 18, 24), and results are averaged across mappers; the full-depth mapper uses the same anchors. Thus, the table evaluates cross-layer applicability, rather than only each mapper's fit to its own training layers.

The same residual-refined mapper also transfers from the fitting scale to the larger recovery scale without refitting (Table 13):

Evaluation size, frames × height × widthAttention NMSE ↓Block NMSE ↓Block cosine ↑
17×256×448, fitting scale0.272370.011210.99439
81×480×832, larger scale0.320780.014180.99296

Both rows use the original OpenVid evaluation subset, five timesteps, and all 30 layers. These results explain why the one-time mapper fit can support different replacement subsets and larger video shapes.

(4) Complementary generated-video evidence.

We additionally evaluate all three 7-GDN initializations after 1,500 recovery updates with identical generation settings, all 18 VBench-2.0 dimensions, and 3,860 videos per method.

InitializationVBench-2.0 total ↑
Random46.03
Manual46.56
LinMorph52.81

LinMorph improves the total by 6.78 and 6.24 points over Random and Manual, respectively, from unrounded scores. This comparison matches recovery updates and uses one recovery-training run per method; the earlier alignment comparison accounts for total compute directly.

AuthorsOpenReview ↗

Response to Reviewer 5ztz (Part 2/3)

W2: The hybrid is larger, retains nine softmax layers, and originally accelerates only longer videos.

Replacing 21 of 30 softmax layers substantially reduces the quadratic computation and yields a shallower measured runtime curve over the tested video lengths. Retaining a quadratic component does not negate the observed backbone and end-to-end speedups. Our optimized implementation also extends the benefit to 81-frame videos.

(1) Reduced quadratic computation and better measured scaling.

For fixed channel dimensions, the attention contribution changes schematically from 30aN230aN^2 to 9aN2+21bN9aN^2 + 21bN, where NN is the video-token count and a,ba,b collect fixed-width operator costs. The retained layers preserve a quadratic asymptotic term, but 21 layers no longer incur quadratic token interactions. Thus, the hybrid reduces the quadratic coefficient while introducing linear-cost replacement operations.

The original A800 measurements already show this benefit at the operator, backbone, and full-generation levels:

FramesGDN attention-module speedupFull DiT speedupEnd-to-end speedup
241——1.27×
3212.02×1.48×1.47×
481—1.76×—
6413.91×——

“—” denotes a value not included in this summary. Here attention-module timing measures one complete replacement module, including projections and adapters; it differs from the all-layer H20 timing below.

FramesVideo tokensGDN attention-module latency
321126,360≈231.7 ms
641251,160≈464.7 ms

Doubling the token count approximately doubles GDN latency. Despite retaining softmax layers, the hybrid achieves measured backbone and end-to-end speedups; we claim improved practical scaling, not an asymptotically linear backbone.

(2) Optimized H20 execution accelerates short videos as well.

FramesAll-layer attention speedupDiT speedupEnd-to-end speedup
811.273×1.205×1.195×
1611.771×1.612×1.610×
2412.068×1.897×1.850×
3212.272×2.097×2.068×
4012.418×2.265×—
4812.509×2.348×—
5612.615×——
6412.665×——

Measurements use one H20 per run, BF16, batch size one, and 480×832 resolution, against Wan2.1 with FlashAttention-2. All-layer attention includes all 30 self-attention layers, including the nine retained softmax layers; DiT measures one full backbone forward. Both use medians over three timing rounds. End-to-end measurements use one timed generation after warmup, with 50 denoising steps and CFG 6, including text encoding and VAE decoding. “—” denotes settings outside the scheduled measurement range.

FramesSoftmax generation (s)LinMorph generation (s)Speedup
81317.46265.701.195×
3213,587.601,734.702.068×

The original A800 experiments used Triton GDN kernels; the new H20 implementation enables Hopper-specific FlashQLA kernels. We apply three execution optimizations:

  • Kernel adaptation: support the existing K=256/V=512 head dimensions.
  • Intra-GPU sequence parallelism: use larger streaming chunks to improve parallel execution.
  • Projection composition: combine input/output width adapters with adjacent linear projections at load time, reducing matrix multiplications and intermediate activations.

The checkpoint files and mapper design are unchanged. Full-DiT checks at 81 and 161 frames show relative output RMSE below 2% versus serial execution, reflecting BF16 rounding changes.

These measurements include the retained softmax layers and the expanded GDN projections. The end-to-end advantage is therefore a property of the executed hybrid system, not an inference from operator asymptotics alone. Longer-video measurements assess runtime scaling; generation-quality evaluation uses 81 frames.

(3) Parameter and memory expansion is operator-specific.

The original A800 measurements in Appendix C.7 report the following parameter and full-generation memory costs:

ResourcePretrained Wan2.1LinMorph, 21 GDN
Denoiser parameters1.42B4.59B
Peak memory, 81 frames15.50 GiB28.40 GiB
Peak memory, 241 frames18.42 GiB44.61 GiB
Peak memory, 321 frames19.87 GiB52.72 GiB

Memory is peak PyTorch allocated memory under the original A800 implementation. These costs arise from the chosen GDN width expansion, additional projections and gates, and width bridges. The mapper itself is removed after materialization.

The GLA experiment below demonstrates that learned weight transfer does not require this expansion: its attention-module parameter increase is only 0.30%, while it retains substantial initialization-alignment gains.

AuthorsOpenReview ↗

Response to Reviewer 5ztz (Part 3/3)

Q2: Which parts of the formulation are general, and does it apply beyond GDN?

We have fitted a Softmax-to-GLA mapper and verified substantial alignment gains on source-ID-disjoint clips. This tests the shared weight-transfer formulation with a different linear-attention operator.

(1) General framework and operator-specific construction.

ComponentReuse or adaptation
Source-conditioned cross-projection transformationReused
Sharing the mapper across layersReused
Local teacher alignment and residual refinementReused
Target tensor shapes and auxiliary-parameter constructorsAdapted to the operator
GDN-specific componentsUpdate/decay parameters, short convolutions, width bridges
GLA-specific constructionQ/K/V/O, input-dependent key decay, output gate, normalization

GLA does not require the GDN delta-rule parameters or our GDN width expansion. The mapper is fitted separately for the new target; the reusable element is the formulation and training procedure.

(2) GLA initialization results.

We use Gated Linear Attention and the same two-stage alignment procedure. Evaluation uses 37 QWorld clips with source-video IDs excluded from mapper training, 17 frames at 448×256, all 30 layers, and five timesteps: 5,550 paired local cases per method, with zero recovery updates. All methods use the same target architecture, and Manual retains nonzero gates.

InitializationAttention NMSE ↓Block NMSE ↓Block cosine ↑
Random1.01870.094570.96135
Manual copying2.03790.108420.95475
LinMorph base1.10610.060250.97032
LinMorph full0.53060.023200.98844

The full mapper reduces Block NMSE by 75.5% relative to Random and 78.6% relative to Manual, with improvement on each of the 37 clips. Results use one fitted mapper seed.

Attention moduleParametersIncrease over source
Source softmax9.446M—
GLA replacement9.475M0.30%

These results support applicability beyond GDN and show that a large target-parameter expansion is not intrinsic to LinMorph.

Minor remark: Incorrect boldface in the right-hand subtable of Table 2.

Thank you for pointing this out. The lowest Output NMSE and highest Output cosine in the Wan2.2 subtable belong to Manual at 2,500 updates (0.0222 and 0.9889). We will correct the boldface accordingly; the numerical results are unchanged.

Reviewer_s68POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies initialization for post-hoc linearization of video DiTs. LinMorph learns a shared mapper from pretrained Q/K/V/O weights to layer-specific GDN parameters, followed by distillation and joint fine-tuning. Results on two Wan backbones show better functional alignment with fewer recovery updates.

Critical strengths
  1. Initialization is a worthwhile angle on this problem. In Table 2, Wan2.1 block NMSE drops from 0.1359 with manual copying to 0.0525 with LinMorph at the same recovery budget, with improvements in final-output error as well. The gains are substantial, and the Wan2.2 results provide further support.
  2. The shared mapper produces different weights for each layer and can be discarded afterward. The cross-scale experiments and source-assignment interventions add useful analysis beyond recovery curves.
Critical weaknesses
  1. LinMorph receives additional local training, and the zero-recovery copying baseline has a degenerate output gate. The post-training interventions in Figure 4 do not replace a restricted mapper retrained under the same budget, authors may provide a clearer ablation about cross-parameter mixing, additional training, and initialization choices.
  2. Table 3 lacks a direct comparison of the three initializations under the same target architecture and comparable total budgets.
Questions for authors

Please see weaknesses.

5 comments
AuthorsOpenReview ↗

Response to Reviewer s68P (Part 1/2)

We thank the reviewer for recognizing initialization as a useful direction and highlighting the shared mapper and cross-scale analysis.

W1: Separate the effects of initialization choices, additional local training, and cross-parameter mapping.

We have added Manual initialization followed by direct local fitting under the same objective and base-stage update budget, using nonzero gates. This control directly tests what manually initialized modules can achieve when given the local fitting stage.

(1) Nonzero-gate initialization and matched local fitting.

Manual copies Q/K/V/O with width adaptation, retains native nonzero initialization for unmatched parameters, and then directly optimizes all 30 independent GDN modules and their boundary adapters. The comparison fixes:

SettingShared by both fitting methods
Target architectureGDN, hidden width 3,072, 24 heads
Training dataSame QWorld data-construction protocol
Local objectiveAttention-output MSE + block-output MSE
OptimizationSame learning rate; 4,096 updates / 16,384 exposures
EvaluationSame 64 clips, five timesteps, all 30 layers, 17×256×448

Both methods are evaluated on the same 9,600 paired local cases. The untrained Manual row below also uses nonzero gates.

MethodAttention NMSE ↓Block NMSE ↓Block cosine ↑
Manual, before local fitting1.0079780.0945780.962799
Manual + direct local fitting0.1446540.0059730.997023
LinMorph, base fitting0.2755090.0110470.994484

Manual before fitting and LinMorph report three-seed means; directly fitted Manual uses one training seed. Direct fitting substantially improves Manual and achieves lower local error than the base mapper. This shows that local fitting itself contributes to alignment and that a degenerate output gate is not required for the comparison.

(2) A shared transformation reaches strong fidelity with a much smaller fitting parameterization.

Fitting methodTrainable parametersLogged GPU-hours
Manual + direct local fitting4,820.35M11.32
LinMorph, base fitting84.93M7.83

LinMorph optimizes 56.75× fewer parameters by sharing a source-conditioned transformation across layers. The comparison exposes the accuracy–cost tradeoff: independent per-layer fitting obtains the lowest local error, while the shared mapper achieves strong alignment with a smaller fitting parameterization and lower measured cost. Both produce the same target inference architecture.

AuthorsOpenReview ↗

Response to Reviewer s68P (Part 2/2)

W2: Compare the initializations under the same target architecture and comparable total budgets.

We now report both recovery alignment at comparable total compute and generated-video quality at matched recovery updates, using the same 7-GDN architecture.

(1) Recovery alignment including the full mapper cost.

InitializationRecovery updatesTotal GPU-hours ↓Block NMSE ↓Output NMSE ↓
LinMorph1,500119.370.0524510.010125
Manual2,000121.180.1146300.018132
Random2,000120.740.1361950.021124
Manual3,000171.580.0904560.013700
Random3,000170.800.1167540.016820

GPU-hours include the original 23.48 GPU-hour mapper fit for LinMorph; device counts are four A800-80GB GPUs for fitting and eight for recovery. Evaluation uses the same 64 videos and five timesteps. These are the original recovery runs, separate from the newly trained local control in W1.

At approximately 120 GPU-hours, LinMorph achieves better Block and Output NMSE than either baseline. It also outperforms both 3,000-update baselines while using 30.4% less compute than Manual and 30.1% less than Random.

(2) Controlled VBench-2.0 evaluation.

All methods use 1,500 recovery updates, identical training data and recovery settings, and identical generation prompts and seeds. Generation uses 81 frames at 480×832, UniPC, 50 steps, CFG 6, and shift 8. All 18 dimensions and 3,860 videos per method are complete.

MeasureRandomManualLinMorph
Total46.0346.5652.81
Creativity39.9342.4243.97
Commonsense61.1165.7164.56
Controllability16.6318.3528.82
Human Fidelity63.7755.2869.13
Physics48.7051.0657.55
Total training GPU-hours95.7296.00119.37

VBench-2.0 scores are on a 0–100 scale; higher is better. LinMorph improves total VBench-2.0 by 6.24 points over Manual and 6.78 over Random, calculated from unrounded scores. It improves all five categories over Random and four of five over Manual. These scores use one recovery-training run per initialization.

The cost row makes the additional mapper fitting explicit: the video-quality experiment matches recovery updates, while the first table compares alignment at closely matched total compute. Together, these results demonstrate both better recovery efficiency and a direct generation-quality benefit under controlled settings.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer s68P,

Thank you again for your thoughtful review. Following our rebuttal and additional experiments, we would appreciate hearing whether any of your concerns remain. We briefly summarize our responses below:

  • Additional local training and initialization controls: We added Manual + direct local fitting, using nonzero gates and matching the local objective and base-stage update budget. Direct fitting achieves lower local error than the base mapper, making the contribution of local optimization explicit. LinMorph reaches strong alignment with 56.75× fewer trainable parameters and lower measured fitting cost (7.83 versus 11.32 GPU-hours). We report this accuracy–cost tradeoff directly.

  • Comparable total training budgets: Including the full mapper-fitting cost, LinMorph achieves better Block and Output NMSE than both baselines at approximately 120 GPU-hours. It also outperforms their 3,000-update checkpoints while using 30.4% less compute than Manual and 30.1% less than Random.

  • Direct generation-quality comparison: With the same 7-GDN architecture, training data, and 1,500 recovery updates, LinMorph achieves 52.81 on VBench-2.0, versus 46.56 for Manual and 46.03 for Random. All methods use identical generation prompts, seeds, and sampling settings, with 3,860 videos per method. We explicitly distinguish this matched-update generation comparison from the total-compute recovery-alignment comparison.

We hope these additions clarify the role of local fitting and provide the direct initialization comparisons you requested. We welcome any further questions and would be happy to clarify any remaining concerns.

Thank you for your time and consideration.

Best regards, The Authors

Reviewer_RVEbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes LinMorph, a learned weight transformation for initializing Gated DeltaNet modules when replacing softmax attention in pretrained Video DiTs. The mapper combines pretrained Q/K/V/O weights, is shared across layers, and is followed by progressive distillation and joint fine-tuning. Experiments on Wan2.1 and Wan2.2 report improved feature alignment, fewer recovery updates, competitive VBench scores, and speedups for long videos.

Critical strengths
  1. The paper identifies initialization as an underexplored component of post-hoc attention conversion and proposes a reasonably novel cross-parameter weight-mapping formulation.
  2. The paper includes controlled initialization comparisons on two backbones and provides unusually detailed implementation and runtime information in the appendix.
Critical weaknesses
  1. The efficiency claim is not convincing under end-to-end accounting. The reported reduction in recovery updates excludes 23.48 GPU-hours of mapper fitting. Moreover, at the 81-frame setting used for quality evaluation, the converted model is slower than Wan (213.2 s vs. 166.9 s) and uses substantially more memory (28.4 vs. 15.5 GiB). It also increases the denoiser from 1.42B to 4.59B parameters. Speedups appear only for much longer videos and come with large memory overhead.
  2. The main initialization evidence relies on proxy alignment metrics from one training run per configuration. No matched VBench/VBench-2.0 comparison between LinMorph, random initialization, and manual copying is provided. Comparisons with prior methods also use different prompts, CFG settings, training data, and inference budgets, so they do not establish that the proposed initialization improves final generation quality under controlled conditions.
Questions for authors
  1. When mapper fitting and all recovery stages are included, what is the total GPU-hour cost relative to random and manual initialization under an equal-total-compute budget?
  2. Can the authors report matched generated-video metrics for LinMorph, random initialization, and manual copying across multiple seeds, using identical prompts, CFG, data, and total training compute?
6 comments
AuthorsOpenReview ↗

Response to Reviewer RVEb (Part 1/4)

We thank the reviewer for recognizing our motivation, cross-parameter mapping formulation, and controlled initialization comparisons.

W1 & Q1: Does the conversion remain efficient after including mapper fitting and recovery costs?

Yes. Including the entire mapper-fitting cost, LinMorph achieves better 7-GDN recovery alignment with approximately 30% less training compute. The advantage also holds at closely matched total budgets.

(1) Direct comparison including mapper fitting.

The table below charges LinMorph the full 23.48 GPU-hours for base and residual fitting. All checkpoints use the same 7-GDN architecture and evaluation protocol: 64 videos and five diffusion timesteps. GPU-hours use the actual device count: four A800-80GB GPUs for mapper fitting and eight for recovery.

InitializationRecovery updatesTotal GPU-hours ↓Block NMSE ↓Output NMSE ↓
LinMorph1,500119.370.0524510.010125
Manual2,000121.180.1146300.018132
Random2,000120.740.1361950.021124
Manual3,000171.580.0904560.013700
Random3,000170.800.1167540.016820

LinMorph outperforms both baselines at approximately 120 GPU-hours. It also outperforms their 3,000-update checkpoints while using 30.4% less compute than Manual and 30.1% less than Random. These savings do not depend on amortizing the mapper across multiple conversions.

(2) Complete accounting through the final generation models.

The following breakdown includes both mapper stages, every progressive distillation stage, and final joint fine-tuning. Each column charges one full mapper fit; entries are GPU-hours unless marked as updates.

Stage7-GDN recovery14-GDN final model21-GDN final model
Mapper base fitting12.2912.2912.29
Mapper residual fitting11.1911.1911.19
7-GDN distillation95.8895.8895.88
14-GDN distillation—80.2080.20
21-GDN distillation——84.77
Joint fine-tuning—193.281,022.97
Joint fine-tuning updates—4,00020,000
Total GPU-hours119.37392.841,307.31

Totals use unrounded measurements. The same fitted mapper initializes newly replaced layers at each conversion depth. Its cost accounts for 19.7%, 6.0%, and 1.8% of the respective totals. The approximately 30% comparative saving above concerns the controlled 7-GDN recovery experiment; the other columns provide complete costs for our final 14-/21-GDN models.

(3) Updated mapper-fitting pipeline.

The new QWorld runs use improved data loading and caching, reducing mapper fitting to approximately 15 GPU-hours. We retain the original measured cost in all recovery comparisons above.

Mapper-fitting stageOriginal OpenVid runsNew QWorld runs, three-seed mean
Base fitting12.297.831
Residual fitting11.197.213
Total GPU-hours23.4815.044 ± 0.072

The uncertainty is the sample standard deviation across the three runs.

AuthorsOpenReview ↗

Response to Reviewer RVEb (Part 2/4)

W1: The converted model is slower at 81 frames and has a larger parameter and memory footprint.

Our optimized H20 implementation is faster even at the 81-frame quality-evaluation setting, and its advantage grows with video length. The parameter and memory expansion comes from the chosen GDN architecture, which we account for separately below.

(1) Measured speedups with the existing checkpoint.

FramesAll-layer attention speedupDiT speedupEnd-to-end speedup
811.273×1.205×1.195×
1611.771×1.612×1.610×
2412.068×1.897×1.850×
3212.272×2.097×2.068×
4012.418×2.265×—
4812.509×2.348×—
5612.615×——
6412.665×——

Measurements use one H20 per run, BF16, batch size one, and 480×832 resolution, against Wan2.1 with FlashAttention-2. All-layer attention includes all 30 self-attention layers, including the nine retained softmax layers; DiT measures one full backbone forward. Both use medians over three timing rounds. End-to-end measurements use one timed generation after warmup, with 50 denoising steps and CFG 6, including text encoding and VAE decoding. “—” denotes settings outside the scheduled measurement range.

For direct latency comparison:

FramesMeasurementSoftmax (s)LinMorph (s)Speedup
81All-layer attention2.4571.9301.273×
81DiT forward3.0582.5391.205×
81End-to-end generation317.46265.701.195×
321End-to-end generation3,587.601,734.702.068×

Thus, the optimized implementation reduces 81-frame generation latency by 16.3%, and more than halves it at 321 frames.

(2) Why the short-video result improves.

The original A800 experiments used Triton GDN kernels; the new H20 implementation enables Hopper-specific FlashQLA kernels. We apply three execution optimizations:

  • Kernel adaptation: support the existing K=256/V=512 head dimensions.
  • Intra-GPU sequence parallelism: use larger streaming chunks to improve parallel execution.
  • Projection composition: combine input/output width adapters with adjacent linear projections at load time, reducing matrix multiplications and intermediate activations.

The checkpoint files and mapper design are unchanged. Full-DiT checks at 81 and 161 frames show relative output RMSE below 2% versus serial execution, reflecting BF16 rounding changes.

(3) Parameter and memory costs reflect the target operator.

The original A800 resource measurements in Appendix C.7 are:

ResourcePretrained Wan2.1LinMorph, 21 GDN
Denoiser parameters1.42B4.59B
Peak memory, 81 frames15.50 GiB28.40 GiB
Peak memory, 241 frames18.42 GiB44.61 GiB
Peak memory, 321 frames19.87 GiB52.72 GiB

Memory is peak PyTorch allocated memory during full generation under the original A800 implementation. The larger GDN modules use expanded internal dimensions, additional projections and gates, and width bridges. The mapper is discarded after materialization and adds no inference-time computation.

This expansion is not required by learned weight transfer. Our additional GLA instantiation provides a substantially smaller target parameterization:

Attention moduleParametersIncrease over source
Source softmax9.446M—
GLA replacement9.475M0.30%

The fitted GLA mapper improves Block NMSE by 75.5% over Random and 78.6% over Manual on source-ID-disjoint clips. Together, the H20 timings and GLA experiment show that practical execution and target-operator design can be improved while retaining the learned-initialization framework.

AuthorsOpenReview ↗

Response to Reviewer RVEb (Part 3/4)

W2 & Q2: Do the alignment gains translate to controlled generated-video quality, and are they stable across seeds?

We now provide a direct VBench-2.0 comparison under identical generation settings, alongside three-seed initialization results with nonzero-gate controls.

(1) Same-architecture generated-video comparison.

All methods use the same 7-GDN architecture, training data, and recovery configuration, with 1,500 recovery updates. Generation fixes the prompts, seeds, 81-frame length, 480×832 resolution, UniPC sampler, 50 denoising steps, CFG 6, and shift 8. Evaluation covers all 18 official dimensions and 3,860 videos per method.

MeasureRandomManualLinMorph
Total46.0346.5652.81
Creativity39.9342.4243.97
Commonsense61.1165.7164.56
Controllability16.6318.3528.82
Human Fidelity63.7755.2869.13
Physics48.7051.0657.55
Total training GPU-hours95.7296.00119.37

VBench-2.0 scores are on a 0–100 scale; higher is better. LinMorph improves the total by 6.24 points over Manual and 6.78 over Random, calculated from unrounded scores. It improves all five categories over Random and four of five over Manual.

This comparison matches recovery updates; mapper fitting is included in the cost row. The preceding total-compute table separately establishes the recovery-alignment advantage at comparable GPU-hours. Generated-video scores use one recovery-training run per initialization.

(2) Three-seed initialization stability.

We train the mapper with three independent seeds and evaluate Manual and Random with three seeds each. Manual copies Q/K/V/O with shape adaptation and retains native nonzero initialization for GDN-specific parameters. Manual and Random share those parameters within each seed.

Evaluation uses identical inputs: 64 held-out videos, five diffusion timesteps, and all 30 layers, yielding 9,600 local comparisons per checkpoint. Values below are mean ± sample standard deviation across the three seed-level means.

InitializationAttention NMSE ↓Block NMSE ↓Block cosine ↑
Random1.002123 ± 0.0000290.093871 ± 0.0001840.962711 ± 0.000044
Manual1.007978 ± 0.0003990.094578 ± 0.0008050.962799 ± 0.000276
LinMorph base0.275509 ± 0.0020320.011047 ± 0.0001300.994484 ± 0.000060
LinMorph full0.251178 ± 0.0004950.009979 ± 0.0000170.995012 ± 0.000008

The full mapper reduces mean Block NMSE by 89.45% relative to Manual and 89.37% relative to Random, outperforming both in every evaluated seed. These pre-recovery metrics measure the mapper's immediate objective—reproducing the teacher's local features—while the VBench-2.0 experiment above verifies the resulting generation benefit after recovery.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviews

We thank all reviewers for recognizing the value of initialization for Video DiT linearization and our shared weight-transformation formulation. We have added controlled experiments addressing conversion cost, generation quality, initialization controls, practical inference efficiency, and applicability beyond GDN.

  1. Total conversion cost. Including the full mapper fit, LinMorph reaches better 7-GDN recovery alignment with 30.4% less compute than Manual and 30.1% less than Random. The advantage also holds at closely matched total GPU-hours.

  2. Controlled generation quality. With the same 7-GDN architecture and 1,500 recovery updates, VBench-2.0 improves to 52.81, versus 46.56 / 46.03 for Manual / Random, using identical generation settings.

  3. Initialization controls. Three-seed results use nonzero-gate baselines. A new Manual + direct local fitting control matches the local objective and base-stage update budget, explicitly comparing fitting accuracy, trainable parameters, and GPU-hours.

  4. Practical inference efficiency. Optimized single-H20 inference achieves 1.195× end-to-end speedup at 81 frames and 2.068× at 321 frames, using the existing 21-GDN checkpoint.

  5. Beyond GDN. A separately fitted GLA mapper reduces Block NMSE by 75.5% / 78.6% relative to Random / Manual, with only 0.30% more attention-module parameters than the source module.

The generation comparison matches recovery updates; the total-compute comparison measures recovery alignment. Three-seed experiments assess initialization stability. We distinguish these settings in the detailed responses below.

Changes in the revised PDF. We have incorporated the additional evidence and clarifications into the revised manuscript, with new and revised content highlighted in blue:

  1. Training-cost accounting: The main text now summarizes recovery efficiency including mapper fitting. The appendix provides checkpoint-level GPU-hours, the complete 14-/21-GDN training-stage breakdown, mapper reuse, and updated fitting costs.

  2. Controlled generation quality: We added a three-way 7-GDN VBench-2.0 comparison to the main text, with category scores, generation settings, and training-budget details in the appendix.

  3. Initialization controls and stability: We added three-seed statistics and the Manual + direct local fitting control, explicitly distinguishing initialization variability from recovery-training and generation results.

  4. Inference efficiency: We added H20 speedups and explained the A800/Ampere versus H20/Hopper implementation difference. The appendix details the optimizations, timing scopes, numerical checks, and full results.

  5. Generality beyond GDN: We added the GLA experiment, including alignment results, parameter counts, and the distinction between reusable mapping components and operator-specific construction.

  6. Evaluation and presentation clarifications: We clarified the two VBench static-prompt corrections and corrected the best-value boldface in the initialization-ablation table without changing its numerical results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.