acceptodds
Under review as a conference paper at ICLR 2027

KineTra: Fast Video Diffusion by Generating the Weights of Latent-Frame Transitions

Abstract

Video diffusion transformers generate every latent frame as a full grid of spatial tokens, so their cost grows with the number of frames times the frame size, although consecutive frames share most of their content. We propose KineTra, in which the diffusion transformer generates how latent frames change and a small network carries their content forward. KineTra splits the latent video into short segments. For each segment, the transformer generates 256 tokens that form the channel-mixing matrices of a shared executor network; the executor applies these matrices to the last latent frame of the previous segment and returns the latent frames of the current segment, whose last frame starts the next segment. The generated weights therefore encode motion and newly visible content, while appearance travels with the carried frame. A slot encoder that maps videos to these weights is trained with the executor through their own predicted boundary frames, and the transformer then learns to generate the weights with flow matching. A rank bound shows that the number of generated values must grow with the number of independently changing segments, and an error recursion bounds how errors in the carried boundary frame, the only tensor passed between segments, grow along the video. Relative to Dense on SANA-Video 2.0, LTX-Video, and CogVideoX, KineTra generates 129-frame videos 2.81–4.07× faster with VBench Quality changes of −0.05 to +0.15 points, and 961-frame videos up to 7.76× faster on a single GPU.

Primary area:
generative models
Keywords:
video diffusion, transition weights, efficient video generation, generative models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gErzOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents KineTra, an alternative representation for video diffusion transformers. Instead of generating a dense spatial latent grid for every frame, the model predicts an initial latent frame and segment-specific weight tokens. These weights parameterize a shared executor that transforms the carried boundary frame into a nonlinear latent trajectory, whose endpoint is passed to the next segment.

Training proceeds in two stages. The first stage jointly trains a slot encoder and the executor through predicted boundary frames, reducing the mismatch between training and generation. The second stage freezes this representation and trains the pretrained video transformer with rectified flow to generate the initial frame and transition weights. The paper also provides theoretical analyses of representation capacity and boundary-error propagation.

Critical strengths
  1. A meaningful representation-level contribution

KineTra replaces dense frame-wise generation with compact, segment-specific transition weights applied to a carried latent state. This directly addresses temporal redundancy and constitutes a substantive change to the generative representation.

  1. Strong empirical support

The evaluation is broad and carefully controlled. The results consistently support a favorable efficiency–quality trade-off and indicate that the method generalizes beyond a single architecture or setting.

  1. Clear analysis of scope and limitations

The paper explains why the representation can reduce computation and how recurrent boundary errors affect generation. It also identifies regimes in which the method is less effective, making the claims more credible and scientifically useful.

Critical weaknesses
  1. The quality-preservation claim lacks uncertainty and non-inferiority evidence

The paper mainly reports point estimates. Without paired confidence intervals and a non-inferiority margin, failure to detect a difference does not establish quality preservation.

Resolution condition: Report paired seed- and prompt-level intervals and test them against a meaningful degradation margin.

  1. The matched training budget cannot be independently verified

The “equal training budget” comparison is not auditable because absolute updates, processed samples, and compute are not reported. Since the stages have different per-step costs, the fairness of the comparison remains unclear.

Resolution condition: Report per-stage updates, samples, GPU time or FLOPs, and the exact alignment with Dense.

  1. The contribution of generated transition weights is not isolated from simpler alternatives

The ablations do not isolate dynamic weight generation from simpler alternatives: the interpolation baseline is weak, fixed-executor conditioning is absent, and the causal comparison lacks an official few-step Self-Forcing setup with KV-cache.

Resolution condition: Compare these alternatives under matched output formats, hardware, budgets, and quality–efficiency metrics.

Questions for authors
  1. How is the “equal training budget” defined? Please report trainable parameters, updates, processed samples, and GPU hours or FLOPs for Stage 1, Stage 2, and Dense, and state how the budgets are matched.

  2. How does KineTra compare with sparse keyframe generation followed by learned video completion? The linear-interior control is weak. Please add a matched learned completion baseline and report quality, motion, long-video consistency, and end-to-end runtime and memory.

  3. Is dynamic weight generation necessary? Compare the generated matrices with a parameter- and compute-matched fixed executor conditioned through cross-attention or feature modulation. Similar performance would weaken the weight-generation claim.

  4. Can code capacity be separated from executor capacity? The width sweep changes code size, executor capacity, and cost together. Please vary weight-token capacity and executor size independently, reporting quality, runtime, and memory.

  5. Was KineTra fairly compared with Self-Forcing plus KV-cache? Clarify whether KV-cache and the official few-step sampler are used. Otherwise, add a same-hardware comparison with official Self-Forcing and report quality, time, memory, GPU count, and GPU hours.

  6. How sensitive is KineTra to the Stage 1 budget and training-data scale? Vary the Stage 1 allocation and data fraction at fixed total compute, and report quality, long-video consistency, and training cost. Strong scale dependence should be stated as a limitation.

3 comments
AuthorsOpenReview ↗

Response to Reviewer gErz (1/3): non-inferiority and the training budget

(1/3) Non-inferiority and the training budget

Thank you for a careful and generous review. You read KineTra as "a meaningful representation-level contribution", found the evaluation "broad and carefully controlled", and singled out the analysis of scope and limitations as what makes the claims credible. That is the reading we hoped for. Your three resolution conditions were concrete enough to turn directly into experiments; all of them are in the new Appendix H (blue), and the main text now states their outcome. Section and table numbers refer to the revised PDF.

W1. Quality preservation needs intervals and a margin

You are right that point estimates cannot establish preservation. Table 23 (Appendix H.2) now tests every Quality, Semantic, and VMBench change in Tables 1 and 2 against a non-inferiority margin of 0.5 points. The margin is about the Quality cost of the faster training-free caching presets on long LTX videos (Table 20: −0.39 to −0.54), a quality cost that published caching methods accept. We report both intervals you asked for: paired seed-level t intervals over the three within-seed differences, and paired prompt-cluster bootstrap intervals (2,000 resamples, seeds pooled).

  • All 17 prompt-level lower bounds lie above −0.5; the lowest is -0.45.
  • 16 of 17 seed-level lower bounds lie above −0.5. The exception is causal long-video Quality, with seed interval [-0.64, 0.32]. With three seeds the t factor is 4.30, so we claim non-inferiority for the causal model at the prompt level only, and say so in §4.1.
  • On long LTX videos Quality improves, and both intervals exclude zero: seed [0.17, 0.91], prompt [0.27, 0.82].

We also rewrote Appendix D.2: with chance-level rater agreement, the human study is uninformative about parity, so the paper no longer leans on it.

W2 / Q1. The matched budget must be auditable

The budget is matched in A100 GPU-hours, counting video encoding, both stages, validation, and communication (Appendix A.4). Table 21 lists, per body and seed, trainable parameters, updates, processed clips, GPU-seconds per update, and GPU-hours. For LTX:

RunTrainable parametersUpdatesClipsGPU-h
Densefull body36,000576,0002,746.3
KineTra Stage 1slot encoder + executor (59.2M)116,0001,856,000688.9
KineTra Stage 2full body + 4.2M new I/O projections132,0002,112,0002,054.6

KineTra's total stays within 0.2% of Dense's GPU-hours on all four bodies. Because a Stage-2 update runs the transformer on 3–17× fewer tokens, Stage 2 makes more updates than Dense under the same GPU-hours (3.7× on LTX). This is the asymmetry you pointed to, so we checked whether it leaves Dense undertrained. Table 22 evaluates the saved SANA checkpoints of both models along the budget (three seeds, a fixed 236-prompt VBench subset): Dense gains only +0.04 Quality in the last quarter of its budget, so it is close to converged, and the parity in Table 1 does not come from Dense stopping early. Continuing Dense to KineTra's update count would need several times its budget; the flat end of its curve answers the same question.

AuthorsOpenReview ↗

Response to Reviewer gErz (2/3): simpler alternatives and code vs. executor capacity

(2/3) Simpler alternatives and code vs. executor capacity

W3 / Q2 / Q3. Is dynamic weight generation needed?

We agree this was the main gap, and it was the most useful request in the review. New Table 4 (§4.2; all columns in Table 24) trains the alternatives you named with the same GPU-hour budget, data order, output format, and hardware as Dense and KineTra, and reports quality, motion, long-video consistency, end-to-end time, and memory. These runs use SANA-Video 2.0 and one training seed; Dense and KineTra rows are the same seed.

Variant (SANA, seed 1)TokensDAVIS PSNRQual. 129 fDyn. binding 129 fQual. 961 fSubj. 961 fSec. 961 fGiB 961 f
Dense46,46434.4285.3912.8183.2793.361108.235.79
Keyframes + learned completion11,90433.4485.0410.9182.9493.30331.725.84
Fixed executor + cross-attention8,06433.6985.1912.2683.0693.28251.523.58
KineTra8,06434.0585.5313.0683.2893.45250.823.43

Learned completion (Q2). The keyframe model generates every boundary frame and fills the interior with a spatio-temporal transformer conditioned on both endpoints and the text, with parameters matched to our encoder and executor. It is a far stronger control than the linear interior of Table 3. It still needs 11,904 tokens, runs 1.32× slower than KineTra, and loses 2.15 dynamic binding: without a code for the segment, events between the endpoints can only be guessed.

Fixed executor (Q3). This variant keeps the slot encoder, the same 256 tokens, the carried frame, and the trajectory, and matches shared parameters and executor FLOPs to KineTra within 1%. It comes closest: it trails by 0.34 short-video and 0.22 long-video Quality, about twice the paired seed SD in Table 23, and more clearly by 0.36 dB of reconstruction and 0.80 dynamic binding. A generated 32×32 matrix transforms every feature of the carried frame at every position, while cross-attention adds a mixture of code vectors, and detail lost at one boundary is carried into all later ones. With one seed we treat the Quality gaps as indicative and the reconstruction gap as the firm result; we would extend these controls to more seeds and LTX for the final version.

Table 4 also includes temporal patching and a MAGViT-style compressor at KineTra's token count; both are fast but lose quality or reconstruction (details in our reply to Reviewer 5KnK).

Q4. Code capacity versus executor capacity

Table 25 separates the two where the question matters most, at R = 8, where quality drops. Starting from the existing R = 8 SANA Stage-1 checkpoint, we trained a quarter of the Stage-1 schedule more with the setup unchanged (control), with twice the generated tokens per segment, or with twice the shared executor, and measured reconstruction and executor time:

R = 8 variantDAVIS PSNRExecutor s (961 f)
R = 4 default34.057.2
control (256 tokens, 33.8M)33.496.8
512 tokens33.977.3
2× executor (67.4M)33.5713.6

Doubling the tokens recovers 0.48 of the 0.56 dB lost against R = 4; doubling the executor recovers 0.08 dB at twice its run time. Per-segment code capacity, the quantity in Theorem 1, is the binding constraint. A full code × executor grid with generation at every cell needs a training run per cell; this reconstruction-level version answers the question it was meant to answer.

AuthorsOpenReview ↗

Response to Reviewer gErz (3/3): official Self Forcing, Stage-1 share and data scale, summary

(3/3) Official Self Forcing, Stage-1 share and data scale, summary

Q5. Was KineTra compared with Self Forcing plus KV cache?

Not in the submission, and thank you for making us state this clearly. The causal Dense row of Table 2 uses the Self Forcing weights but denoises all frames jointly for 50 steps under a block-causal mask, without the KV cache or the few-step sampler; it isolates the representation, not the deployed pipeline. Table 29 now adds the released checkpoint with its official four-step chunk-wise sampler and rolling KV cache, on the same A100, output format, and VBench-Long protocol, next to KineTra with four-step distillation:

Configuration (961 frames)Qual.Subj.Sec.GiBGPUsGPU-h per 1,000 videos
Dense, joint 50-step80.9291.107275.762.5048084.1
KineTra, 50-step80.7691.47491.138.041136.4
Official Self Forcing, 4-step, KV cache79.3789.82131.121.64136.4
KineTra + 4-step DMD80.3991.38195.938.11154.4

The two new rows use all 946 VBench-Long prompts (one sampling seed for Self Forcing, training seed 1 for KineTra + DMD). Official Self Forcing is 1.49× faster than four-step KineTra and streams its first chunk, which KineTra cannot, since it generates all weights jointly. Over 961 frames, far beyond its five-second training clips, its Quality is 1.02 points and its subject consistency 1.56 points lower. Both facts are in §4.4: the deployed causal pipeline wins on latency, and KineTra keeps long-video quality.

Q6. Sensitivity to the Stage-1 budget and data scale

Rather than retraining both stages at several splits and data fractions, Table 26 tests the part that Stage 1 controls, the representation, on SANA. The saved checkpoint at half of the Stage-1 updates reconstructs DAVIS at 33.83 dB instead of 34.05. Training Stage 1 from scratch on 2.5% of the unique sources costs 0.64 dB on out-of-domain DAVIS, more than KineTra's whole gap to its VAE, and 0.31 dB on held-out MiraData; at 10% most of the loss is gone. Since generated weights cannot reconstruct more than encoded ones, this bounds what Stage 2 can reach. As you suggested, §4.4 now states data scale as a limitation; each setting's training cost is that of a Stage-1 run in Table 21.

For the discussion with the AC

  • Quality preservation is now tested, not inferred: 17/17 prompt-level and 16/17 seed-level intervals clear a 0.5-point margin set by the Quality cost of training-free caching (Table 23).
  • The budget is auditable per stage, and the convergence curves show Dense is not stopped early (Tables 21–22).
  • At the same budget, generated weights beat learned completion and a parameter- and FLOP-matched fixed executor, most clearly in reconstruction and dynamic binding (Table 4).
  • The R = 8 limit is code capacity, not executor size (Table 25), as Theorem 1 predicts.
  • The limits you asked about (Self Forcing streaming, data scale) are measured and stated in §4.4.

Your review turned the paper's central claim into something we could test, and the paper is better for it. If these results resolve your concerns, we would be grateful if you would consider raising your score and supporting the paper in the discussion. We are glad to answer anything further.

Reviewer_5KnKOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

KineTra accelerates video diffusion by having the DiT generate transformation weights rather than individual latent frames. It predicts the first latent frame and 256 weight tokens per segment of R frames. These tokens parameterize block-diagonal channel-mixing matrices used by a lightweight shared executor to generate each segment from the preceding boundary frame, maintaining temporal continuity.

Training proceeds in two stages: first, jointly training a slot encoder and executor using predicted boundary frames, then training the DiT to generate the encoded weights through flow matching. The paper also provides a rank bound on representation capacity and a recursive analysis of boundary errors.

Experiments across SANA-Video 2.0, LTX-Video, CogVideoX1.5, and causal Wan show 2.81–4.07× speedups at 129 frames with VBench Quality differences within ±0.15. At 961 frames, speedups reach 4.42× on SANA and 7.76× on LTX using a single GPU. The method is also compatible with caching, sparse attention, and distillation.

Critical strengths
  1. Novel representation for video generation. Instead of generating dense latent frames, KineTra generates segment-specific transition weights that operate on a carried boundary frame. This is an interesting departure from conventional video diffusion and offers a different way to exploit temporal redundancy.

  2. Substantial efficiency improvements. KineTra achieves 2.81–4.07× speedups on 129-frame generation and up to 7.76× on 961-frame generation, with largely preserved VBench scores. The results across SANA-Video, LTX-Video, and CogVideoX demonstrate the practical potential of the approach.

  3. Thorough experimental analysis. The paper includes matched training comparisons, reconstruction studies, component ablations, segment-length analysis, and boundary interventions. The experiments in Table 3 and Appendix F provide useful evidence for the importance of predicted-boundary training and recurrent state propagation.

Critical weaknesses
  1. Missing token-matched baseline. The cost analysis and IRASim results suggest that most efficiency gains come from reducing transformer token count rather than the weight representation itself. For example, LTX uses 131,072 weight values per segment versus approximately 196,608 latent values, with the reduction largely achieved by packing weights into 512-dimensional tokens. A key missing baseline is a Dense model with coarser spatiotemporal patching or temporal latent compression, trained under the same budget and with a comparable token count. ProMAG and DLFR-Gen are evaluated only under different resolutions and frame counts, while content–motion approaches (CMD, REDUCIO) lack direct comparisons. Without a token-matched baseline, it remains unclear whether the proposed representation offers advantages beyond token reduction.

  2. Insufficient evaluation of representation limitations. Most quality metrics differ by less than ~0.2 points, potentially masking meaningful perceptual differences. The human study is also inconclusive, given its near-chance inter-rater agreement (−0.028 to 0.033). More challenging scenarios, such as scene cuts, large disocclusions, rapid camera motion, and new objects entering within a segment, are not adequately tested. Boundary error propagation is another concern: replacing predicted boundaries with reference frames improves Physics-IQ Original by 4.4–6.0 points. Moreover, the reported long-video consistency improvements may partly result from appearance preservation rather than better motion modeling, or from Dense operating at 46k tokens beyond its effective training regime. Stress tests on content-heavy, multi-shot videos and clarification of Dense's training lengths would strengthen the evaluation.

Questions for authors
  1. How does KineTra perform on videos with abrupt scene transitions, rapid camera motion, or large amounts of newly appearing content? Can the authors provide a failure-case analysis grouped by motion complexity?

  2. Can the authors report the measured boundary-error amplification factors from Theorem 2 across different segment lengths? Do these factors explain the observed long-video degradation?

  3. What is the end-to-end training cost of KineTra compared with Dense, including the representation-learning stage? How sensitive are the results to the allocation of training budget between the two stages?

  4. Can the authors clarify the long-video speedup comparisons when Dense requires four GPUs but KineTra uses one, and provide matched-hardware results where feasible?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 5KnK (1/3): token-matched baselines and content-motion methods

(1/3) Token-matched baselines and content–motion methods

Thank you for a thoughtful and precise review. You describe the representation as "an interesting departure from conventional video diffusion", note the "substantial efficiency improvements" across SANA-Video, LTX-Video, and CogVideoX, and point to Table 3 and Appendix F as evidence that predicted-boundary training and recurrent state propagation matter. It means a lot that you read the analysis that closely. Your two concerns, whether the benefit is only token reduction and whether the evaluation hides the representation's limits, were exactly the right ones to press, and we answer both with new matched experiments (Appendix H, blue). Numbers refer to the revised PDF.

W1. A token-matched baseline

You are right that the speed comes from fewer transformer tokens; Eq. (4) says so, and the IRASim result (Table 18) shows no saving when token counts are equal. Your arithmetic is also right: per LTX segment, 131,072 weight values replace 196,608 latent values, only 1.5× fewer, and the 5.8× token reduction comes from packing values into 512-wide tokens. So the real question is whether the weight representation keeps quality at that token count better than simpler ways of reaching it. New Table 4 (§4.2; all columns in Table 24) answers it at the same GPU-hour budget and data order. These runs use SANA-Video 2.0 and one training seed (seed 1 for every row):

Variant (SANA, seed 1)TokensDAVIS PSNRQual. 129 fQual. 961 fSubj.Dyn.Sec. 961 f
Dense46,46434.4285.3983.2793.3668.441108.2
Dense, temporal patch 68,06434.4284.5582.4993.6263.87246.1
3D-conv compressor, 6→18,06432.9584.6982.6393.1965.94248.6
KineTra8,06434.0585.5383.2893.4568.71250.8
  • Six-frame temporal patches reach KineTra's token count and are 2% faster, since there is no executor, but lose 0.84 short-video Quality and 4.57 long-video dynamic degree against Dense.
  • The MAGViT-style compressor has KineTra's token count, token width, and values per latent frame, so it controls exactly for the packing you describe. It loses 1.47 dB of reconstruction against the VAE (KineTra: 0.37), because each compressed frame must encode appearance again; the carried frame is what lets KineTra spend its values on change.

CMD and REDUCIO. Both decode a 16-frame clip from a content frame and a compact motion latent with a fixed decoder, on their own image backbones and resolutions, so a native comparison would mix in backbone, data, and resolution. The matched counterpart in Table 4 is our fixed-executor control: the carried frame as content, the same 256 tokens as the motion code, and a fixed decoder that reads them through cross-attention, as REDUCIO's decoder reads its content features, with parameters and FLOPs matched within 1%. Against generated weights it loses 0.36 dB of reconstruction and 0.80 dynamic binding, and trails by 0.22 in long-video Quality. §2 now states this correspondence.

AuthorsOpenReview ↗

Response to Reviewer 5KnK (2/3): limits of the representation, boundary errors, long-video consistency

(2/3) Limits of the representation, boundary errors, long-video consistency

W2a. Differences below 0.2 points may hide perceptual changes

A fair concern. We now test the changes rather than read them: Table 23 compares every Quality, Semantic, and VMBench change against a 0.5-point margin, about the Quality cost of the faster training-free caching presets, with paired seed- and prompt-level 95% intervals; all 17 prompt-level and 16 of 17 seed-level lower bounds lie above −0.5. On the human study you are right, and we no longer use it as support: Appendix D.2 now says that with agreement between −0.028 and 0.033 it cannot resolve small differences.

W2b / Q1. Hard content and a failure analysis by motion complexity

Table 30 reconstructs held-out 129-frame MiraData windows with the existing LTX Stage-1 model and encoded weights, which isolates the representation from generation: flow quartiles over a random 2,000-window subset, and 100 windows per event drawn from the top 5% of all 10,000 held-out windows.

GroupVAEKineTraΔ (dB)Fail (Δ < −1.5)
Motion Q1 (slowest)34.6234.41-0.211 / 500
Motion Q4 (fastest)31.4230.83-0.5919 / 500
Rapid camera motion30.6129.87-0.749 / 100
Large disocclusion31.0730.12-0.9514 / 100
Scene cut inside a segment31.2128.37-2.8474 / 100

The gap grows steadily with motion and new content, and a cut inside a segment is a genuine failure mode: the carried frame holds nothing of the new shot, so 256 tokens must describe a whole new frame. In generation (Table 31; 100 prompts per set, one seed), KineTra realizes a prompted cut in 47 of 100 videos against 66 for Dense, while camera motion, reveals, and entering objects change Quality by at most 0.27 points. §4.4 and the Conclusion now state the scene-cut limit.

W2c / Q2. Measured amplification factors across segment lengths

Table 27 measures the path gains of Theorem 2 along rollouts of the existing LTX Stage-1 models with encoded weights (100 held-out 961-frame videos, three seeds, VAE latents as reference):

Training, RMean gainInterior residuale at K/2e at KSubj.
Predicted, 20.970.0640.1710.22492.45
Predicted, 40.940.0810.1520.19693.27
Predicted, 80.890.1290.1600.21191.12
Teacher, 41.030.0840.2070.318n/a

Yes, they explain the degradation. With predicted-boundary training the mean gain stays below one, so boundary errors are damped on average. R = 2 has twice the boundaries and a gain closer to one, and ends with the largest boundary error; R = 8 has the largest interior residual. Together they produce the consistency peak at R = 4. Teacher boundaries push the gain above one and the final error up 1.62-fold, which is the loss in Table 3. The Physics-IQ gain from reference boundaries (Table 16) is an oracle that shows where accuracy is decided, not a gap at deployment.

W2d. Appearance, motion, or Dense outside its regime?

Dense is trained on the same 961-frame windows (Appendix A.4), so 46k tokens is within its training regime. Your suspicion is still partly right: the gain is specific to LTX. On mostly linear SANA, KineTra only matches Dense (Table 2), and there Dense with temporal patches at KineTra's token count raises consistency only by 0.26 while losing 4.57 dynamic degree (Table 4). §4.1 now reads the LTX gain as appearance kept by the carried frame where full attention over 46k tokens loses some consistency, with dynamic degree unchanged (60.37 vs. 60.63), and no longer suggests better motion.

AuthorsOpenReview ↗

Response to Reviewer 5KnK (3/3): training cost, matched hardware, summary

(3/3) Training cost, matched hardware, summary

Q3. End-to-end training cost and the split between stages

Table 21 now reports, per body and seed, updates, processed clips, GPU-seconds per update, and A100 GPU-hours for Dense and for both KineTra stages, with GPU-hours counting video encoding, validation, and communication. The two stages split the matched budget 25/75 by time, and KineTra's total stays within 0.2% of Dense's on all four bodies. For LTX, Dense uses 2,746.3 GPU-hours per seed; Stage 1 uses 688.9 and Stage 2 2,054.6. Stage 1 trains only the slot encoder and executor (59.2M parameters), so its cost per update (21.4 GPU-s) is mostly video encoding and the rollout through predicted boundaries. Because Stage 2 runs on fewer tokens, it makes more updates than Dense; Table 22 follows saved SANA checkpoints of both models along the budget, and Dense gains only +0.04 Quality in its last quarter, so the comparison does not rest on an undertrained Dense.

On the split, rather than retraining both stages at several shares, Table 26 tests the part Stage 1 controls. On SANA, the saved checkpoint at half of the Stage-1 updates reconstructs DAVIS at 33.83 dB instead of 34.05, and Stage 1 trained on 2.5% of the sources loses 0.64 dB out of domain. The representation therefore needs a full Stage 1 and diverse data, which §4.4 now lists as a limitation.

Q4. Long-video speedups when Dense needs four GPUs

Thank you for pressing on this; the one-versus-four-GPU comparison deserved a matched version. On SANA and LTX, all long-video comparisons in Table 2 already use one GPU on both sides. For CogVideoX and the causal model, Dense needs four GPUs to hold 185,856 and 370,176 tokens, so Table 28 reports two matched views:

Body, 961 framesDense, 4 GPUsKineTra, 4 GPUsSame-hardware speedupGPU-second ratio
CogVideoX1.5 5B6347.2 s321.4 s19.7×25.6×
Self Forcing / Wan2.1 1.3B7275.7 s164.3 s44.3×59.3×

KineTra on four GPUs splits transformer tokens and VAE decoding across devices, while its K endpoints stay sequential. On identical hardware the speedup is far larger than the one-versus-four wall-clock ratio in Table 2, since Dense's cost is dominated by full attention over its long token sequence. §4.1 now reports the per-GPU-second ratios next to the wall-clock ones.

For the discussion with the AC

  • At KineTra's token count, temporal patching and a MAGViT-style compressor lose quality or reconstruction at the same budget; the benefit is not token reduction alone (Table 4).
  • The matched counterpart of CMD/REDUCIO, a fixed content–motion decoder on the same tokens, trails generated weights (Table 4).
  • Failure modes are measured, not hidden: the gap grows with motion, and scene cuts inside a segment fail (Tables 30–31, §4.4).
  • Measured boundary gains explain the R trade-off and the teacher-boundary loss (Table 27).
  • The long-LTX consistency gain is now read conservatively, and matched-hardware speedups are reported (§4.1, Table 28).

Your review changed how we describe the long-video result, and the paper is more accurate for it. If the new evidence resolves your concerns, we would be grateful if you would consider raising your score and supporting the paper in the discussion. We are happy to clarify anything else.

I would like to thank the authors for addressing my concerns. I believe the paper has a value for the community and will keep my positive assessment of the paper.

Reviewer_TP2sOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Standard video diffusion transformers process every latent frame as a dense spatiotemporal grid of tokens, leading to expensive computational and memory cost even though consecutive frames contain highly redundant information. To address this problem, the paper propose the method that decouples scene appearance from temporal changes by having the diffusion transformer generate the boundary-frame latents (modeling content) along with the weights of a small executor network (modeling temporal dynamics) rather than predicting full latent grids. The proposed method leverages video DiT to generate the first latent frame and, for each fixed-length (R = 4) segment subsequent of latent frames, 256 weight tokens. The tokens are then reshaped into 32×32 block-diagonal channel-mixing matrices of a shared executor module. The executor acts on the latent boundary frame from the previous segment and produce the segment's frames along an endpoint-preserving nonlinear trajectory. Experiments show that the proposed method enables significant speedups while preserving quality on VBench and achieves comparable reconstruction quality compared to the original base 3DVAE.

Critical strengths
  • The key idea of further compressing 3DVAE latent feature into the weight space of the executor is interesting and technically sound.
  • The proposed method achieve significant token reduction, resulting in downstream DiT acceleration. The proposed method gives 3.44× (129 frames) and 7.76× (961 frames) speedups on LTX-Video 13B.
  • The authors also complement the empirical results with theoretical grounding with the two theorems on local rank bounds establishing the necessity of per-segment weights, and the boundary error recursion formalizing error propagation.
Critical weaknesses
  • The experiment on segment length in Section 4.2 demonstrates that increasing segment length from 4 to 8 frames causes PSNR and subject consistency to drop. Generating only 256 weight tokens per segment may lack the capacity to represent rapid non-linear dynamics over longer temporal horizons, constraining KineTra to short segments (R=4). It's not clear whether the limitation is in the number of tokens, the simplistic (interpolation-based) temporal trajectory model, the executor capacity, or the modulation mechanism. Deeper analysis to pinpoint such capacity/design vs. compression/quality would be helpful to understand the effectiveness of the proposed framework.
  • It is also not clear how the model perform on arbitrary resolutions.
  • Conceptually, the proposed method is a second-stage temporal token compressor, compressing the base 3DVAE latent into a new latent space with fewer number of tokens (with more channels). While the specific design for then new latent space (boundary frame latents + weight-space temporal latents) is interesting, it is not clear why such design is more beneficial compared to simpler alternatives. For example, how would the simple baselines (such as the dense model using coarser temporal patching at matched budget, or applying MAGViTv2-style 3D convolution layers to compress R latent frames into 1 (while expanding the latent channel dimension) and let the executor be the regular 3D-conv decoder reversing that process back to the original latent) perform in comparison to the proposed method?
  • Training cost is not quantified. It's not clear how expensive it is to train the proposed latent compression module. More importantly, it is not clear wether the resulting latent is more difficult for the DiT to learn. Quantitative comparison in training convergence between DiT training with the proposed latent vs. the dense latent baseline would be important to assess the generation-friendliness of the resulting latents.
  • No video results were provided. It is difficult to assess the temporal quality of the reconstructed/generated from frame sequences.
Questions for authors
  • What are the training cost (GPU-hours and iterations) for Stage 1, Stage 2 and the Dense fine-tuning?
  • How do simple baselines such as Dense model with coarser temporal patching or MAGViT-style second-stage latent compression at matched budget compare in time and quality?
  • How does the proposed method perform on videos with different resolutions? Can the resulting composed VAE encode and decode videos with arbitrary lengths and resolutions?
  • How fast does DiT training with the resulting new latent converge compared to DiT training with the original latent and comparable dense baselines?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TP2s (1/2): simpler compressors and the segment-length limit

(1/2) Simpler compressors and the segment-length limit

Thank you for a clear and constructive review. You found the idea of compressing the 3D-VAE latent into the weight space of an executor "interesting and technically sound", highlighted the 3.44× and 7.76× speedups on LTX-Video 13B, and valued the two theorems as theoretical grounding for per-segment weights and boundary error propagation. We are glad those parts came through. Your framing of KineTra as a second-stage temporal compressor was especially helpful: it told us exactly which baselines a skeptical reader would want, and we have now trained them. All new results are in Appendix H (blue); numbers refer to the revised PDF.

W3 / Q2. Coarser temporal patching and a MAGViT-style second-stage compressor

New Table 4 (§4.2; all columns in Table 24) trains both baselines you proposed at the matched GPU-hour budget. These runs use SANA-Video 2.0 and one training seed (seed 1 for every row):

Variant (SANA, seed 1)TokensDAVIS PSNRQual. 129 fDyn. binding 129 fQual. 961 fDyn. 961 fSec. 961 f
Dense46,46434.4285.3912.8183.2768.441108.2
Dense, temporal patch 68,06434.4284.5511.0882.4963.87246.1
MAGViT-style 3D-conv compressor, 6→18,06432.9584.6911.8382.6365.94248.6
KineTra8,06434.0585.5313.0683.2868.71250.8

The compressor follows your description: a causal 3D-convolutional encoder maps six latent frames to one 16×24×512 frame, and its decoder reverses this, so it has KineTra's token count, token width, and values per latent frame. Both baselines match KineTra's speed (patching is 2% faster, having no executor) but not its quality. Patching loses 0.84 short-video Quality and 4.57 dynamic degree, because one token now stands for six latent frames. The compressor loses 1.47 dB of reconstruction against the VAE, against 0.37 dB for KineTra. The reason is the part of the design you found interesting: the compressor must re-encode appearance in every compressed frame, while KineTra carries appearance in the boundary frame and spends its values on change. Table 4 also adds learned keyframe completion and a fixed executor that reads the same tokens by cross-attention; generated weights stay ahead of both at the same budget.

W1. Is the R = 8 drop due to tokens, trajectory, executor, or modulation?

Table 25 separates these on SANA. Starting from the existing R = 8 Stage-1 checkpoint, we trained a quarter of the Stage-1 schedule more with the setup unchanged (control), with twice the tokens per segment, or with twice the executor:

RTokens / segmentExecutorTotal tokensDAVIS PSNRExecutor s
425633.8M8,06434.057.2
8 (control)25633.8M4,22433.496.8
851233.8M8,06433.977.3
825667.4M4,22433.5713.6
  • Tokens: doubling the code at R = 8 recovers 0.48 of the 0.56 dB.
  • Executor: doubling it recovers only 0.08 dB, at twice its run time.
  • Trajectory: it is not interpolation. Eq. (2) adds the term τ(1−τ)U(B, w, τ), which the executor evaluates at every interior time; removing it is the linear-interior ablation in Table 3 (dynamic binding 13.00 → 9.05 on SANA).
  • Modulation: a fixed executor reading the same tokens by cross-attention, at matched parameters and FLOPs, trails generated mixing (Table 4).

So the limit is code capacity per latent frame, as Theorem 1 predicts. Table 27 adds the error side on LTX: R = 8 has the largest interior residual, R = 2 the largest boundary error. §4.2 now states this.

AuthorsOpenReview ↗

Response to Reviewer TP2s (2/2): training cost and convergence, resolution and length, videos, summary

(2/2) Training cost and convergence, resolution and length, videos, summary

W4 / Q1. Training cost of Stage 1, Stage 2, and Dense

Table 21 reports, per body and seed, updates, processed clips, GPU-seconds per update, and A100 GPU-hours (including video encoding, validation, and communication). Per seed on LTX:

RunIterationsA100 GPU-h
Dense fine-tuning36,0002,746.3
KineTra Stage 1 (encoder + executor)116,000688.9
KineTra Stage 2 (transformer)132,0002,054.6

The latent compression module (Stage 1) costs a quarter of the budget, and KineTra's total stays within 0.2% of Dense's on all four bodies.

W4 / Q4. Is the new latent harder for the DiT to learn?

Thank you for this question; it is the right test of generation-friendliness. Table 22 evaluates the saved SANA checkpoints of both models along the matched budget, from the pretrained body to the final model (three seeds, a fixed 236-prompt VBench subset):

SANA, budget fraction00.250.50.751.0
Dense84.6285.0185.1985.3085.34
KineTra (Stage 2 starts at 0.25)n/an/a84.9385.3685.49

KineTra starts behind, because its weight-token input and output projections are new, and closes the gap within Stage 2; in the last quarter Dense gains +0.04 and KineTra +0.13, so both end close to converged. Together with the class-conditional result already in the paper (UCF101 gFVD 54.2–59.9 against 53.2–61.9 for Dense, Table 15), this indicates that generated weights are about as easy to model as dense latents, at a short warm-up cost.

W2 / Q3. Other resolutions and lengths

Yes: the composed VAE accepts any length, through the segment count K, and any resolution, since the slot queries attend to any number of positions and the executor embeds continuous coordinates. The code size is fixed, though, so Table 32 measures what that costs (LTX, 100 held-out long sources, encoder and executor unchanged):

Setting320×512512×768 (train)704×1280961 frames (train)1921 frames
Δ PSNR vs. VAE (dB)-0.15-0.38-0.67-0.50-0.67

The gap grows with resolution and with length, as a fixed code predicts. In zero-shot generation at 704×1280 (Table 33; 236-prompt subset), the speedup rises from 3.44× to 4.53× because Eq. (4) favors larger frames, while KineTra's Quality falls 0.37 points more than Dense's. Higher resolutions call for a larger code, which §4.4 now states.

W5. No video results

Agreed, frames are a limited way to judge temporal quality. The paper already measures what frames cannot show: per-clip consistency and dynamic degree over the full minute (Figure 5), temporal flickering and motion smoothness (Tables 2 and 6), and VMBench temporal coherence and motion smoothness (Table 8), all close to Dense.

For the discussion with the AC

  • Your two simple baselines, temporal patching and a MAGViT-style compressor at KineTra's token count, are fast but lose quality or reconstruction (Table 4).
  • The R = 8 drop is code capacity per latent frame, not executor, trajectory, or modulation (Tables 4, 25).
  • Training cost is reported per stage; the new latent converges within the matched budget (Tables 21–22).
  • Resolution and length limits are measured and stated (Tables 32–33).

Your review sharpened the paper's comparison to the most natural alternatives, and we are grateful for it. If these results address your concerns, we would be thankful if you would consider raising your score and supporting the paper in the discussion. We would be glad to answer any further question.

I would like to thank the authors for their efforts responding to my review. Most of my concerns have been addressed in the authors' responses. I intend to keep my recommendation for acceptance. I still believe video results are important to incorporate though, since numerical metrics (even the ones designed for temporal quality) are often too coarse to reveal all temporal artifacts. I highly recommend the authors to include them in the supplementary materials in their revision.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General response (1/2): thanks, shared points, and list of revisions

(1/2) Thanks, shared points, and list of revisions

We thank the three reviewers and the AC for reviews that were generous and exacting at the same time. Each review engaged with how the representation works, not only with its numbers, and the questions converged on a small set of points that could be tested directly. We have tested them, and every change in the revised PDF is marked in blue.

What the reviewers agreed on

  • The representation is a substantive change: "a meaningful representation-level contribution" (gErz), "an interesting departure from conventional video diffusion" (5KnK), "interesting and technically sound" (TP2s).
  • The efficiency gains are large and hold across bodies: 2.81–4.07× at 129 frames and up to 7.76× at 961 frames (5KnK, TP2s).
  • The study is careful: an evaluation that is "broad and carefully controlled" with its limits stated (gErz), Table 3 and Appendix F as evidence that predicted-boundary training matters (5KnK), and two theorems that give "theoretical grounding" (TP2s).

Shared questions and where they are answered

  1. Is the benefit only token reduction? (5KnK, TP2s, gErz) New Table 4 in §4.2 trains four token-reducing or conditioning alternatives on SANA at the same budget; all columns in Table 24.
  2. Is quality preservation shown, or only not refuted? (gErz, 5KnK) Table 23 tests all 17 quality changes against a 0.5-point non-inferiority margin with paired seed- and prompt-level intervals.
  3. What does training cost, and is the budget fair? (all three) Tables 21–22: updates, clips and GPU-hours per stage, and convergence curves for both models.
  4. Why do longer segments lose quality? (TP2s, gErz) Table 25 separates code capacity from executor capacity at R = 8.
  5. Where does the representation fail? (5KnK, TP2s, gErz) Stress tests, resolutions and lengths, measured boundary gains, official Self Forcing, matched hardware, and data scale (Tables 26–33).

List of changes (blue in the PDF)

  • §2: how KineTra differs from CMD/REDUCIO, and which control in Table 4 is their matched counterpart.
  • §4 setup: budget matched in A100 GPU-hours, same training windows for both models, convergence result.
  • §4.1: non-inferiority result; a corrected reading of the long-LTX consistency gain (it is specific to LTX, and we no longer suggest better motion).
  • §4.2: new Table 4 and its paragraph; the code-capacity result for R = 8.
  • §4.3: measured boundary gains.
  • §4.4: Limitations rewritten with measured numbers (scene cuts, resolution, streaming, data scale, human study). Conclusion: one clause on scene cuts.
  • Appendix A.4: budget definition and long-window length. Appendix D.2: the human study is uninformative about parity, and we say so.
  • New Appendix H (H.1–H.7, Tables 21–33) with all protocols and results.
  • AI-use statement: LLM assistance in drafting the revised text and these responses.

New training runs use SANA and one training seed, and everything else reuses existing checkpoints and outputs; Appendix H states the scale of each analysis. Title and abstract are unchanged, and the main text still ends on page 10. We also corrected transcription errors in a few table cells. Every added experiment checks a claim that the submission already makes; none adds a new claim. Section and table numbers in all our replies refer to the revised PDF (the new Table 4 shifts later table numbers by one).

AuthorsOpenReview ↗

General response (2/2): added experiments at a glance

(2/2) Added experiments at a glance

New training runs use SANA-Video 2.0 with training seed 1 and the matched budget of §4; all other analyses reuse existing checkpoints and outputs (three seeds unless noted).

QuestionExperimentMain resultWhere
Only token reduction?Dense with 6-frame temporal patches, same 8,064 tokens (SANA)2% faster than KineTra but −0.84 Quality, −4.57 dynamic degreeTable 4
MAGViT-style 3D-conv compressor, same tokens and width (SANA)−1.47 dB reconstruction vs. VAE (KineTra −0.37)Table 4
Learned interior?Keyframes + learned completion (SANA)1.32× slower, −2.15 dynamic bindingTable 4
Are generated weights needed?Fixed executor + cross-attention, same tokens, params and FLOPs (SANA)−0.36 dB, −0.80 dynamic binding, −0.22 long-video QualityTable 4
Quality preserved?Paired seed and prompt 95% CIs, margin 0.5 (existing outputs)17/17 prompt-level and 16/17 seed-level lower bounds above −0.5Table 23
Fair budget?Per-stage accounting; convergence of saved checkpoints (SANA)GPU-hours within 0.2%; Dense gains +0.04 in its last quarterTables 21–22
Why does R = 8 lose?2× tokens vs. 2× executor at R = 8 (SANA Stage 1)2× tokens recover 0.48 of 0.56 dB; 2× executor 0.08 dBTable 25
Data scaleStage 1 on 2.5% / 10% of sources (SANA)−0.64 dB out-of-domain reconstruction at 2.5%Table 26
Boundary error growthMeasured gains, R = 2, 4, 8, teacher (LTX)mean gain 0.94 at R = 4; teacher 1.03, final error ×1.62Table 27
Same hardwareKineTra on the 4 GPUs Dense uses19.7× (CogVideoX), 44.3× (causal)Table 28
Official Self Forcing4-step, rolling KV cache1.49× faster than 4-step KineTra, streams; −1.02 Quality at 961 framesTable 29
Failure casesFlow quartiles, camera, disocclusion, scene cut (LTX)Δ from -0.21 to -0.59 dB with motion; -2.84 dB at a cutTables 30–31
Resolution, length320×512 to 704×1280; up to 1921 frames (LTX)gap grows to -0.67 dB; speedup 4.53× at 704×1280Tables 32–33

Two findings changed how we describe the paper, and we want to state them plainly. First, the long-video consistency gain over Dense appears on LTX but not on SANA, so §4.1 now claims appearance preservation on LTX and parity elsewhere, not better motion. Second, a scene cut inside a segment is a real failure mode, and the representation needs data diversity; both are now in the Limitations.

We are grateful to all three reviewers. Their questions made the central claim, that generated transition weights keep quality where simpler token reductions do not, testable in a way the submission was not, and the revised paper is stronger for it. We remain available for any further question during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.