- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
SPT encodes video history into a fixed-length condition for a smaller DiT. Pretraining randomly drops history features and predicts sampled future chunks. During inference, masking is disabled, and the history condition is reused across denoising steps.
Critical strengths
The experiments demonstrate a useful training-efficiency improvement. At 355M parameters, SPT achieves FVD 68.2 with 8.38 EFLOP, compared with 74.1 and 19.27 EFLOP for teacher forcing.
Critical weaknesses
-
History compression and masking lack targeted validation. Although the encoder retains historical KV states, the DiT receives only a fixed-length summary. The history-shuffling experiment does not establish whether distant details survive this bottleneck. Separately, the masking ablation uses a different encoder configuration and lacks comparisons against structured selection at the same retention rate. The information cost of compression and the specific benefit of random masking therefore remain insufficiently isolated.
-
The inference advantage may become practically negligible under few-step generation. The evaluation uses 50 denoising steps, favoring computation moved into an encoder evaluated once per chunk. With few-step distillation, the repeated DiT computation saved becomes much smaller while history-encoding costs remain. Without a matched-quality latency comparison in this setting, the practical benefit for latency-sensitive applications is unclear.
Questions for authors
- Whether this model can keep the history information as good as the original structure? For example, a 360 degree rotation back to the origin.
- Does the latency advantage remain meaningful when both methods use comparable few-step training or distillation?
2 comments
Supplementary response to Reviewer MiDz
Thank you for the specific suggestions on structured selection, sampling steps, and distant-detail retention. We report policy comparisons and quality-latency measurements, followed by controlled history interventions.
History policy in the main architecture.
We compare three policies in the same 355M hierarchy. Independent masking resamples 100 retained positions per chunk; tube masking reuses 100 positions across a clip; dense retains all 1,024. Each run has 10k pretraining updates followed by 2k full-history, next-chunk tuning updates.
| History policy | Pretraining FVD | Post-tuning FVD | Post-tuning LPIPS | Total EFLOP |
|---|---|---|---|---|
| Dense | 65.3 | 67.3 | 0.508 | 20.48 |
| Independent | 91.1 | 68.6 | 0.511 | 8.38 |
| Tube | 76.0 | 76.0 | 0.514 | 8.38 |
FLOP budgets are matrix-operation estimates. Tube is better than independent masking before tuning; independent masking gives lower FVD and LPIPS after the common tuning stage at the same estimated budget. Thus, the pattern comparison favors independent masking for this full training recipe, without establishing universal superiority across stages. Dense retains the best endpoint FVD but costs substantially more.
Q2: Quality and latency at fewer sampling steps.
We evaluate three trained 355M models on 2,048 common clips, generating eight chunks autoregressively from one ground-truth chunk. This is a separate quality-latency evaluation; the history-policy table above uses fixed 2k tuning endpoints. Each entry is FVD / LPIPS / total end-to-end seconds per eight-chunk (128-frame) continuation:
| Steps per chunk | SPT | TF | DF |
|---|---|---|---|
| 4 | 164.0 / 0.497 / 0.91 | 135.6 / 0.497 / 2.23 | 140.4 / 0.496 / 2.22 |
| 8 | 88.1 / 0.500 / 1.27 | 84.0 / 0.501 / 3.84 | 86.8 / 0.502 / 3.85 |
| 16 | 70.7 / 0.506 / 2.06 | 73.6 / 0.506 / 7.05 | 79.7 / 0.507 / 6.96 |
| 50 | 68.2 / 0.511 / 5.44 | 74.0 / 0.511 / 20.70 | 88.3 / 0.514 / 20.65 |
Latency includes codec processing and is the median of eight samples after warm-up on one H100 at batch size 1. Quality evaluation is separate. Steps are counted per chunk: 16 steps means 8 x 16 = 128 denoising evaluations per complete continuation. At this setting, SPT has lower observed FVD than TF, equal reported LPIPS, and a 3.42x speedup. At four steps per chunk, SPT is faster but has worse FVD than both baselines. These measurements characterize a quality-latency tradeoff beyond the 50-step setting; they use sampling-step changes on trained checkpoints and do not substitute for the matched few-step training/distillation comparison you propose.
Q1: Localizing temporal dependence and testing distant content.
For a 775M model, we fix the target and sampling noise on 256 distinct-source clips, each with eight ground-truth history chunks and generation of the ninth:
| History condition | Latent MSE | Difference from chronological | Paired 95% interval for difference |
|---|---|---|---|
| Chronological | 0.5598 | -- | -- |
| Shuffle earliest six; preserve latest two | 0.5599 | +0.0001 | [-0.0025, +0.0024] |
| Swap latest two | 1.0423 | +0.4825 | [+0.4518, +0.5140] |
We observe a clear effect of swapping recent chunks and no clear effect of shuffling early order in this test. Shuffling early chunks preserves their content, so we also test content replacement on 28 distinct-source reappearance candidates selected from 1,235 source-deduplicated held-out windows. Samples and regions are fixed before inference. With the target and sampling noise fixed, we preserve the latest two chunks and replace the target region in the earliest six with corresponding recent pixels; the control intervention replaces an equal-area, non-overlapping region. RGB MSE is measured over the target region in all 16 generated frames.
For 355M SPT, target-region intervention minus control-region intervention gives mean MSE difference +0.00033, with paired bootstrap 95% interval [-0.000005, +0.000656]. However, target intervention minus unmodified history is -0.00141, so this intervention does not increase mean error relative to full history. The SPT-minus-TF difference in the target-versus-control effect is approximately +0.00027, with interval [-0.00018, +0.00078]. Only three candidates are manually confirmed as identifiable same-detail reappearances. We therefore regard the regional test as exploratory. These interventions do not establish distant-detail recovery, equivalence to TF, or 360-degree revisit consistency.
Thank you for highlighting the distinction between access to historical KV states and recovery of distant visual details, and the importance of a matched-quality few-step comparison.
Q1: Historical information and the fixed-length conditioning interface.
Your 360-degree return-to-origin example is a useful test of distant-detail retention. We distinguish that capability from sensitivity to temporal order. To evaluate the latter, we compare next-chunk prediction under chronological and shuffled histories: with eight ground-truth history chunks followed by generation of the ninth, mean latent MSE changes from 0.4055 to 1.2996 under shuffling, a 3.2x increase (Sec. 4.3 and App. E).
During video continuation, the DiT receives M=S conditioning tokens with masking disabled. The condition is recomputed for each new chunk by attending to historical KV states, then reused across that chunk's denoising steps (Sec. 3.6 and App. G). This preserves a path to history while limiting the decoder's readout to a fixed-length interface. These architectural properties explain how history is accessed, but do not guarantee lossless recall. The shuffle changes recent observations as well as distant ones, so its result supports ordered-history use without establishing memory preservation comparable to the baseline on a return-to-origin test. We agree that such a targeted test would strengthen the evidence for information retention.
Masking validation and structured token selection.
To quantify the quality-cost effect of history masking, we compare dense, sparse, and fully masked pretraining within the causal ViT-DiT variant, using a common unmasked continuation-tuning configuration (Fig. 3(a)):
At 90% masking, FVD improves by 12.9% over dense history while per-step pretraining compute decreases by 73.0%. Encoder-denoiser separation is held fixed in this comparison. The 100% endpoint removes observed-history conditioning during pretraining and yields FVD 75.21 after the same tuning configuration, supporting the value of retaining sparse predictive evidence.
The mask-ratio study uses a causal ViT-DiT; the final architecture uses a per-chunk ViT plus Causal Transformer. In the latter, masking follows dense per-chunk encoding, so retained features can include information from other positions in the same chunk (Sec. 3.2). This comparison quantifies the effectiveness of independent random masking relative to dense history and complete masking in the studied architecture. Your proposed same-retention comparison would more directly assess the choice of selection pattern. The mask-ratio study supports the effectiveness of random masking, while its relative merit against structured selection remains unmeasured.
Q2: Does the latency advantage persist with comparable few-step training or distillation?
A matched-quality few-step comparison is the appropriate test for the deployment question you raise. We quantify streaming latency over 16 generated chunks using 50 denoising steps per chunk (Fig. 1(b) and Sec. 4.2):
These measurements give speedups of 5.76x over TF and 5.39x over DF under the evaluated protocol. The encoder performs history-dependent work once per new chunk, while the DiT repeatedly attends to a fixed-length condition. Encoder-update cost and KV memory continue to grow with history.
Eq. (6) makes the step dependence explicit: C_next = C_encoder(H) + N * C_denoiser + C_codec. As N decreases, the once-per-chunk encoder cost accounts for a larger share of the total, so we agree that the observed speedup may shrink. Its magnitude at comparable quality needs measurement with comparable few-step training or distillation; truncating Euler integration alone would not answer your question. The reported speedups apply to the 50-step regime and should not be read as few-step results. The pretraining-compute comparison measures a different benefit: training evaluates one sampled noise time per target, so the 73.0% reduction in per-step pretraining cost does not depend on using 50 inference steps.