acceptodds
Withdrawn from ICLR 2027

Towards End-to-End Pretraining of Streaming Video Models from Sparse Causal Histories

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We propose Sparse Predictive Transformers (SPT), a simple framework for efficient, end-to-end pretraining of streaming video generators and video representations. A single flow loss jointly trains a ViT, a Causal Transformer, and a DiT to predict a complete chunk sampled from a short future horizon using sparse history features. Streaming generation reuses the history state across denoising steps. At 355M and 775M parameters, SPT achieves lower FVD than teacher-forcing and diffusion-forcing baselines with roughly half their training compute and lower streaming latency. Generative pretraining also improves frozen history-encoder SSv2 linear-probe Top-1 by 12.71 points over random initialization with the same pretrained codec. Pixel-space rendered-text generation and GLUE transfer provide initial evidence for language modeling through video.

Primary area:
generative models
Keywords:
video generation, world model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_MiDzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SPT encodes video history into a fixed-length condition for a smaller DiT. Pretraining randomly drops history features and predicts sampled future chunks. During inference, masking is disabled, and the history condition is reused across denoising steps.

Critical strengths

The experiments demonstrate a useful training-efficiency improvement. At 355M parameters, SPT achieves FVD 68.2 with 8.38 EFLOP, compared with 74.1 and 19.27 EFLOP for teacher forcing.

Critical weaknesses
  1. History compression and masking lack targeted validation. Although the encoder retains historical KV states, the DiT receives only a fixed-length summary. The history-shuffling experiment does not establish whether distant details survive this bottleneck. Separately, the masking ablation uses a different encoder configuration and lacks comparisons against structured selection at the same retention rate. The information cost of compression and the specific benefit of random masking therefore remain insufficiently isolated.

  2. The inference advantage may become practically negligible under few-step generation. The evaluation uses 50 denoising steps, favoring computation moved into an encoder evaluated once per chunk. With few-step distillation, the repeated DiT computation saved becomes much smaller while history-encoding costs remain. Without a matched-quality latency comparison in this setting, the practical benefit for latency-sensitive applications is unclear.

Questions for authors
  1. Whether this model can keep the history information as good as the original structure? For example, a 360 degree rotation back to the origin.
  2. Does the latency advantage remain meaningful when both methods use comparable few-step training or distillation?
2 comments

Thank you for highlighting the distinction between access to historical KV states and recovery of distant visual details, and the importance of a matched-quality few-step comparison.

Q1: Historical information and the fixed-length conditioning interface.

Your 360-degree return-to-origin example is a useful test of distant-detail retention. We distinguish that capability from sensitivity to temporal order. To evaluate the latter, we compare next-chunk prediction under chronological and shuffled histories: with eight ground-truth history chunks followed by generation of the ninth, mean latent MSE changes from 0.4055 to 1.2996 under shuffling, a 3.2x increase (Sec. 4.3 and App. E).

During video continuation, the DiT receives M=S conditioning tokens with masking disabled. The condition is recomputed for each new chunk by attending to historical KV states, then reused across that chunk's denoising steps (Sec. 3.6 and App. G). This preserves a path to history while limiting the decoder's readout to a fixed-length interface. These architectural properties explain how history is accessed, but do not guarantee lossless recall. The shuffle changes recent observations as well as distant ones, so its result supports ordered-history use without establishing memory preservation comparable to the baseline on a return-to-origin test. We agree that such a targeted test would strengthen the evidence for information retention.

Masking validation and structured token selection.

To quantify the quality-cost effect of history masking, we compare dense, sparse, and fully masked pretraining within the causal ViT-DiT variant, using a common unmasked continuation-tuning configuration (Fig. 3(a)):

History maskingContinuation FVDPretraining TFLOPs/step
0%64.632,018.11
90%56.28545.03
100%75.21450.13

At 90% masking, FVD improves by 12.9% over dense history while per-step pretraining compute decreases by 73.0%. Encoder-denoiser separation is held fixed in this comparison. The 100% endpoint removes observed-history conditioning during pretraining and yields FVD 75.21 after the same tuning configuration, supporting the value of retaining sparse predictive evidence.

The mask-ratio study uses a causal ViT-DiT; the final architecture uses a per-chunk ViT plus Causal Transformer. In the latter, masking follows dense per-chunk encoding, so retained features can include information from other positions in the same chunk (Sec. 3.2). This comparison quantifies the effectiveness of independent random masking relative to dense history and complete masking in the studied architecture. Your proposed same-retention comparison would more directly assess the choice of selection pattern. The mask-ratio study supports the effectiveness of random masking, while its relative merit against structured selection remains unmeasured.

Q2: Does the latency advantage persist with comparable few-step training or distillation?

A matched-quality few-step comparison is the appropriate test for the deployment question you raise. We quantify streaming latency over 16 generated chunks using 50 denoising steps per chunk (Fig. 1(b) and Sec. 4.2):

MethodCumulative latency (s)
SPT116.53
TF671.13
DF, four-chunk window628.08

These measurements give speedups of 5.76x over TF and 5.39x over DF under the evaluated protocol. The encoder performs history-dependent work once per new chunk, while the DiT repeatedly attends to a fixed-length condition. Encoder-update cost and KV memory continue to grow with history.

Eq. (6) makes the step dependence explicit: C_next = C_encoder(H) + N * C_denoiser + C_codec. As N decreases, the once-per-chunk encoder cost accounts for a larger share of the total, so we agree that the observed speedup may shrink. Its magnitude at comparable quality needs measurement with comparable few-step training or distillation; truncating Euler integration alone would not answer your question. The reported speedups apply to the 50-step regime and should not be read as few-step results. The pretraining-compute comparison measures a different benefit: training evaluates one sampled noise time per target, so the 73.0% reduction in per-step pretraining cost does not depend on using 50 inference steps.

AuthorsOpenReview ↗

Supplementary response to Reviewer MiDz

Thank you for the specific suggestions on structured selection, sampling steps, and distant-detail retention. We report policy comparisons and quality-latency measurements, followed by controlled history interventions.

History policy in the main architecture.

We compare three policies in the same 355M hierarchy. Independent masking resamples 100 retained positions per chunk; tube masking reuses 100 positions across a clip; dense retains all 1,024. Each run has 10k pretraining updates followed by 2k full-history, next-chunk tuning updates.

History policyPretraining FVDPost-tuning FVDPost-tuning LPIPSTotal EFLOP
Dense65.367.30.50820.48
Independent91.168.60.5118.38
Tube76.076.00.5148.38

FLOP budgets are matrix-operation estimates. Tube is better than independent masking before tuning; independent masking gives lower FVD and LPIPS after the common tuning stage at the same estimated budget. Thus, the pattern comparison favors independent masking for this full training recipe, without establishing universal superiority across stages. Dense retains the best endpoint FVD but costs substantially more.

Q2: Quality and latency at fewer sampling steps.

We evaluate three trained 355M models on 2,048 common clips, generating eight chunks autoregressively from one ground-truth chunk. This is a separate quality-latency evaluation; the history-policy table above uses fixed 2k tuning endpoints. Each entry is FVD / LPIPS / total end-to-end seconds per eight-chunk (128-frame) continuation:

Steps per chunkSPTTFDF
4164.0 / 0.497 / 0.91135.6 / 0.497 / 2.23140.4 / 0.496 / 2.22
888.1 / 0.500 / 1.2784.0 / 0.501 / 3.8486.8 / 0.502 / 3.85
1670.7 / 0.506 / 2.0673.6 / 0.506 / 7.0579.7 / 0.507 / 6.96
5068.2 / 0.511 / 5.4474.0 / 0.511 / 20.7088.3 / 0.514 / 20.65

Latency includes codec processing and is the median of eight samples after warm-up on one H100 at batch size 1. Quality evaluation is separate. Steps are counted per chunk: 16 steps means 8 x 16 = 128 denoising evaluations per complete continuation. At this setting, SPT has lower observed FVD than TF, equal reported LPIPS, and a 3.42x speedup. At four steps per chunk, SPT is faster but has worse FVD than both baselines. These measurements characterize a quality-latency tradeoff beyond the 50-step setting; they use sampling-step changes on trained checkpoints and do not substitute for the matched few-step training/distillation comparison you propose.

Q1: Localizing temporal dependence and testing distant content.

For a 775M model, we fix the target and sampling noise on 256 distinct-source clips, each with eight ground-truth history chunks and generation of the ninth:

History conditionLatent MSEDifference from chronologicalPaired 95% interval for difference
Chronological0.5598----
Shuffle earliest six; preserve latest two0.5599+0.0001[-0.0025, +0.0024]
Swap latest two1.0423+0.4825[+0.4518, +0.5140]

We observe a clear effect of swapping recent chunks and no clear effect of shuffling early order in this test. Shuffling early chunks preserves their content, so we also test content replacement on 28 distinct-source reappearance candidates selected from 1,235 source-deduplicated held-out windows. Samples and regions are fixed before inference. With the target and sampling noise fixed, we preserve the latest two chunks and replace the target region in the earliest six with corresponding recent pixels; the control intervention replaces an equal-area, non-overlapping region. RGB MSE is measured over the target region in all 16 generated frames.

For 355M SPT, target-region intervention minus control-region intervention gives mean MSE difference +0.00033, with paired bootstrap 95% interval [-0.000005, +0.000656]. However, target intervention minus unmodified history is -0.00141, so this intervention does not increase mean error relative to full history. The SPT-minus-TF difference in the target-versus-control effect is approximately +0.00027, with interval [-0.00018, +0.00078]. Only three candidates are manually confirmed as identifiable same-detail reappearances. We therefore regard the regional test as exploratory. These interventions do not establish distant-detail recovery, equivalence to TF, or 360-degree revisit consistency.

Reviewer_P29jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Sparse Predictive Transformers (SPT), an end-to-end framework for streaming video generation and representation learning. SPT uses a ViT to encode each video chunk, a Causal Transformer to integrate sparse historical features, and a conditional DiT to generate a future chunk. A single flow-matching loss trains all three modules. Experiments show lower training compute and lower FVD than teacher-forcing and diffusion-forcing baselines at 355M and 775M parameters. The paper also explores frozen-feature transfer and language modeling with rendered text.

Critical strengths

(1) The proposed method is with a simple & coherent design, where SPT jointly trains history encoding and future generation with a single objective, without auxiliary reconstruction loss or teacher-model distillation.

(2) The proposed method only uses roughly half the training compute of the baselines and reduces streaming generation latency.

(3) Experiments are comprehensive, which studied history masking, future-target horizon, and encoder–denoising capacity allocation.

(4) Results are promising. The frozen history encoder improves Something-Something V2 linear-probe Top-1 accuracy by 12.71 percentage points over random initialization using the same pre-trained codec.

Critical weaknesses

(1) When increasing the model size, the gains achieved by SPT at 775M are modest, and SSIM is slightly lower than teacher forcing.

(2) The results on longer rollouts with generated frames were not listed.

(3) The propose method is a combo of training and architecture changes, and it hards to tell the unique contribution of each change.

Questions for authors

(1) The latest video generation baselines are with much larger model sizes, such as H3, and what's the result on these stronger models?

(2) How does SPT perform over longer rollouts when its history consists of generated rather than ground-truth frames?

(3) What's the effect of sparse-history pre-training from the architectural difference between SPT and the single causal DiT baselines?

2 comments

Thank you for the questions about scaling and contribution attribution. We report the quality-compute comparison, large-model adaptation, and controlled training studies below.

W1: Modest quality gains at 775M and slightly lower SSIM.

We evaluate the quality-compute tradeoff at approximately equal total trainable parameter counts, using shared video data, codec, and continuation protocol. The training budgets and FVDs are (Table 1):

SizeMethodTraining EFLOPFVD
355MTF19.2774.1
355MDF16.2688.3
355MSPT8.3868.2
775MTF36.1553.5
775MDF30.5073.0
775MSPT17.0151.3

At 775M, SPT uses 52.9% less training compute than TF while improving FVD by 4.1%; relative to DF, it uses 44.2% less compute and improves FVD by 29.7%. At 355M, it uses 56.5% less compute than TF and improves FVD by 8.0%. The compute reduction remains substantial at 775M even as the FVD gap to TF narrows.

The SSIM decrease you identify is part of this tradeoff. At 775M, SSIM is 0.346 versus TF's 0.351, LPIPS is 0.510 versus 0.509, and PSNR is 14.99 versus 14.95 dB. We therefore interpret the result as lower FVD at 355M and 775M with much lower training compute and comparable framewise metrics, while acknowledging the decrease in SSIM.

Q1: Applicability to a stronger model such as H3.

We investigate large-model adaptation with an SPT variant of the 33B MiniMax-H3 model (App. C). Starting from the SolarWM H3 Stage-0.5 EMA checkpoint, we use a causal history ViT to condition the generator through cross-attention and train rank-384 LoRA modules in the generator for 3,000 steps. We report held-out camera-pose-controlled image-to-video samples at 768 x 1344 resolution (Fig. 5). This post-training study tests whether the predictive interface can be applied to a much larger pretrained generator. The 33B evidence is qualitative; quantitative training-efficiency conclusions are based on the controlled model-scale comparisons in Table 1.

Q2 and W2: Longer rollouts with generated histories.

Longer generated histories are important for assessing accumulated error. We evaluate continuation by observing 16 frames and autoregressively generating the next 128, or eight chunks (Secs. 3.6 and 4.1). Completed generated chunks are encoded into the history for subsequent predictions, and all quality metrics in Table 1 cover generated frames. The 16-chunk result in Fig. 1(b) measures latency. These measurements establish eight-chunk continuation quality, but do not quantify how quality changes over substantially longer trajectories; your question identifies an important limit of this evaluation.

Q3 and W3: Separating sparse pretraining from architectural changes.

We separate these questions through controlled comparisons. For masking, we fix the causal ViT-DiT architecture, including encoder-denoiser separation, and use a common continuation-tuning configuration. Changing history masking from 0% to 90% improves FVD from 64.63 to 56.28 while reducing pretraining TFLOPs/step from 2,018.11 to 545.03 (Fig. 3(a)). For future-target sampling, we vary K at 90% masking and obtain FVD 56.28/55.41/53.13/53.77 for K=1/2/3/4 (Fig. 3(b)).

For capacity allocation, we compare split encoders at approximately fixed total parameter count. ViT:Causal Transformer:DiT ratios of 0.5:2:1, 0.5:3:1, and 0.5:4:1 yield FVD 58.71, 68.18, and 73.09, with relative inference speedups of 1.00x, 1.26x, and 1.77x (Fig. 3(c)). Together, these comparisons quantify three roles: sparsity improves the training quality-compute tradeoff in the causal ViT-DiT variant, a short future-target horizon improves continuation in that variant, and capacity allocation trades generation quality for inference speed in the split encoder. The encoder configurations differ between these studies (Sec. 3.2); the individual effect sizes are measured within their respective architectures.

AuthorsOpenReview ↗

Supplementary response to Reviewer P29j

Thank you for the questions about computational scaling, architectural attribution, and generated-history continuation. We report budget-near baseline checkpoints, a same-architecture training comparison, and rollout evaluations.

Training compute and model scale.

To examine quality at closer training budgets, we compare SPT with measured earlier TF/DF checkpoints. Each entry below is training EFLOP / FVD:

Model sizeSPTTF, 2.5k updatesDF, 5k updates
125M3.32 / 118.33.68 / 227.24.14 / 169.4
355M8.38 / 68.29.63 / 143.510.84 / 103.8
775M17.01 / 51.318.08 / 80.820.34 / 87.2

These are measured checkpoints rather than interpolated equal-budget estimates. TF/DF use approximately 6%-29% more compute than the corresponding SPT run. At 775M, for example, SPT obtains FVD 51.3 at 17.01 EFLOP versus TF's 80.8 at 18.08 EFLOP. This helps distinguish the compute-efficiency result from the modest FVD gap to the more extensively trained 36.15-EFLOP TF model. The submitted SSIM comparison remains a quality tradeoff, not an across-metric dominance claim.

Separating sparsity from architecture.

Within the same 355M hierarchical architecture, dense pretraining uses 1.707 PFLOP/update and independent masking 0.496 PFLOP/update. At 10k updates, dense obtains FVD 65.3 versus 91.1 for independent masking: dense is better at equal update counts. At nearby pretraining budgets, however, independent masking obtains FVD 94.4 at 3.72 EFLOP versus dense's 201.1 at 4.27 EFLOP. The sparse run has three times as many updates, reflecting the additional optimization and data exposure enabled by lower update cost.

After 2k full-history, next-chunk tuning updates, dense and independent masking obtain FVD 67.3 and 68.6 using total estimated budgets of 20.48 and 8.38 EFLOP, respectively. This places independent masking within 1.3 FVD points at 59.1% lower compute in the main architecture. The tuning curve also quantifies its contribution: independent-mask FVD changes from 91.1 to 73.4/71.8/68.2/68.6 after 500/1,000/1,500/2,000 tuning updates. The fixed 2k endpoint is 68.6; the best evaluated point on this curve is 68.2 at 1.5k tuning updates.

Generated-history continuation and sampling cost.

We evaluate three trained 355M models on 2,048 common clips, observing one chunk and autoregressively generating eight. At 16 denoising steps per chunk:

MethodFVDLPIPSTotal latency per sample (s)
SPT70.70.5062.06
TF73.60.5067.05
DF79.70.5076.96

Latency is the total end-to-end time to generate eight chunks (128 frames) per sample, including codec processing. It is measured separately on one H100 at batch size 1 as the median of eight samples after warm-up. At 16 steps per chunk, each continuation uses 8 x 16 = 128 denoising evaluations. SPT takes 29.2% of TF's time with lower observed FVD and the same reported LPIPS. This is a sampling-step evaluation of trained checkpoints, not a comparison of few-step-distilled models.

We also generate 32 chunks per sequence for 16 shared starting clips and inspect five temporal positions. The examples contain both retained scene structure and early degradation: some SPT scenes remain recognizable at chunk 32, while game/person examples fragment at chunks 8-16. Without ground truth across the full 512 generated frames, these observations are qualitative and do not establish a general long-horizon quality advantage.

Applicability beyond the main model scales.

The 33B H3 adaptation provides qualitative camera-conditioned generation, without a paired quantitative H3 baseline. Separately, an approximately 1.5B natural-video model trained from random initialization in pixel space reaches a best evaluated FVD of 217.0 on 2,048 clips. This tests pixel-space feasibility at a larger scale; its modeling space differs from the latent models, so it is not a controlled quality or efficiency comparison against them.

Reviewer_pHagOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SPT, which separates chunk encoding, causal history integration, and future generation for efficient streaming video modeling. Sparse history features are used during pretraining, while a single flow-matching loss jointly trains the encoder and generator. The method achieves strong quality–compute trade-offs and lower streaming latency.

Critical strengths
  1. The paper presents a clean separation between history encoding and iterative denoising, substantially reducing repeated computation during streaming generation.
  2. Sparse-history pretraining is empirically effective: high masking ratios reduce training cost while maintaining or even improving continuation quality.
  3. The method scales consistently across model sizes and achieves competitive or better video quality with significantly lower training compute.
Critical weaknesses
  1. The paper clearly distinguishes SPT from SCD and MiMo, but the remaining methodological delta appears relatively small. Since encoder–denoiser separation and masked-history modeling are already established, it would be helpful to show more directly that independent sparse-history sampling with a single complete-future loss provides benefits beyond implementation simplicity or compute reduction.
  2. The contribution of the unmasked next-chunk continuation-tuning stage is not isolated.
  3. Long-horizon robustness remains unclear due to teacher-forced training and limited rollout evaluation.
  4. Representation evaluation is still narrow: SSv2 probing and history-order sensitivity do not establish broader dynamics or long-term temporal structure.
Questions for authors
  1. Can the authors provide a dense-history split baseline to better isolate the contribution of sparsity?
  2. How much of the final performance comes from continuation fine-tuning?
  3. What is uniquely enabled by sparse-history pretraining beyond prior encoder–denoiser separation methods?
  4. How does SPT behave under substantially longer rollouts?
  5. Can the authors provide broader evidence for the quality of the learned history representation?
2 comments

Thank you for highlighting the importance of separating the pretraining contribution from architectural changes. We report controlled comparisons and representation evaluations to address these questions.

Q1 and Q3: Dense-history baseline and contribution beyond encoder-denoiser separation.

Our contribution concerns how history representations are pretrained: sparse history integration is supervised by complete future chunks through one sampled-future flow loss, without an auxiliary reconstruction objective. This builds on encoder-denoiser separation in SCD and masked-history modeling in MiMo (Sec. 2). We evaluate whether this training recipe improves continuation quality as well as computation.

To isolate masking, we compare seven mask ratios with the causal ViT-DiT architecture fixed. Each setting follows the same unmasked, next-chunk continuation-tuning configuration. We report the resulting continuation FVD and per-step pretraining cost (Fig. 3(a)):

History maskingContinuation FVDPretraining TFLOPs/step
0%64.632,018.11
30%60.471,399.51
60%59.35908.48
80%56.63653.49
85%56.45597.44
90%56.28545.03
100%75.21450.13

The 90% setting improves FVD by 8.35 (12.9%) over dense history and reduces per-step pretraining compute by 73.0%. Since encoder-denoiser separation is held fixed, the difference measures the contribution of sparse pretraining within this architecture. At 100% masking, observed-history conditioning is removed during pretraining and FVD rises to 75.21 after the common tuning stage. The results favor retaining sparse predictive evidence over either dense history or removing history entirely under this protocol.

The 0% row is the dense-history baseline for the causal ViT-DiT ablation. The main system uses a per-chunk ViT followed by a Causal Transformer (Sec. 3.2). These studies address complementary questions: the former isolates masking within a fixed architecture, while Table 1 evaluates the combined system. We recognize that matching the final split encoder in the dense-versus-sparse comparison would make attribution to sparsity more direct.

We separately vary the future-target horizon at 90% masking. FVD is 56.28, 55.41, 53.13, and 53.77 for K=1, 2, 3, and 4 (Fig. 3(b)). The minimum at K=3 supports supervision from a short range of future targets; the increase at K=4 identifies a tradeoff in extending the prediction horizon.

Q2: Contribution of continuation tuning.

We evaluate pretraining choices through a shared dense next-chunk continuation-tuning protocol (Sec. 4.3). The final FVDs of 64.63, 56.28, and 75.21 for 0%, 90%, and 100% pretraining masking show that the pretraining condition continues to affect generation after tuning. This supports a contribution from the pretraining recipe, but does not give a numerical breakdown of the gain from tuning itself. A matched before-and-after evaluation would be needed to answer that part of your question, and the post-tuning results should be interpreted with this distinction in mind.

Q4: Longer rollouts and generated histories.

We share your concern about error accumulation under generated histories. We evaluate autoregressive continuation by observing 16 frames and generating 128 further frames, or eight chunks. Each completed generated chunk is encoded into the history for subsequent predictions (Secs. 3.6 and 4.1). Table 1 measures quality under this feedback, using only generated frames. The 16-chunk measurement in Fig. 1(b) concerns latency. These results establish performance over eight generated chunks, while leaving substantially longer-horizon quality and the teacher-forcing exposure gap open (App. H).

Q5: Quality of the learned representation.

To assess transfer beyond generation, we probe frozen history representations on SSv2 with the same frozen VideoRAE and probe protocol (Sec. 4.4 and App. F). Top-1 increases from 17.05 +/- 0.08 to 29.76 +/- 0.03, and Top-5 from 37.92 +/- 0.07 to 57.01 +/- 0.10: gains of 12.71 and 19.09 percentage points. The variation is across three classifier seeds. With the codec held fixed, this comparison measures the effect of generative training on the history encoder's action-relevant features. History-order sensitivity provides a complementary test of temporal conditioning. We agree that broader dynamics evaluation would strengthen the representation evidence beyond these targeted tests. Rendered-text GLUE transfer (Table 2) separately tests the predictive interface's cross-modal applicability, using the unmasked configuration in App. B.

AuthorsOpenReview ↗

Supplementary response to Reviewer pHag

Thank you for the suggestions on contribution attribution, continuation tuning, and temporal evaluation. We report comparisons in the main hierarchical architecture and quantify the role of tuning below.

Q1/Q3: Dense-history control in the main architecture.

We compare dense, independently sampled, and tube-masked histories within the same 355M hierarchical architecture. Dense retains 1,024 feature positions per chunk; both sparse policies retain 100. Independent masking resamples positions per chunk, whereas tube masking shares positions within a clip. The 12k endpoints include 10k pretraining updates followed by 2k full-history, next-chunk tuning updates. Compute budgets are training matrix-operation estimates.

History policyPretraining EFLOP10k FVD / LPIPSTotal EFLOP12k FVD / LPIPS
Dense17.0765.3 / 0.51020.4867.3 / 0.508
Independent4.9691.1 / 0.5218.3868.6 / 0.511
Tube4.9676.0 / 0.5228.3876.0 / 0.514

Dense performs better at equal update counts. After the common tuning stage, independent masking is 1.3 FVD points behind dense while using 59.1% less estimated total compute. Thus, the benefit is a quality-compute tradeoff, rather than higher quality per update.

We also compare measured pretraining checkpoints at nearby compute budgets, using the same 2,048 test clips and 50 denoising steps:

History policyUpdatesEFLOPFVDLPIPS
Dense2,5004.27201.10.530
Independent7,5003.7294.40.526
Independent10,0004.9691.10.521

At 3.72 versus 4.27 EFLOP, independent masking uses 12.9% less compute and reduces FVD by 53.1%. The budgets are close, not identical. Sparse training obtains three times as many updates in this comparison, so data exposure differs; this is part of the compute-efficiency benefit and does not establish better sample efficiency. Together, these measurements isolate the masking tradeoff within the main architecture, complementing the causal ViT-DiT mask-ratio study.

Q2: Quantifying continuation tuning.

Starting from the independent-mask 10k checkpoint, we evaluate the tuning trajectory:

Tuning updates05001,0001,5002,000
FVD91.173.471.868.268.6
LPIPS0.5210.5160.5120.5110.511

Tuning is important: the 2k endpoint improves FVD by 22.5, with 17.7 points gained in the first 500 updates. The best evaluated FVD is 68.2 at 1,500 tuning updates; the fixed 2k endpoint used in the policy comparison is 68.6. Restoring full history and switching to next-chunk targets occur together, so this curve measures their joint adaptation effect.

Q4/Q5: Longer rollouts and temporal information.

We generate 32 chunks (512 frames) from each of 16 common starting clips for SPT, TF, and DF, giving 48 trajectories. Inspection of midpoint frames at generated chunks 1/8/16/24/32 shows mixed outcomes: SPT retains recognizable structures in some 32-chunk sequences, but also exhibits fragmentation by chunks 8-16 in others. Full-length ground truth is unavailable, so this is qualitative appearance evidence, not long-horizon FVD or a demonstration of consistently stable dynamics.

To examine temporal-order dependence, we use 256 distinct-source clips with eight ground-truth history chunks and a fixed ninth-chunk target and sampling noise. Shuffling the earliest six chunks while preserving the latest two changes latent MSE by +0.0001 (paired 95% interval [-0.0025, +0.0024]); swapping the latest two changes it by +0.4825 ([+0.4518, +0.5140]). Under these perturbations, we observe a clear recent-order effect and no clear early-order effect. This does not show that early content is irrelevant, nor establish distant-detail recall or broad dynamics understanding. The frozen SSv2 probe provides the separate action-recognition transfer evidence.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.