acceptodds
Under review as a conference paper at ICLR 2027

Inertial Video World Models

Abstract

The physical world has inertia: it evolves from what already exists rather than being recreated at every instant. However, many existing video world models lack an explicit mechanism for inertial state evolution, leaving the continuity of motion and structure implicit in video prediction. During autoregressive rollouts, repeated reconstruction can accumulate errors and destabilize the predicted dynamics. To address this limitation, we introduce , an Inertial Video World Model that makes inertia a structural prior for autoregressive prediction: existing scene evidence is carried across time rather than reconstructed at every step. Central to this formulation is Inertial Memory, which turns memory from passive predictive context into an evolving video state with two complementary views: a dense Eulerian grid preserves scene-aligned evidence, while persistent Lagrangian carriers bind local content to learned canonical coordinates. At each step, the model transports the carrier content according to action-conditioned motion, predicts the complete next video state from the transported carriers and dense grid, and uses a learned gate to combine transported and newly predicted content into the memory for the next step. Extensive experiments across multiple domains, including robot manipulation, 3D game simulation, and open-world navigation, demonstrate that enables more accurate and stable autoregressive prediction across interactive video domains. Codes link: https://anonymous.4open.science/r/Anonymous_InertialWorld-7165/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_W6F3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

InertialWorld predicts video by transporting persistent content before updating its recurrent memory. A dense grid and regional carriers provide complementary state representations, with action-conditioned motion and learned write gates. Training uses predicted-only autoregressive rollouts. Experiments cover RT-1 manipulation, CSGO, and RECON navigation, with additional motion and policy-evaluation diagnostics.

Critical strengths
  1. The carry-then-update transition is a concrete architectural proposal for reducing repeated reconstruction. At 30k steps, the full model improves PSNR from 22.321 to 23.367 dB over the matched Plain AR baseline despite having slightly fewer parameters.

  2. Improvements appear across three domains and persist on diagnostics focused on moving regions. These results support the usefulness of the complete system, though they do not yet isolate its proposed transport mechanism.

Critical weaknesses
  1. The controlled comparison supports the complete Inertial Memory system, but does not isolate transport from the benefit of recurrent memory. The carrier and gate ablations are inference-time interventions, so their drops show reliance on those paths rather than superiority over separately trained alternatives. This limits the mechanistic interpretation of the gains.

  2. The controlled ablation uses one training seed, and the main comparisons use different baseline training recipes and native interfaces. The close-drawer study has only eight trajectories per displayed cell and unpaired real-world references; it should remain preliminary evidence for policy evaluation.

  3. Appendix B.2 lists staged losses for prediction, appearance, transport, binding, rigidity, contact, and deformation without fully specifying their formulas or numerical weights. Reproducing the gains and separating the contribution of transport from the training recipe are therefore difficult. The missing recurrent-memory control and incomplete training specification lead to weak rejection.

Questions for authors
  1. How does the full model compare with a separately trained recurrent dense-memory baseline using the same tokenizer, objectives, history, and compute, but without explicit carrier transport?

  2. Does the matched Plain AR advantage persist across independent training seeds and on CSGO or RECON? Episode-level standard deviations do not establish training stability.

  3. Can you provide the complete loss definitions, weights, and staging schedule from Appendix B.2, and clarify which terms are shared with Plain AR? This is needed to reproduce and interpret the controlled comparison.

9 comments
AuthorsOpenReview ↗

Response to Reviewer W6F3 — Part 1/5

Dear Reviewer W6F3,

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we respond to your questions point by point.

Q1. The controlled comparison supports the complete Inertial Memory system, but does not isolate transport from the benefit of recurrent memory. The carrier and gate ablations are inference-time interventions, so their drops show reliance on those paths rather than superiority over separately trained alternatives. This limits the mechanistic interpretation of the gains. How does the full model compare with a separately trained recurrent dense-memory baseline using the same tokenizer, objectives, history, and compute, but without explicit carrier transport?

A1. We agreed and removed the inference-time interventions. In the original submission, limited compute prevented us from training additional matched 0.8B variants from scratch, so we used those interventions as preliminary diagnostics. With additional rebuttal compute, we independently trained all four 0.8B-scale models from scratch for the same 30k updates and reported them in revised Table 3. Dense-only was the closest requested control: it shared the frozen tokenizer, 320-token Eulerian scene, 64 interaction slots, four-frame hard-prediction history, data, actions, optimizer, AR8→AR16→AR32 curriculum, full BPTT, batch/GPU topology, update budget, and ordered 1,024-episode evaluation. It removed carriers, canonical content, source pointers, the transported route, and losses defined on those absent variables. Recurrent Memory was a second trained control that used an action-conditioned GRU instead of the structured Eulerian–Lagrangian state. To test the proposed persistence mechanism directly, we compared all variants using the same three ground-truth-track metrics: long-lived content error, post-reappearance content error, and late-track content error. These measured content retention over persistent tracks, recovery after occlusion, and accuracy in the rollout's later half, respectively, rather than generic frame averages dominated by static pixels. All three metrics gave the same ordering: Full performed best, followed by Dense-only, Recurrent Memory, and Plain AR.

ModelLong-lived content ↓Post-reappearance content ↓Late-track content ↓
Plain AR0.5144 ± 0.12750.5144 ± 0.15380.1009 ± 0.0558
Recurrent Memory0.4414 ± 0.12890.4375 ± 0.15250.0935 ± 0.0565
Dense-only0.4247 ± 0.13340.4183 ± 0.16370.0896 ± 0.0573
Full0.4145 ± 0.13280.4036 ± 0.16200.0876 ± 0.0552

Full-favorable paired differences were:

ControlLong-lived Δ [95% CI]Post-reappearance Δ [95% CI]Late-track Δ [95% CI]
Plain AR0.0999 [0.0932,0.1067]0.1108 [0.1007,0.1210]0.0133 [0.0121,0.0145]
Recurrent Memory0.0269 [0.0218,0.0321]0.0339 [0.0260,0.0419]0.0059 [0.0051,0.0067]
Dense-only0.0102 [0.0054,0.0149]0.0147 [0.0072,0.0222]0.0020 [0.0010,0.0032]

All Full-vs-control intervals excluded zero, confirming that Full's advantage over each independently trained control was statistically resolved. Revised Sec. 4.4/Table 3 and Appendix E documented the trained 0.8B-scale controls, metric construction, support, and paired intervals.

AuthorsOpenReview ↗

Response to Reviewer W6F3 — Part 2/5

Q2. The controlled ablation uses one training seed, and the main comparisons use different baseline training recipes and native interfaces. Does the matched Plain AR advantage persist across independent training seeds and on CSGO or RECON? Episode-level standard deviations do not establish training stability.

A2. We agreed that episode-level standard deviations did not measure training-run variation. We therefore trained a second matched Full/Plain AR pair from scratch on RT-1 with seed 1 and another matched pair on CSGO. Within each pair, architecture-specific state aside, all models were at the same 0.8B scale, and the tokenizer, data, prediction targets, optimizer, curriculum, update budget, and evaluation were matched. All controlled ablation runs used the same 30k-update training budget. Full achieved the best result on all three persistence metrics for both RT-1 seeds and on CSGO.

SettingModelLong-lived content ↓Post-reappearance content ↓Late-track content ↓
RT-1, seed 0Plain AR0.5144 ± 0.12750.5144 ± 0.15380.1009 ± 0.0558
RT-1, seed 0Full0.4145 ± 0.13280.4036 ± 0.16200.0876 ± 0.0552
RT-1, seed 1Plain AR0.5172 ± 0.12720.5073 ± 0.15690.1051 ± 0.0579
RT-1, seed 1Full0.4153 ± 0.13540.4053 ± 0.16440.0888 ± 0.0561
CSGO, seed 0Plain AR0.6076 ± 0.07610.5961 ± 0.09150.4864 ± 0.1211
CSGO, seed 0Full0.5679 ± 0.09170.5624 ± 0.10760.4485 ± 0.1200

The paired margins were computed as Plain AR minus Full, so positive values favored Full:

SettingComparisonLong-lived Δ [95% CI]Post-reappearance Δ [95% CI]Late-track Δ [95% CI]
RT-1, seed 0Plain AR − Full0.0999 [0.0932,0.1067]0.1108 [0.1007,0.1210]0.0133 [0.0121,0.0145]
RT-1, seed 1Plain AR − Full0.1019 [0.0952,0.1086]0.1020 [0.0920,0.1121]0.0163 [0.0148,0.0177]
CSGO, seed 0Plain AR − Full0.0396 [0.0343,0.0450]0.0337 [0.0274,0.0401]0.0379 [0.0331,0.0427]

All nine intervals excluded zero. This directly evaluated an independent training seed and a second domain rather than repurposing episode dispersion as seed uncertainty. With two RT-1 seeds, we conservatively stated that the advantage was not specific to seed 0, without claiming that two runs exhausted training variance. These results and episode support were added to Table 3 and Appendix E.

AuthorsOpenReview ↗

Response to Reviewer W6F3 — Part 3/5

Q3. The close-drawer study has only eight trajectories per displayed cell and unpaired real-world references; it should remain preliminary evidence for policy evaluation.

A3. We agreed with both limitations and replaced the original probe. The revised protocol crossed four frozen RT-1 policies with two balanced drawer tasks and 18 requests per policy–task cell, yielding 144 fully annotated trajectories per world model and 720 across five methods. All policies used the same 36 initial requests, and all methods shared the current observation for each request; thereafter each policy saw only that method's generated frame for at most 113 control transitions.

MethodBegin successRT-1-X successRT-1 15% successConverged successOverall MAE to real-world success (pp) ↓
SIMPLER real reference0.0%63.0%79.7%87.1%—
RLVR-World16.7%22.2%38.9%58.3%31.7
Vid2World30.6%47.2%47.2%50.0%29.0
WorldGym19.4%50.0%52.8%63.9%20.6
DriftWorld19.4%16.7%33.3%30.6%42.1
InertialWorld13.9%55.6%58.3%69.4%15.1

We agreed that this remained a preliminary Real2Sim evaluation because the SIMPLER real-world success rates were external aggregates rather than trajectory-paired targets. Within this limitation, all world models were compared under the same protocol, and InertialWorld reproduced the policy-level success profile more faithfully than every baseline, attaining the lowest MAE to real-world success for each policy and overall. We therefore treated this as comparative evidence of preserved policy competence rather than a substitute for paired real-world trials. In revised Sec. 4.3, Fig. 7(a) showed success across the four policies and Fig. 7(b) showed MAE to the SIMPLER reference. For convenient inspection, the figure was made available directly at:

https://anonymous.4open.science/r/InertialWorld_rebuttal-53CC/real2sim_policy_success_and_mae.pdf

Appendix C.5 gave the complete protocol and its limitation.

Reviewer_WBfSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Autoregressive video world models re-synthesise every predicted frame from a fixed observation, so persistent scene content is regenerated at each step and reconstruction errors feed back into the rollout. The paper makes persistence structural. The recurrent state is split into a dense Eulerian grid aligned with scene coordinates and 16 Lagrangian carriers that bind regional content to learned translation, rotation and scale coordinates. Each transition first predicts carrier motion from the action and carries stored content to the new coordinates by forward splatting, then predicts the complete next Dynamic code together with a candidate memory update and a per-cell write gate, and commits the mixture as the next state. A frozen FSQ tokenizer supplies a rollout-level Context code and frame-level Dynamic codes, and training backpropagates through the full predicted-only rollout under a horizon curriculum that grows from 8 to 32 steps. Evaluation compares released checkpoints on RT-1 (1,024 episodes), CSGO (70) and RECON (128) at 32 predicted frames, adds three motion diagnostics, a real-to-sim policy probe and a capacity-matched Plain AR ablation.

Critical strengths
  1. Making the recurrent state an object that is moved before it is written is a structural change rather than a new loss on an unchanged generator. Splitting it into a scene-aligned field and a small set of learned coordinate-bound carriers answers what persistent-memory work usually leaves implicit, namely what persists and where it moves. The ordering carry, predict, gate gives inertia an operational meaning, and it transfers to any autoregressive video backbone.

  2. The protocol is documented to a standard this literature rarely reaches. Episode lists materialise before any metric, each baseline keeps its released interface and mapping, the target tensor is shared, and the 1,015 of 1,024 episodes that lie in a baseline's training subset are stated rather than buried.

Critical weaknesses
  1. No comparison in the paper carries an uncertainty, only per-episode dispersion, and three headline items are then read off rank order. On CSGO the LPIPS win is 0.516 against Vid2World's 0.518 with per-episode standard deviations of 0.086 and 0.084 over 70 episodes, which is a fraction of one standard error and is counted nonetheless as one of the twelve dataset and metric combinations in which the model ranks first. The flow-EPE gain in Table 2, from 1.110 to 1.032, sits at roughly two standard errors if episodes are treated as independent and cannot be resolved from summaries alone. The real-to-sim probe behind question (iii) is four successes in eight trials against six in eight, p=0.61p = 0.61 two-sided, on 72 of 144 labelled trajectories. The abstract promises consistently better prediction and the experiments promise policy discrimination, and these are the margins that would have to carry both. Paired tests over the same episodes and the remaining labels would settle the question at little cost, since every per-episode value the paper already averages is on disk.

  2. The one controlled test of the mechanism, Table 3, does not isolate transport, and its own numbers work against the reading offered in the text. Full and Plain AR at a matched 30k steps differ by 1.05 dB PSNR, on one seed and only on RT-1, which is the paper's entire capacity-controlled evidence for the inertial state. The other two rows are interventions on the trained checkpoint, so the conclusion that disabling the carrier path shows the learned transition relies on persistent carrier transport is not licensed by them. The carrier-disabled model scores 20.49 dB, below Plain AR's 22.32 dB, so the 2.87 dB drop is at least as consistent with a state the network never learned to run without, and the paper concedes this a few sentences later while leaving the stronger claim standing. What would identify the mechanism is a dense-only model trained from scratch at the same budget rather than a disabled checkpoint. As written, a three-part story of dense field, carriers and gate rests on one aggregate comparison and two probes whose drop sizes cannot themselves be ranked.

Questions for authors
  1. Can you report paired tests or bootstrap intervals over the same episodes for the domain-level margins and for Table 2, and say which of the twelve wins survive them?

  2. What does a dense-only variant trained from scratch at 30k steps score? The intervened variant scoring below Plain AR is what makes me ask.

  3. What are the values of λmem\lambda_\mathrm{mem} and λreg\lambda_\mathrm{reg}, how are the memory and transport terms staged in over the first 12k steps, and what is the gate initialised to?

  4. In the real-to-sim probe, do the two RT-1 checkpoints start from identical initial states with the same 40-frame budget, and are the remaining 72 labels available?

7 comments
AuthorsOpenReview ↗

Response to Reviewer WBfS — Part 1/5

Dear Reviewer WBfS,

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we respond to your questions point by point.

Q1. No comparison in the paper carries an uncertainty, only per-episode dispersion, and three headline items are then read off rank order. On CSGO the LPIPS win is 0.516 against Vid2World's 0.518 with per-episode standard deviations of 0.086 and 0.084 over 70 episodes, which is a fraction of one standard error and is counted nonetheless as one of the twelve dataset and metric combinations in which the model ranks first. The flow-EPE gain in Table 2, from 1.110 to 1.032, sits at roughly two standard errors if episodes are treated as independent and cannot be resolved from summaries alone. Can you report paired tests or bootstrap intervals over the same episodes for the domain-level margins and for Table 2, and say which of the twelve wins survive them?

A1. Main-table paired uncertainty. For every baseline in each domain–metric cell, we computed episode-wise paired differences on identical episodes. We used 100,000 paired episode-bootstrap resamples for 95% percentile intervals and 100,000 two-sided paired sign-flip permutations, with Holm correction jointly over all 36 comparisons. Positive margins below always favored InertialWorld: ours minus baseline for PSNR/SSIM and baseline minus ours for LPIPS/MSE/MAE. Unmarked cells passed both criteria; † denoted an unresolved comparison.

DomainBaselinePSNR Δ [95% CI]SSIM Δ [95% CI]LPIPS Δ [95% CI]MSE/MAE Δ [95% CI]
RT-1RLVR-World2.422 [2.330,2.514]0.0592 [0.0574,0.0609]0.0214 [0.0199,0.0228]0.00355 [0.00333,0.00377]
RT-1Vid2World3.050 [2.947,3.154]0.1144 [0.1115,0.1173]0.0967 [0.0949,0.0986]0.00501 [0.00475,0.00529]
RT-1WorldGym2.650 [2.564,2.736]0.0567 [0.0551,0.0584]0.0577 [0.0564,0.0591]0.00435 [0.00412,0.00459]
RT-1DriftWorld3.539 [3.443,3.635]0.1079 [0.1053,0.1104]0.1187 [0.1166,0.1208]0.00650 [0.00622,0.00678]
CSGODIAMOND1.317 [1.016,1.626]0.0521 [0.0407,0.0659]0.0601 [0.0508,0.0698]0.02528 [0.01838,0.03302]
CSGOVid2World1.675 [1.341,2.014]0.0781 [0.0678,0.0890]0.0028 [-0.0065,0.0119]†0.02996 [0.02361,0.03700]
RECONNWM3.676 [3.469,3.882]0.1359 [0.1293,0.1427]0.1060 [0.0991,0.1130]0.07285 [0.06811,0.07762]
RECONRAE-NWM2.039 [1.834,2.246]0.1223 [0.1176,0.1271]0.0571 [0.0511,0.0634]0.03907 [0.03584,0.04233]
RECONVid2World1.107 [0.907,1.311]0.0580 [0.0536,0.0625]0.0146 [0.0079,0.0214]0.02045 [0.01745,0.02358]

Thus 35/36 baseline-specific comparisons passed both analyses. Among the 12 headline domain–metric comparisons against the strongest baseline, 11/12 survived: all four on RT-1, PSNR/SSIM/MAE on CSGO, and all four on RECON. The sole unresolved case was CSGO LPIPS vs. Vid2World (pHolm=0.5599p_{\mathrm{Holm}}=0.5599); we described it as unresolved rather than a confirmed win. RT-1 was our primary setting and was trained for 60k updates, whereas CSGO and RECON were 30k cross-domain case studies intended to assess cross-domain applicability under our compute budget. The full matrix was added as revised Appendix D.3, Table 9.

AuthorsOpenReview ↗

Response to Reviewer WBfS — Part 2/5

A1 continued. Uncertainty in motion diagnostics. We applied the same episode-paired analysis to every baseline and all three motion diagnostics. Margins were baseline minus ours for lower-is-better Flow EPE/Dynamic LPIPS and ours minus baseline for higher-is-better Flow Cosine.

BaselineFlow EPE Δ [95% CI]Flow Cosine Δ [95% CI]Dynamic LPIPS Δ [95% CI]
RLVR-World0.076 [0.054,0.097]0.118 [0.108,0.127]0.071 [0.067,0.075]
Vid2World0.296 [0.264,0.327]0.045 [0.035,0.055]0.104 [0.100,0.108]
WorldGym0.515 [0.479,0.553]0.203 [0.193,0.213]0.112 [0.108,0.116]
DriftWorld0.332 [0.304,0.360]0.135 [0.126,0.144]0.169 [0.165,0.172]

All 12 intervals excluded zero and all comparisons survived joint Holm correction (pHolm=1.2×10−4p_{\mathrm{Holm}}=1.2\times10^{-4}). In revised Sec. 4.2, Fig. 6 showed the full episode distributions for the three motion diagnostics: diamonds denoted episode means and capped bars denoted one episode standard deviation over the common 1,024 episodes; InertialWorld had the lowest Flow EPE and dynamic-region LPIPS and the highest Flow Cosine. The paired-margin table was placed immediately below the figure. For convenient inspection, the figure was made available directly at:

https://anonymous.4open.science/r/InertialWorld_rebuttal-53CC/fig_motion_fidelity.pdf

We also softened the abstract/introduction and main-results wording: the revised paper no longer treated rank order or episode standard deviation as evidence of uncertainty, and it explicitly identified the small CSGO LPIPS margin as statistically unresolved.

AuthorsOpenReview ↗

Response to Reviewer WBfS — Part 3/5

Q2. The one controlled test of the mechanism, Table 3, does not isolate transport, and its own numbers work against the reading offered in the text. Full and Plain AR at a matched 30k steps differ by 1.05 dB PSNR, on one seed and only on RT-1, which is the paper's entire capacity-controlled evidence for the inertial state. The other two rows are interventions on the trained checkpoint, so the conclusion that disabling the carrier path shows the learned transition relies on persistent carrier transport is not licensed by them. The carrier-disabled model scores 20.49 dB, below Plain AR's 22.32 dB, so the 2.87 dB drop is at least as consistent with a state the network never learned to run without, and the paper concedes this a few sentences later while leaving the stronger claim standing. What would identify the mechanism is a dense-only model trained from scratch at the same budget rather than a disabled checkpoint. As written, a three-part story of dense field, carriers and gate rests on one aggregate comparison and two probes whose drop sizes cannot themselves be ranked. What does a dense-only variant trained from scratch at 30k steps score? The intervened variant scoring below Plain AR is what makes me ask.

A2. We agreed that inference-time pathway removal could not establish how a model trained without that pathway would behave. In the original submission, limited compute prevented us from training a complete set of matched 0.8B controls from scratch, so we used inference-time interventions as preliminary diagnostics. With additional rebuttal compute, we removed those interventions and trained four separate 0.8B-scale models from scratch for the same 30k updates. They shared the tokenizer, data, actions, optimizer, BF16 precision, full BPTT, AR8→AR16→AR32 curriculum, global batch, compute topology, and ordered 1,024-episode H32 protocol.

The revised Table 3 directly measured persistent content on ground-truth-defined DINO tracks; lower was better.

ModelLong-lived content ↓Post-reappearance content ↓Late-track content ↓
Plain AR0.5144 ± 0.12750.5144 ± 0.15380.1009 ± 0.0558
Recurrent Memory0.4414 ± 0.12890.4375 ± 0.15250.0935 ± 0.0565
Dense-only0.4247 ± 0.13340.4183 ± 0.16370.0896 ± 0.0573
Full0.4145 ± 0.13280.4036 ± 0.16200.0876 ± 0.0552

All three measures ordered Full, Dense-only, Recurrent Memory, then Plain AR. Most directly, the Full-favorable paired Dense-only differences were 0.0102 [0.0054,0.0149], 0.0147 [0.0072,0.0222], and 0.0020 [0.0010,0.0032]; every 95% paired bootstrap interval excluded zero. This provided a trained, same-scale 0.8B test of the incremental value of carriers rather than a disabled-checkpoint result. Revised Sec. 4.4/Table 3 and Appendix E (Tables 12–13) gave the complete variant contracts, metric construction, episode support, and paired intervals.

Reviewer_r97uOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes InertialWorld to reduce the accumulation of appearance and structural errors caused by repeatedly predicting existing scene content in action-conditioned autoregressive video prediction. Its core component, Inertial Memory, combines a dense Eulerian grid that maintains scene-wide features with persistent Lagrangian carriers that bind canonical content to learned coordinate systems. At each transition, the model first predicts action-conditioned carrier motion, transports existing content accordingly, and then predicts the next frame’s Dynamic code. A learned gate subsequently combines the transported features with newly proposed content to update the recurrent memory.

The model uses a fixed Context code extracted from the initial observation and a predicted Dynamic code for each future frame, without feeding future ground-truth states into the recurrence during either training or inference. Models trained separately on RT-1, CSGO, and RECON outperform the evaluated released checkpoints on average over 32-frame predictions. On RT-1, the method also improves flow-based motion diagnostics and dynamic-region LPIPS, and the paper includes a preliminary closed-loop policy evaluation. However, while these results demonstrate the effectiveness of the complete system, evidence isolating the contribution of the proposed transport mechanism and establishing stability over longer horizons remains limited.

Critical strengths
  1. The method explicitly implements state persistence within the transition operation. Beyond providing historical features as attention context, the model transports stored canonical content according to predicted coordinate changes. In particular, performing transport before writing new content establishes an explicit computational path for persistence. This is a well-motivated design for addressing gradual identity and structural drift during autoregressive prediction.

  2. The dense and regional representations serve complementary roles. The Eulerian grid maintains scene-wide evidence, while the carriers bind local content to coordinates. The background branch and fallback flow avoid requiring a complete decomposition of the scene into distinct objects. Furthermore, vacating the source locations of transported content and separately handling newly revealed regions addresses potential duplication and disocclusion issues during transport.

Critical weaknesses
  1. The central claim of persistent content preservation is not directly validated. The main evaluation measures prediction accuracy over 32 steps, matching the maximum training horizon. These results do not establish whether the carriers preserve the same content over extended periods, recover previously occluded content when it reappears, or simply improve next-frame prediction. The paper also lacks direct analysis of carrier identity persistence, actual content transport, and the extent to which the write gates overwrite stored content. Thus, although the improvement in video prediction quality is supported, evidence that the proposed persistent state evolution operates as intended remains insufficient.
Questions for authors
  1. Could the authors directly evaluate content persistence? Could you provide rollouts beyond 32 steps on sequences involving reappearance after prolonged occlusion or revisiting previously observed scenes, together with carrier correspondences, transport-route usage rates, and write-gate visualizations? These analyses would help establish whether the model preserves existing content or largely regenerates it.
5 comments
AuthorsOpenReview ↗

Response to Reviewer r97u — Part 1/2

Dear Reviewer r97u,

We are truly grateful for the time you have taken to review our paper and your insightful review. Here we respond to your questions point by point.

Q1. The central claim of persistent content preservation is not directly validated. The main evaluation measures prediction accuracy over 32 steps, matching the maximum training horizon. These results do not establish whether the carriers preserve the same content over extended periods, recover previously occluded content when it reappears, or simply improve next-frame prediction. The paper also lacks direct analysis of carrier identity persistence, actual content transport, and the extent to which the write gates overwrite stored content. Thus, although the improvement in video prediction quality is supported, evidence that the proposed persistent state evolution operates as intended remains insufficient. Could the authors directly evaluate content persistence? Could you provide rollouts beyond 32 steps on sequences involving reappearance after prolonged occlusion or revisiting previously observed scenes, together with carrier correspondences, transport-route usage rates, and write-gate visualizations? These analyses would help establish whether the model preserves existing content or largely regenerates it.

A1. Long-horizon prediction and reappearance. We extended RT-1 prediction from the maximum training horizon of 32 frames to one uninterrupted 64-frame rollout. Exactly 35 of the frozen 1,024 test episodes contained 64 aligned future frames and actions; we evaluated all 35 without subsampling. Eligibility depended only on sequence length and was fixed before inference. All methods used the same ordered episodes, anchors, actions, and targets, retained their native conditioning interfaces, and were not restarted at step 32.

Values were episode mean ± sample standard deviation.

MethodPSNR ↑SSIM ↑LPIPS ↓MSE ↓
RLVR-World20.22 ± 2.780.730 ± 0.0990.213 ± 0.0750.0138 ± 0.0096
Vid2World19.04 ± 2.350.636 ± 0.1070.315 ± 0.0670.0175 ± 0.0116
WorldGym19.98 ± 2.310.736 ± 0.0930.250 ± 0.0670.0144 ± 0.0074
DriftWorld19.51 ± 2.380.678 ± 0.0940.325 ± 0.0680.0154 ± 0.0083
InertialWorld22.58 ± 2.800.795 ± 0.0950.188 ± 0.0760.0085 ± 0.0071

InertialWorld ranked first on all four metrics at twice the training horizon. We added the protocol and aggregate results to revised Appendix D.4 (Table 10).

We also added a representative mechanism case in revised Fig. 5. A Context-visible region was occluded by the robot arm around t=30t=30, subsequently reappeared, and remained preserved through t=64t=64. Compared with the baselines, InertialWorld most closely retained the recovered region, drawer geometry, and task progression. We explicitly identified this as an illustrative case; the quantitative conclusion above used all 35 eligible episodes. For convenient inspection, we made the figure available directly at:

https://anonymous.4open.science/r/InertialWorld_rebuttal-53CC/fig_carrier_transport_write.png

AuthorsOpenReview ↗

Response to Reviewer r97u — Part 2/2

A1 continued. Mechanism visualization. We first added revised Fig. 5 to show how the mechanism behaved in the representative H64 occlusion–reappearance case from Part 1. The top rows showed that, compared with the baselines, InertialWorld most closely retained the recovered region, drawer geometry, and task progression. The bottom three rows directly corresponded to the three analyses requested in the review. For convenient inspection, we provided Fig. 5 directly at:

https://anonymous.4open.science/r/InertialWorld_rebuttal-53CC/fig_carrier_transport_write.png

  1. Carrier correspondences. The first row showed the full 16×1616\times16 matrix CtC^t between current carriers and the carrier bank initialized from Context. Its diagonal measured same-slot similarity, ``ID'' was the fraction whose nearest Context match remained the same slot, and Δ\Delta was the diagonal similarity minus the strongest off-diagonal match. In this case, ID remained 100% and the diagonal stayed dominant through t=64t=64, showing stable carrier identity correspondence over time.
  2. Transport-route usage rates. The second row mapped m∈[0,1]m\in[0,1] in f=(1−m)fE+mfL\mathbf f=(1-m)\mathbf f^E+m\mathbf f^L: f\mathbf f was the final state-transport flow, fE\mathbf f^E dense Eulerian flow, and fL\mathbf f^L carrier-derived Lagrangian flow. Thus m=0/1m=0/1 meant pure Eulerian/carrier flow. Here mm remained high (0.70 at t=64t=64), and bright carrier-dominant cells followed the coherently moving robot/drawer regions while more static areas were darker. This physically consistent pattern showed active transport rather than merely a selected route label.
  3. Write-gate visualization. The third row mapped the post-carry write gate gg. Mean gg decreased from 0.023 at t=10t=10 to 0.016 at t=64t=64. Most cells retained the transported state, while localized bright writes aligned with the moving robot, drawer, and newly exposed region rather than rewriting the entire scene.

All-episode quantification. We then measured all 35 eligible H64 episodes. Values were mean ± episode s.d.; window columns averaged their indicated steps and t=64t=64 was terminal. Retrieval asked whether the Context slot was nearest, cosine measured retention, and margin subtracted the strongest other-slot match.

DiagnosticSteps 1–32Steps 33–64t=64t=64
Carrier same-slot retrieval (%)100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
Carrier same-slot cosine0.924 ± 0.0020.753 ± 0.0030.682 ± 0.003
Carrier separation margin0.466 ± 0.0030.380 ± 0.0040.344 ± 0.006
Canonical-content retrieval (%)100.0 ± 0.0100.0 ± 0.0100.0 ± 0.0
Canonical-content cosine0.993 ± 0.0020.980 ± 0.0060.977 ± 0.006
Canonical-content separation margin0.740 ± 0.0690.728 ± 0.0690.726 ± 0.069
Mean Lagrangian flow share mm (%)85.0 ± 2.176.3 ± 4.169.4 ± 5.6
Locations with m>0.5m>0.5 (%)99.2 ± 1.292.8 ± 3.782.5 ± 8.5
Scene write, mean (%)2.64 ± 0.511.84 ± 0.461.76 ± 0.52
Scene write, q0.90q_{0.90} (%)6.05 ± 0.904.54 ± 0.964.34 ± 1.19
Carrier write, mean (%)0.46 ± 0.070.54 ± 0.180.58 ± 0.20
Canonical-feature write, mean (%)0.97 ± 0.030.18 ± 0.020.11 ± 0.02
Canonical-alpha write, mean (%)1.44 ± 0.020.27 ± 0.020.17 ± 0.03

Carrier retrieval remained 100% while cosine decreased from 0.924 to 0.753 across the two windows; the still-positive terminal margin (0.344) showed that descriptors adapted without losing slot identity or collapsing together. Canonical content was more stable: retrieval remained 100%, terminal cosine was 0.977, and terminal margin was 0.726. Transport also remained active beyond training: mean mm was 76.3% over Steps 33–64, with m>0.5m>0.5 at 92.8% of locations (69.4% and 82.5%, respectively, at t=64t=64). For writing, g=0g=0 retained and g=1g=1 replaced the post-transport state. Scene writes averaged 1.84% with a 4.54% 90th percentile over Steps 33–64, while carrier, canonical-feature, and canonical-alpha writes were only 0.54%, 0.18%, and 0.27%. Together with their localized responses in Fig. 5, these low but nonzero rates indicated selective innovation rather than an inactive gate or wholesale regeneration. Full definitions were added to revised Appendix D.4 (Table 11).

Together with the H64 gains in Part 1, these results supported carry-before-write: content stayed bound to traceable carriers, carrier-derived motion transported it, and localized gates updated only where new evidence was needed. This distinguished persistent evolution from repeated scene regeneration.

We sincerely thank you again for this constructive suggestion. It substantially strengthens the direct validation of the proposed persistent-state mechanism. If you have any questions, please feel free to let us know. We will gladly address them as thoroughly as possible.

Dear Reviewer r97u,

We sincerely thank you again for your thoughtful comments and for the time you devoted to reviewing our work. Following your suggestions, we added quantitative 64-step comparisons with four strong baselines to evaluate prediction beyond the training horizon. We also presented a representative occlusion–reappearance rollout alongside the corresponding baseline predictions. For this case, we visualized the three mechanism analyses requested in your review: carrier correspondences, transport-route usage rates, and write-gate behavior. To ensure that the evidence was not limited to this case study, we further reported aggregate quantitative results for these mechanism diagnostics across all H64 samples. These additions directly assess long-horizon content persistence, carrier-derived transport usage, and spatially selective post-carry writing. All of these results, analyses, and visualizations have been incorporated into the revised manuscript.

With fewer than two days remaining in the discussion period, if your schedule permits, we would be deeply grateful if you could briefly review our response. Thank you again for your valuable time and feedback. We will gladly address any remaining questions as thoroughly as possible.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.