- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
InertialWorld predicts video by transporting persistent content before updating its recurrent memory. A dense grid and regional carriers provide complementary state representations, with action-conditioned motion and learned write gates. Training uses predicted-only autoregressive rollouts. Experiments cover RT-1 manipulation, CSGO, and RECON navigation, with additional motion and policy-evaluation diagnostics.
Critical strengths
-
The carry-then-update transition is a concrete architectural proposal for reducing repeated reconstruction. At 30k steps, the full model improves PSNR from 22.321 to 23.367 dB over the matched Plain AR baseline despite having slightly fewer parameters.
-
Improvements appear across three domains and persist on diagnostics focused on moving regions. These results support the usefulness of the complete system, though they do not yet isolate its proposed transport mechanism.
Critical weaknesses
-
The controlled comparison supports the complete Inertial Memory system, but does not isolate transport from the benefit of recurrent memory. The carrier and gate ablations are inference-time interventions, so their drops show reliance on those paths rather than superiority over separately trained alternatives. This limits the mechanistic interpretation of the gains.
-
The controlled ablation uses one training seed, and the main comparisons use different baseline training recipes and native interfaces. The close-drawer study has only eight trajectories per displayed cell and unpaired real-world references; it should remain preliminary evidence for policy evaluation.
-
Appendix B.2 lists staged losses for prediction, appearance, transport, binding, rigidity, contact, and deformation without fully specifying their formulas or numerical weights. Reproducing the gains and separating the contribution of transport from the training recipe are therefore difficult. The missing recurrent-memory control and incomplete training specification lead to weak rejection.
Questions for authors
-
How does the full model compare with a separately trained recurrent dense-memory baseline using the same tokenizer, objectives, history, and compute, but without explicit carrier transport?
-
Does the matched Plain AR advantage persist across independent training seeds and on CSGO or RECON? Episode-level standard deviations do not establish training stability.
-
Can you provide the complete loss definitions, weights, and staging schedule from Appendix B.2, and clarify which terms are shared with Plain AR? This is needed to reproduce and interpret the controlled comparison.
9 comments
Response to Reviewer W6F3 — Part 2/5
Q2. The controlled ablation uses one training seed, and the main comparisons use different baseline training recipes and native interfaces. Does the matched Plain AR advantage persist across independent training seeds and on CSGO or RECON? Episode-level standard deviations do not establish training stability.
A2. We agreed that episode-level standard deviations did not measure training-run variation. We therefore trained a second matched Full/Plain AR pair from scratch on RT-1 with seed 1 and another matched pair on CSGO. Within each pair, architecture-specific state aside, all models were at the same 0.8B scale, and the tokenizer, data, prediction targets, optimizer, curriculum, update budget, and evaluation were matched. All controlled ablation runs used the same 30k-update training budget. Full achieved the best result on all three persistence metrics for both RT-1 seeds and on CSGO.
| Setting | Model | Long-lived content ↓ | Post-reappearance content ↓ | Late-track content ↓ |
|---|---|---|---|---|
| RT-1, seed 0 | Plain AR | 0.5144 ± 0.1275 | 0.5144 ± 0.1538 | 0.1009 ± 0.0558 |
| RT-1, seed 0 | Full | 0.4145 ± 0.1328 | 0.4036 ± 0.1620 | 0.0876 ± 0.0552 |
| RT-1, seed 1 | Plain AR | 0.5172 ± 0.1272 | 0.5073 ± 0.1569 | 0.1051 ± 0.0579 |
| RT-1, seed 1 | Full | 0.4153 ± 0.1354 | 0.4053 ± 0.1644 | 0.0888 ± 0.0561 |
| CSGO, seed 0 | Plain AR | 0.6076 ± 0.0761 | 0.5961 ± 0.0915 | 0.4864 ± 0.1211 |
| CSGO, seed 0 | Full | 0.5679 ± 0.0917 | 0.5624 ± 0.1076 | 0.4485 ± 0.1200 |
The paired margins were computed as Plain AR minus Full, so positive values favored Full:
| Setting | Comparison | Long-lived Δ [95% CI] | Post-reappearance Δ [95% CI] | Late-track Δ [95% CI] |
|---|---|---|---|---|
| RT-1, seed 0 | Plain AR − Full | 0.0999 [0.0932,0.1067] | 0.1108 [0.1007,0.1210] | 0.0133 [0.0121,0.0145] |
| RT-1, seed 1 | Plain AR − Full | 0.1019 [0.0952,0.1086] | 0.1020 [0.0920,0.1121] | 0.0163 [0.0148,0.0177] |
| CSGO, seed 0 | Plain AR − Full | 0.0396 [0.0343,0.0450] | 0.0337 [0.0274,0.0401] | 0.0379 [0.0331,0.0427] |
All nine intervals excluded zero. This directly evaluated an independent training seed and a second domain rather than repurposing episode dispersion as seed uncertainty. With two RT-1 seeds, we conservatively stated that the advantage was not specific to seed 0, without claiming that two runs exhausted training variance. These results and episode support were added to Table 3 and Appendix E.
Response to Reviewer W6F3 — Part 3/5
Q3. The close-drawer study has only eight trajectories per displayed cell and unpaired real-world references; it should remain preliminary evidence for policy evaluation.
A3. We agreed with both limitations and replaced the original probe. The revised protocol crossed four frozen RT-1 policies with two balanced drawer tasks and 18 requests per policy–task cell, yielding 144 fully annotated trajectories per world model and 720 across five methods. All policies used the same 36 initial requests, and all methods shared the current observation for each request; thereafter each policy saw only that method's generated frame for at most 113 control transitions.
| Method | Begin success | RT-1-X success | RT-1 15% success | Converged success | Overall MAE to real-world success (pp) ↓ |
|---|---|---|---|---|---|
| SIMPLER real reference | 0.0% | 63.0% | 79.7% | 87.1% | — |
| RLVR-World | 16.7% | 22.2% | 38.9% | 58.3% | 31.7 |
| Vid2World | 30.6% | 47.2% | 47.2% | 50.0% | 29.0 |
| WorldGym | 19.4% | 50.0% | 52.8% | 63.9% | 20.6 |
| DriftWorld | 19.4% | 16.7% | 33.3% | 30.6% | 42.1 |
| InertialWorld | 13.9% | 55.6% | 58.3% | 69.4% | 15.1 |
We agreed that this remained a preliminary Real2Sim evaluation because the SIMPLER real-world success rates were external aggregates rather than trajectory-paired targets. Within this limitation, all world models were compared under the same protocol, and InertialWorld reproduced the policy-level success profile more faithfully than every baseline, attaining the lowest MAE to real-world success for each policy and overall. We therefore treated this as comparative evidence of preserved policy competence rather than a substitute for paired real-world trials. In revised Sec. 4.3, Fig. 7(a) showed success across the four policies and Fig. 7(b) showed MAE to the SIMPLER reference. For convenient inspection, the figure was made available directly at:
https://anonymous.4open.science/r/InertialWorld_rebuttal-53CC/real2sim_policy_success_and_mae.pdf
Appendix C.5 gave the complete protocol and its limitation.
Response to Reviewer W6F3 — Part 1/5
Dear Reviewer W6F3,
We are truly grateful for the time you have taken to review our paper and your insightful review. Here we respond to your questions point by point.
A1. We agreed and removed the inference-time interventions. In the original submission, limited compute prevented us from training additional matched 0.8B variants from scratch, so we used those interventions as preliminary diagnostics. With additional rebuttal compute, we independently trained all four 0.8B-scale models from scratch for the same 30k updates and reported them in revised Table 3. Dense-only was the closest requested control: it shared the frozen tokenizer, 320-token Eulerian scene, 64 interaction slots, four-frame hard-prediction history, data, actions, optimizer, AR8→AR16→AR32 curriculum, full BPTT, batch/GPU topology, update budget, and ordered 1,024-episode evaluation. It removed carriers, canonical content, source pointers, the transported route, and losses defined on those absent variables. Recurrent Memory was a second trained control that used an action-conditioned GRU instead of the structured Eulerian–Lagrangian state. To test the proposed persistence mechanism directly, we compared all variants using the same three ground-truth-track metrics: long-lived content error, post-reappearance content error, and late-track content error. These measured content retention over persistent tracks, recovery after occlusion, and accuracy in the rollout's later half, respectively, rather than generic frame averages dominated by static pixels. All three metrics gave the same ordering: Full performed best, followed by Dense-only, Recurrent Memory, and Plain AR.
Full-favorable paired differences were:
All Full-vs-control intervals excluded zero, confirming that Full's advantage over each independently trained control was statistically resolved. Revised Sec. 4.4/Table 3 and Appendix E documented the trained 0.8B-scale controls, metric construction, support, and paired intervals.