- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
RAR makes the temporal step of an interactive world model controllable. It synchronizes video and control sampling across strides, maps temporal RoPE positions to physical time, and accumulates those coordinates during rate changes so cached history can be retained. The paper evaluates rate interpolation and switching in Doom, CS2 and FPV, multi-rate training benefits in a bidirectional Doom comparison, and adaptation to a streaming SANA-WM student.
Critical strengths
- The physical-time formulation is simple and useful. Equal elapsed time produces equal relative RoPE phase (Eqs. 4-5), while accumulated strides preserve relative time coordinates for non-sink frames across rate changes (Eq. 9). Combining this with pre-RoPE cached keys gives a concrete way to change the rollout rate without restarting history.
- The empirical support includes a useful control for rate conditioning. Under the reported SR filtering protocol, Appendix C's comparison with an explicit FPS-embedding baseline reduces mean SR error at unseen rates from 14.50% to 1.62%, using the same three-stage pipeline. On Doom, Table 2 shows lower-rate gains while 70-FPS LPIPS remains close (0.3135 versus 0.3113). The SANA-WM streaming adaptation also improves LPIPS at all evaluated rates, supporting portability, although the extra fine-tuning does not isolate rate conditioning alone.
Critical weaknesses
- Relative rate control is better established than absolute pacing under runtime throughput changes. Appendix B normalizes away each case's overall motion scale, so a constant multiplicative speed error can coexist with SR near one. Appendix E also calibrates each rollout before fitting its dynamic response. These measure cross-rate scaling and schedule shape, but do not establish that a held action produces the intended displacement per wall-clock second. 2.The controlled evidence for the rate-horizon training benefit is confined to bidirectional generation. Section 4.3 explicitly uses the bidirectional stage, and Appendix F evaluates longer sequences by jointly denoising the full sequence. This supports multi-rate training there, but does not establish that the quality-horizon advantage survives causal conversion and accumulated autoregressive errors.
Questions for authors
- Could one causal held-action rollout follow measured or deliberately varied runtime throughput, recording actual generation-completion times and reporting unnormalized displacement/state error separately from accumulated model-time error?
- Does the quality-horizon benefit persist in a matched-history long causal rollout of the final Single and RAR models already trained with comparable budgets? One environment would be sufficient to test this extension of Table 2. Please also identify the evaluated training-stage checkpoints in Secs. 4.2.1 and 4.2.2, so the rate-control evidence can be read unambiguously.
6 comments
Response to Reviewer 7XRK (2/2): Uncalibrated Switching and Causal Long-Horizon Evidence
[W1 & Q1] Uncalibrated switching (continued)
(3) Uncalibrated switching. To address App. E calibration, the first three columns remove filtering and then regression. Counts are 64→32 (LPIPS)→24 (GT reliability).
The last column independently estimates effective stride as generated displacement divided by GT displacement per base-frame interval (same latent interval), then takes the median over all 64 rollouts. No filtering, regression, or scale fitting is used. The first three columns use mean curves, the last a median curve. Cells report Pearson r / nRMSE.
| Env. | Schedule | Paper (n=24) | No filtering (n=64) | + No regression | GT-referenced, uncalibrated |
|---|---|---|---|---|---|
| Doom | Smooth | 0.941 / 0.219 | 0.967 / 0.282 | 0.927 / 0.291 | 0.867 / 0.175 |
| Doom | Abrupt | 0.938 / 0.185 | 0.976 / 0.211 | 0.977 / 0.213 | 0.982 / 0.074 |
| CS2 | Smooth | 0.899 / 0.248 | 0.945 / 0.328 | 0.937 / 0.353 | 0.856 / 0.192 |
| CS2 | Abrupt | 0.963 / 0.157 | 0.968 / 0.277 | 0.970 / 0.281 | 0.860 / 0.197 |
| FPV | Smooth | 0.998 / 0.169 | 0.994 / 0.277 | 0.988 / 0.307 | 0.997 / 0.108 |
| FPV | Abrupt | 0.970 / 0.122 | 0.973 / 0.231 | 0.974 / 0.229 | 0.980 / 0.118 |
- GT-referenced validation retains the tracking conclusion. Across all 64 rollouts without filtering, regression, or scale fitting, Pearson is 0.856–0.997, demonstrating that effective stride still follows the target schedule. nRMSE is 0.074–0.197, quantifying amplitude deviations from the target.
- Filtering and regression are not prerequisites. Removing filtering increases Pearson in five of six settings, with FPV smooth scheduling the sole exception (0.998→0.994). After removing regression, Pearson remains at 0.927–0.988. RAR's schedule-tracking capability therefore does not depend on these processing steps.
Full results are in App. E.
[W2 & Q2] Quality–horizon benefits in long causal rollouts
Sec. 4.3 evaluates bidirectional models. Following your suggestion, we compare matched-history long causal rollouts of final Single and RAR models across all three environments. RAR's advantage persists.
Single and RAR from Table 5 have comparable budgets: identical backbones, windows, and training steps, with all three stages completed. Identical KV-cache windows and sink settings are used to autoregressively generate 425 and 849 frames at base rates (Doom 70, CS2 64, FPV 60 FPS), matching App. F. All results are LPIPS (lower is better).
(a) Within the training window. Doom, corresponding to the upper half of Table 2:
| Protocol | Evaluation | Single | RAR |
|---|---|---|---|
| Causal, matched updates | 14 FPS | 0.4468 | 0.4279 |
| 35 FPS | 0.4180 | 0.3991 | |
| 70 FPS | 0.3988 | 0.3852 |
(b) Long causal rollouts. Corresponding to App. F:
| Environment | 425 frames: Single | 425 frames: RAR | 849 frames: Single | 849 frames: RAR |
|---|---|---|---|---|
| Doom | 0.4143 | 0.4087 | 0.4311 | 0.4306 |
| CS2 | 0.684 | 0.529 | 0.808 | 0.640 |
| FPV | 0.5238 | 0.4697 | 0.5720 | 0.5387 |
- Within the training window, RAR has lower LPIPS at all three rates, including Single's native 70 FPS. Causal conversion preserves lower-rate gains without sacrificing higher-rate quality; CS2 and FPV show the same pattern (App. C).
- At both 425 and 849 frames, RAR has lower LPIPS in all environments, especially CS2/FPV. The bidirectional quality–horizon advantage survives causal conversion and accumulated autoregressive errors.
- For rapid validation, we continue DF/SF from the respective bidirectional models under matched training time. RAR (213-frame window) is comparable to the 425-frame fixed-rate model at 425 frames (LPIPS 0.4344 vs. 0.4314) and outperforms the 849-frame model at 849 frames (0.4791 vs. 0.5769). The rate–horizon trade-off remains. We scope the corresponding matched-time claims to bidirectional evidence.
[Q3] Evaluation checkpoints
Secs. 4.2.1 and 4.2.2 use final Self-Forcing models after all three stages (steps in Table 4); Sec. 4.3 uses bidirectional models. Sec. 4.1 now identifies each section's training stage; App. A summarizes stages, case counts, and metrics.
[Others] SANA-WM fine-tuning
We agree. In Sec. 4.4, LPIPS is an auxiliary quality metric; SR supports multi-rate adaptation: except at the base model's native 16 FPS, RAR brings SR closer to 1 at 10, 20, and 30 FPS. A few fine-tuning steps give SANA-WM effective rate control, demonstrating RAR's applicability to another model.
Revisions. Sec. 4.1/App. A: training stages and evaluation settings; App. B: RF definition and per-rate results; App. C: Table 5 Single/RAR LPIPS; App. E: all-sample uncalibrated switching; App. F: long causal rollouts. Main-text matched-time claims are scoped to bidirectional evidence.
We hope this addresses your concerns and welcome further discussion.
Thank you again for your thoughtful review. Following your suggestions, we have added causal rollout experiments under varying throughput, along with matched-history long causal comparisons across all three environments.
We hope these new results address your two main questions. As the discussion period is coming to an end, we'd appreciate hearing whether you have any remaining concerns. Thanks again for your helpful suggestions!
Response to Reviewer 7XRK (1/2): Throughput Feedback and Absolute Speed
We thank the reviewer for recognizing the physical-time formulation and the rate-conditioning comparisons, and for suggesting two concrete experiments. Following these suggestions, we add causal rollouts under varying throughput using recorded controls, and long causal rollouts of the final Single and RAR models. Our responses follow.
[W1 & Q1] Causal rollouts under varying throughput
We conduct this experiment on Doom, reporting model-time error separately from displacement ratios without cross-rate normalization. Over approximately 10 s on the scaled evaluation clock, throughput follows 70→35→20→50→14 FPS. The fixed-rate world falls behind wall-clock time by 4.63 s, compared with only 0.34 s for RAR; the wall-clock displacement ratio improves from 0.71 to 0.89.
(1) Throughput following. We use the final causal RAR model with the inference settings of Table 1 and 16 initial states from its Doom evaluation set. Both conditions share noise seed 42.
The table reports medians [IQR] over 16 cases:
(2) Absolute speed. To address normalization in App. B, we compute the generated-to-GT speed ratio RF=vpred/vGT over the same physical interval at each rate, then take the median over all 64 cases without cross-rate normalization (ideal 1). Multiplying all generated speeds by a constant directly changes RF, exposing the global multiplicative speed error raised by the reviewer.
The table reports mean ∣RF−1∣ over five rates per environment (%):
RAR achieves the lowest absolute-speed error both overall (8.19%) and at interpolation rates (8.85%). Thus, the generated world preserves not only cross-rate scaling but also absolute pacing consistent with GT. Stride-RoPE's continuous physical-time coordinates maintain accurate absolute speeds at unseen rates. Per-rate results are provided in App. B.