acceptodds
Under review as a conference paper at ICLR 2027

RAR: Rate-Adaptive Rollout for Time-Consistent Interactive World Models

Abstract

Video-based interactive world models aim to enable real-time, user-controllable environments. Yet most existing systems are trained and rolled out at a fixed temporal rate, limiting both runtime-rate adaptation and physical-horizon coverage-a limitation we term temporal-rate coupling. Specifically, when runtime generation throughput differs from the learned temporal rate, the generated world evolves unintentionally more slowly or quickly. Moreover, using a single temporal rate imposes an inherent trade-off between temporal granularity and physical horizon: higher rates capture finer motion but cover a shorter duration under a fixed context window. To address both limitations, we introduce Rate Adaptive Rollout (RAR), a framework that enables interactive world models to roll out at controllable temporal rates. At its core, Stride-RoPE aligns latent frames sampled at different rates on a shared physical-time coordinate, enabling consistent multi-rate rollout, unseen-rate generalization, and online rate switching without discarding history. Meanwhile, RAR uses multi-stride video sampling to enable joint learning of fine-grained dynamics and long-horizon evolution. Across action- and camera-controlled environments, RAR achieves temporally consistent multi-rate rollout and online rate switching while preserving high-rate generation quality and extending physical horizons under matched training budgets. RAR also transfers to pretrained world models while preserving perceptual quality.

Primary area:
generative models
Keywords:
Interactive World Models, Physical time

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7XRKOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RAR makes the temporal step of an interactive world model controllable. It synchronizes video and control sampling across strides, maps temporal RoPE positions to physical time, and accumulates those coordinates during rate changes so cached history can be retained. The paper evaluates rate interpolation and switching in Doom, CS2 and FPV, multi-rate training benefits in a bidirectional Doom comparison, and adaptation to a streaming SANA-WM student.

Critical strengths
  1. The physical-time formulation is simple and useful. Equal elapsed time produces equal relative RoPE phase (Eqs. 4-5), while accumulated strides preserve relative time coordinates for non-sink frames across rate changes (Eq. 9). Combining this with pre-RoPE cached keys gives a concrete way to change the rollout rate without restarting history.
  2. The empirical support includes a useful control for rate conditioning. Under the reported SR filtering protocol, Appendix C's comparison with an explicit FPS-embedding baseline reduces mean SR error at unseen rates from 14.50% to 1.62%, using the same three-stage pipeline. On Doom, Table 2 shows lower-rate gains while 70-FPS LPIPS remains close (0.3135 versus 0.3113). The SANA-WM streaming adaptation also improves LPIPS at all evaluated rates, supporting portability, although the extra fine-tuning does not isolate rate conditioning alone.
Critical weaknesses
  1. Relative rate control is better established than absolute pacing under runtime throughput changes. Appendix B normalizes away each case's overall motion scale, so a constant multiplicative speed error can coexist with SR near one. Appendix E also calibrates each rollout before fitting its dynamic response. These measure cross-rate scaling and schedule shape, but do not establish that a held action produces the intended displacement per wall-clock second. 2.The controlled evidence for the rate-horizon training benefit is confined to bidirectional generation. Section 4.3 explicitly uses the bidirectional stage, and Appendix F evaluates longer sequences by jointly denoising the full sequence. This supports multi-rate training there, but does not establish that the quality-horizon advantage survives causal conversion and accumulated autoregressive errors.
Questions for authors
  1. Could one causal held-action rollout follow measured or deliberately varied runtime throughput, recording actual generation-completion times and reporting unnormalized displacement/state error separately from accumulated model-time error?
  2. Does the quality-horizon benefit persist in a matched-history long causal rollout of the final Single and RAR models already trained with comparable budgets? One environment would be sufficient to test this extension of Table 2. Please also identify the evaluated training-stage checkpoints in Secs. 4.2.1 and 4.2.2, so the rate-control evidence can be read unambiguously.
6 comments
AuthorsOpenReview ↗

Response to Reviewer 7XRK (1/2): Throughput Feedback and Absolute Speed

We thank the reviewer for recognizing the physical-time formulation and the rate-conditioning comparisons, and for suggesting two concrete experiments. Following these suggestions, we add causal rollouts under varying throughput using recorded controls, and long causal rollouts of the final Single and RAR models. Our responses follow.


[W1 & Q1] Causal rollouts under varying throughput

We conduct this experiment on Doom, reporting model-time error separately from displacement ratios without cross-rate normalization. Over approximately 10 s on the scaled evaluation clock, throughput follows 70→35→20→50→14 FPS. The fixed-rate world falls behind wall-clock time by 4.63 s, compared with only 0.34 s for RAR; the wall-clock displacement ratio improves from 0.71 to 0.89.

(1) Throughput following. We use the final causal RAR model with the inference settings of Table 1 and 16 initial states from its Doom evaluation set. Both conditions share noise seed 42.

  • Throughput follows 70→35→20→50→14 FPS, with 2 s per segment and approximately 10 s per case on the scaled evaluation clock. Our single-GPU setup cannot run in real time at the target rates. We therefore measure actual step-completion times, deliberately vary throughput by injecting waits after generation, and apply a fixed time scaling. Timing covers generation and injected waits only, excluding input capture, display, and network delays. All steps complete within their target durations.
  • At each step, RAR estimates throughput rr from the completion-time intervals of the two most recent steps, then selects s=fbase/rs=f_{base}/r, clipped to the training range. The baseline uses the same model and throughput profile with fixed s=1s=1 (70 FPS).
  • Controls. We replay recorded controls rather than a synthetic sustained pure turn, keeping the control distribution consistent with training and obtaining time-aligned GT from the recording. The episode's control stream is replayed at physical time, using the same resampling as in the paper evaluation.
  • GT. We use the recording of the same episode; evaluation time τ corresponds to the frame index nearest to 70τ.
  • Metrics. Model-time error eTe_T is accumulated model time T minus evaluation time τ, with time-progression ratio ρ=T/τ. The displacement ratio compares accumulated optical-flow motion in the central image region with the recording, evaluated separately at matched model time and wall-clock time. No cross-rate normalization or per-case calibration is applied; the ideal displacement ratio is 1.

The table reports medians [IQR] over 16 cases:

ConditionModel-time error eTe_Tρ=T/τDisplacement ratio, model timeDisplacement ratio, wall clock
Fixed rate (s=1s=1)−4.63 s0.5401.17 [0.99, 1.48]0.71 [0.64, 0.85]
RAR−0.34 s0.9660.94 [0.84, 1.29]0.89 [0.84, 1.20]
  • Model time. Within each constant-throughput segment, the fixed-rate time-progression ratio equals that segment's throughput divided by 70. After approximately 10 s of scaled evaluation time, it lags by 4.63 s (overall ρ=0.54), versus 0.34 s for RAR (ρ=0.97). RAR's error accumulates only during the first two steps after each throughput change, while the estimate still reflects the previous throughput; it remains constant thereafter.
  • Displacement. The wall-clock displacement ratio improves from 0.71 for the fixed-rate baseline to 0.89 for RAR, closer to the GT motion expected at the corresponding time.
  • Matched-model-time displacement. This measures motion-execution accuracy; RAR's median is 0.94. The wall-clock ratio additionally includes scheduling effects. Reporting both separately shows that RAR preserves motion fidelity while improving wall-clock pacing.

(2) Absolute speed. To address normalization in App. B, we compute the generated-to-GT speed ratio RF=vpred/vGTRF=v_{pred}/v_{GT} over the same physical interval at each rate, then take the median over all 64 cases without cross-rate normalization (ideal 1). Multiplying all generated speeds by a constant directly changes RF, exposing the global multiplicative speed error raised by the reviewer.

The table reports mean ∣RF−1∣|RF-1| over five rates per environment (%):

SingleEmbedRAR
Doom51.5613.065.27
CS239.7910.6212.89
FPV37.3426.746.40
Average across environments42.9016.818.19
6 interpolation rates (†)30.3526.678.85

RAR achieves the lowest absolute-speed error both overall (8.19%) and at interpolation rates (8.85%). Thus, the generated world preserves not only cross-rate scaling but also absolute pacing consistent with GT. Stride-RoPE's continuous physical-time coordinates maintain accurate absolute speeds at unseen rates. Per-rate results are provided in App. B.

AuthorsOpenReview ↗

Response to Reviewer 7XRK (2/2): Uncalibrated Switching and Causal Long-Horizon Evidence

[W1 & Q1] Uncalibrated switching (continued)

(3) Uncalibrated switching. To address App. E calibration, the first three columns remove filtering and then regression. Counts are 64→32 (LPIPS)→24 (GT reliability).

The last column independently estimates effective stride as generated displacement divided by GT displacement per base-frame interval (same latent interval), then takes the median over all 64 rollouts. No filtering, regression, or scale fitting is used. The first three columns use mean curves, the last a median curve. Cells report Pearson r / nRMSE.

Env.SchedulePaper (n=24)No filtering (n=64)+ No regressionGT-referenced, uncalibrated
DoomSmooth0.941 / 0.2190.967 / 0.2820.927 / 0.2910.867 / 0.175
DoomAbrupt0.938 / 0.1850.976 / 0.2110.977 / 0.2130.982 / 0.074
CS2Smooth0.899 / 0.2480.945 / 0.3280.937 / 0.3530.856 / 0.192
CS2Abrupt0.963 / 0.1570.968 / 0.2770.970 / 0.2810.860 / 0.197
FPVSmooth0.998 / 0.1690.994 / 0.2770.988 / 0.3070.997 / 0.108
FPVAbrupt0.970 / 0.1220.973 / 0.2310.974 / 0.2290.980 / 0.118
  • GT-referenced validation retains the tracking conclusion. Across all 64 rollouts without filtering, regression, or scale fitting, Pearson is 0.856–0.997, demonstrating that effective stride still follows the target schedule. nRMSE is 0.074–0.197, quantifying amplitude deviations from the target.
  • Filtering and regression are not prerequisites. Removing filtering increases Pearson in five of six settings, with FPV smooth scheduling the sole exception (0.998→0.994). After removing regression, Pearson remains at 0.927–0.988. RAR's schedule-tracking capability therefore does not depend on these processing steps.

Full results are in App. E.


[W2 & Q2] Quality–horizon benefits in long causal rollouts

Sec. 4.3 evaluates bidirectional models. Following your suggestion, we compare matched-history long causal rollouts of final Single and RAR models across all three environments. RAR's advantage persists.

Single and RAR from Table 5 have comparable budgets: identical backbones, windows, and training steps, with all three stages completed. Identical KV-cache windows and sink settings are used to autoregressively generate 425 and 849 frames at base rates (Doom 70, CS2 64, FPV 60 FPS), matching App. F. All results are LPIPS (lower is better).

(a) Within the training window. Doom, corresponding to the upper half of Table 2:

ProtocolEvaluationSingleRAR
Causal, matched updates14 FPS0.44680.4279
35 FPS0.41800.3991
70 FPS0.39880.3852

(b) Long causal rollouts. Corresponding to App. F:

Environment425 frames: Single425 frames: RAR849 frames: Single849 frames: RAR
Doom0.41430.40870.43110.4306
CS20.6840.5290.8080.640
FPV0.52380.46970.57200.5387
  • Within the training window, RAR has lower LPIPS at all three rates, including Single's native 70 FPS. Causal conversion preserves lower-rate gains without sacrificing higher-rate quality; CS2 and FPV show the same pattern (App. C).
  • At both 425 and 849 frames, RAR has lower LPIPS in all environments, especially CS2/FPV. The bidirectional quality–horizon advantage survives causal conversion and accumulated autoregressive errors.
  • For rapid validation, we continue DF/SF from the respective bidirectional models under matched training time. RAR (213-frame window) is comparable to the 425-frame fixed-rate model at 425 frames (LPIPS 0.4344 vs. 0.4314) and outperforms the 849-frame model at 849 frames (0.4791 vs. 0.5769). The rate–horizon trade-off remains. We scope the corresponding matched-time claims to bidirectional evidence.

[Q3] Evaluation checkpoints

Secs. 4.2.1 and 4.2.2 use final Self-Forcing models after all three stages (steps in Table 4); Sec. 4.3 uses bidirectional models. Sec. 4.1 now identifies each section's training stage; App. A summarizes stages, case counts, and metrics.


[Others] SANA-WM fine-tuning

We agree. In Sec. 4.4, LPIPS is an auxiliary quality metric; SR supports multi-rate adaptation: except at the base model's native 16 FPS, RAR brings SR closer to 1 at 10, 20, and 30 FPS. A few fine-tuning steps give SANA-WM effective rate control, demonstrating RAR's applicability to another model.


Revisions. Sec. 4.1/App. A: training stages and evaluation settings; App. B: RF definition and per-rate results; App. C: Table 5 Single/RAR LPIPS; App. E: all-sample uncalibrated switching; App. F: long causal rollouts. Main-text matched-time claims are scoped to bidirectional evidence.

We hope this addresses your concerns and welcome further discussion.

Thank you again for your thoughtful review. Following your suggestions, we have added causal rollout experiments under varying throughput, along with matched-history long causal comparisons across all three environments.

We hope these new results address your two main questions. As the discussion period is coming to an end, we'd appreciate hearing whether you have any remaining concerns. Thanks again for your helpful suggestions!

Reviewer_4aFuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

RAR addresses the coupling between a world model’s rollout rate, physical-time horizon, and runtime generation throughput. It combines multi-stride video/control training with temporal rotary positions proportional to physical time. At a fixed rate, positions are pn=nfbase/fp_n = n f_{\mathrm{base}}/f; with changing rates, positions accumulate the chosen strides. Storing keys before rotary embedding allows the cache to be re-rotated without discarding its history. Experiments cover Doom, CS2, FPV, and adaptation of SANA-WM. The contribution is a coherent integration of multi-rate training and cache handling, with evidence for relative rate consistency and within-range interpolation.

Critical strengths
  1. The temporal representation and cache mechanism are coherent and practically relevant. Equations 4–5 ensure that equal physical-time differences have equal rotary phase differences, and Equation 9 extends this to variable strides. The implementation distinguishes ordinary cached frames from fixed-position sink frames. Physical-time RoPE itself has precedent in LTX-Video, Section 2.2.1, which the submission cites; the useful contribution here is its integration with interactive multi-rate training and cache reuse (Sections 3.2–3.3).
  2. The controlled alternatives support the relative-rate result. Appendix C matches the backbone, training pipeline, and optimization-step budget when comparing inference-only rescaling and multi-rate training with continuous FPS embeddings. RAR obtains SR MAEs of 3.54%, 3.54%, and 4.02%, compared with 13.92%, 7.24%, and 9.40% for FPS embeddings. Across unseen rates, the aggregate comparison is 1.62% versus 14.50%. This is meaningful evidence for interpolation on the stated metric. The explicit reporting of out-of-range degradation also helps delimit the contribution (Appendices C–D).
Critical weaknesses
  1. The evaluation establishes relative rate consistency more convincingly than absolute physical-time behavior. In Equations 12–14, generated and ground-truth speeds are each normalized by their own cross-rate geometric mean. Multiplying every generated speed by the same positive constant therefore leaves SR unchanged. The authors correctly identify SR as a relative-scaling metric, but it cannot establish the absolute timing behavior motivating the paper. Dynamic switching is additionally evaluated after per-rollout scale calibration, retaining the lowest-LPIPS half of cases, further reliability filtering, and curve fitting (Appendix E). This supports schedule-shape tracking on selected samples, but does not establish uncalibrated speed accuracy or switching delay over all rollouts. The control semantics also require clarification: Equation 1 treats actions as interval controls, while Equation 6 samples a(tn)a(t_n), without explaining how keyboard states, mouse increments, events, or camera trajectories are aggregated over longer intervals. That distinction matters for interpreting physically equivalent commands across rates (Appendices A–B and E).
  2. The long-horizon efficiency result has not been demonstrated for the final causal streaming model. Section 4.3 explicitly evaluates the bidirectional training stage, and Appendix F jointly denoises the complete 425/849-frame sequences. The approximately matched wall-clock comparisons and training curves are useful; the issue is whether their advantage survives Diffusion Forcing and Self-Forcing in the interactive model. Evidence from that stage, or consistent restriction of the efficiency claim to bidirectional generation, is needed to align the conclusion with the validation (Section 4.3; Appendix F). I currently lean toward rejection because the strongest physical-time and interactive-horizon claims extend beyond the evidence presented. Clarifying interval-level control semantics and providing targeted uncalibrated validation, together with causal-model evidence or appropriately narrower claims, could move my recommendation toward acceptance.
Questions for authors

Targeted validation: Can you report uncalibrated motion/displacement error and raw switching responses, including all-sample results and sample counts before and after each filtering stage? How much of the conclusion survives without scale calibration and curve fitting? Clarification: At training and interpolated strides, how do keyboard states, mouse increments, event actions, and camera trajectories represent control over each physical transition interval? Scope and evidence: Does the final causal model retain the matched-budget long-horizon advantage? If this has not been evaluated, can the corresponding claims be restricted to the bidirectional stage? Please identify the compared checkpoints and the hardware, batch, and evaluation-sample settings underlying the matched-time results.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4aFu (1/2): Absolute Timing and Uncalibrated Switching

Thank you for the detailed feedback. We address uncalibrated validation (W1/Q1), control semantics (W1/Q2), causal evidence and claim scope (W2/Q3), and matched-time settings (Q4).

[W1 & Q1] Absolute timing and uncalibrated validation

(1) Absolute speed. At each rate, RF=vpred/vGTRF=v_{pred}/v_{GT} compares speeds over the same physical interval. We take the median over all 64 cases without cross-rate normalization (ideal 1). Below is mean ∣RF−1∣|RF-1| over five rates per environment (%):

SingleEmbedRAR
Doom51.5613.065.27
CS239.7910.6212.89
FPV37.3426.746.40
Overall42.9016.818.19
6 interpolation rates (†)30.3526.678.85

RAR has the lowest overall and unseen-rate errors, showing that physical-time coordinates preserve relative scaling and accurate absolute pacing. Per-rate RF is in App. B. We use SLAM to directly estimate camera speed on 27 collision-free CS2 clips with only forward controls: mean RAR/GT ratios span 0.93–1.08 across five rates, without a monotonic trend, complementing optical-flow RF. SR reduces scene-dependent motion-scale differences to isolate cross-rate scaling.

(2) Unfiltered SR. Tables 1/5 select the 32 of 64 cases with the lowest GT cross-rate speed coefficient of variation (CV). GT physical speed is identical across rates, so ideal CV is zero and deviations reflect optical-flow noise. Selection uses only GT; all methods share cases. SR MAE (%) over all 64 cases:

SingleEmbedRAR
Doom113.9113.353.46
CS249.558.7210.97
FPV88.5510.362.31
Overall84.0010.815.58
6 interpolation rates (†)76.1216.144.36

RAR retains its overall and interpolation advantages. GT speed CV is 0.122/0.087/0.108 over 64 cases versus 0.091/0.057/0.069 after filtering (Doom/CS2/FPV). Greater measurement noise in the added cases motivates reliability filtering.

(3) Uncalibrated switching. The first three columns remove filtering and then regression; counts are 64→32 (LPIPS)→24 (GT reliability). The last is a separate GT-referenced validation: we estimate stride by dividing generated displacement by GT displacement per base-frame interval (same interval), then take the median across all 64 rollouts. No filtering, regression, or scale fitting is used. The first three columns use mean curves, the last a median curve. Cells report Pearson r / nRMSE.

Env.SchedulePaper protocol (n=24)No filtering (n=64)+ No regressionGT-referenced, uncalibrated
DoomSmooth0.941 / 0.2190.967 / 0.2820.927 / 0.2910.867 / 0.175
DoomAbrupt0.938 / 0.1850.976 / 0.2110.977 / 0.2130.982 / 0.074
CS2Smooth0.899 / 0.2480.945 / 0.3280.937 / 0.3530.856 / 0.192
CS2Abrupt0.963 / 0.1570.968 / 0.2770.970 / 0.2810.860 / 0.197
FPVSmooth0.998 / 0.1690.994 / 0.2770.988 / 0.3070.997 / 0.108
FPVAbrupt0.970 / 0.1220.973 / 0.2310.974 / 0.2290.980 / 0.118

Without filtering, Pearson rises in 5/6 settings, while nRMSE increases to 0.211–0.328, consistent with excluding poorer generations and unreliable measurements. Without regression, Pearson remains 0.927–0.988. GT-referenced Pearson is 0.856–0.997 and nRMSE is 0.074–0.197: the former shows schedule tracking, the latter measures amplitude error. Tracking does not depend on filtering, regression, or scale fitting.

(4) Switching delay. We use the abrupt schedule (six constant-stride segments, five switches) and all 64 rollouts per environment, without filtering, calibration, or regression. Using the GT-referenced method in (3), we estimate stride per latent interval and take the median over 64 rollouts. We shift this curve by −3 to +3 latent steps relative to the target; the shift maximizing Pearson is the alignment lag (positive: delayed). Part (3) aggregates six abrupt-schedule segments; here we use individual latent intervals, hence the different temporal resolutions.

Env.Shift −1 (early)Shift 0Shift +1 (delayed)Peak shift
Doom0.7430.9080.7830
CS20.6330.7630.7190
FPV0.7960.9590.8110

All three peaks occur at zero shift; shifting one step either way lowers correlation. Responses align with the target at latent-step resolution (four frames).

(5) Validation summary.

ConclusionPaper evidenceAll-sample, uncalibrated
Relative/absolute speed consistencySR: relative scalingRF error 8.19% overall, 8.85% unseen; both lowest (1)
Interpolation outperforms EmbedTable 5: 1.62% vs. 14.50%64 cases: 4.36% vs. 16.14% (2)
Continuous schedule trackingProcessed Pearson 0.899–0.998Pearson 0.856–0.997 (3)
Switching aligns with target—Correlation peaks at zero shift (4)
AuthorsOpenReview ↗

Response to Reviewer 4aFu (2/2): Control Semantics, Causal Evidence and Training Budgets

[W1 & Q2] Interval-level control semantics

In Eq. 6, a(tn(s))a(t_n^{(s)}) denotes control between consecutive sampling times, consistent with ana_n in Eq. 1. Inputs come from the base-rate control stream, with identical rules at training and interpolation rates:

Control typeEnv.Interval representation
Held-key statesDoom, CS2Sampled synchronously with stride
Instantaneous events (fire/click)Doom, CS2Triggered if pressed at any time within the interval
Mouse incrementsDoom, CS2Summed over the interval
Camera trajectoriesFPVPose relative to a reference frame at the sampling time
Non-integer stridesAllNearest base-rate frame

Summation preserves cumulative rotation, aggregation prevents missed events, and poses are sampled at aligned times; Stride-RoPE supplies interval duration. Thus the same physical controls preserve rotation, events, and time-aligned poses across rates. We add the explanation after Eq. 6 and the table to App. A.


[W2 & Q3] Long-horizon benefits in the causal stage

Sec. 4.3 evaluates bidirectional models. We add final causal comparisons under matched steps, long rollouts, and matched time, corresponding to Table 2 (upper), App. F, and Table 2 (lower).

(1) Matched windows and training steps. Single and RAR from Table 5 share the backbone, window, and training steps, and complete all three stages. Final causal-model LPIPS (lower is better):

Env.Single native rateSingleRARFive-rate mean: SingleFive-rate mean: RAR
Doom70 FPS0.39880.38520.42100.4061
CS264 FPS0.60550.58230.64530.6087
FPV60 FPS0.51280.49440.56500.5080

RAR has lower LPIPS at all rates in all environments, including Single's native rates. Lower-rate gains and higher-rate quality survive Diffusion Forcing (DF) and Self-Forcing (SF), consistent with bidirectional findings. Per-rate results are in App. C.

Long causal rollouts. The same models use identical KV-cache windows and sink settings to autoregressively generate 425/849 frames at base rates (Doom 70, CS2 64, FPV 60 FPS), matching App. F. LPIPS:

Env.425 frames: Single425 frames: RAR849 frames: Single849 frames: RAR
Doom0.41430.40870.43110.4306
CS20.6840.5290.8080.640
FPV0.52380.46970.57200.5387

RAR is better at both lengths in all environments, with clear gaps on CS2/FPV. Its quality–horizon advantage survives DF/SF conversion and accumulated autoregressive errors.

(2) Matched training time. For rapid validation, RAR (213-frame window) and fixed-rate models (425/849-frame windows) continue from their respective bidirectional models through approximately 62 h of DF and 28 h of SF, then are evaluated after SF. We generate corresponding lengths on Doom at 70 FPS; cases/seed match Table 2 (lower), while time matching, hardware, and per-GPU batch follow Q4. These models differ from the Table 5/long-rollout models above, so absolute LPIPS is not compared across groups.

Evaluation lengthRARCorresponding fixed-rate model
425 frames0.43440.4314
849 frames0.47910.5769

Under matched time, 213-frame-trained RAR achieves comparable quality at 425 frames (long-window fixed-rate training is slightly better) and better quality at 849 frames. Short-window causal training thus supports longer physical horizons. With no lead at 425 frames, we scope the corresponding matched-time statements and claims to bidirectional evidence.


[Q4] Matched-time settings

These are the original bidirectional matched-time settings (Sec. 4.3/App. F); checkpoints give that stage's training steps.

ConfigurationTraining windowHardwareBatch/GPUTime/stepCheckpointApprox. total training timeEvaluation cases
RAR213 frames8×H20180 s10k222 h32
Fixed rate425 frames8×H201125 s6.6k229 h32
Fixed rate849 frames8×H201420 s2k233 h32

Total training times differ by no more than 5%. Each model generates the corresponding 425/849 frames at 70 FPS, using seed 42. This table is added to App. F.

Revisions. Control semantics and the control table are added after Eq. 6 in Sec. 3.3 and to App. A; RF, CS2 SLAM, SR filtering, and all-sample results to App. B; all-sample uncalibrated switching and sample counts to App. E; Table 5 LPIPS to App. C; and the original bidirectional matched-time settings and long causal-rollout results to App. F. Main-text matched-time claims are scoped to the bidirectional evidence.

We hope these responses address your concerns and welcome further discussion.

Thank you again for your detailed feedback. Following your suggestions, we have added uncalibrated validation, clarified the control semantics, and provided new causal long-horizon results.

Since you mentioned that these results could help address your main concerns, we would really appreciate hearing whether they have changed your assessment.

We would really appreciate hearing whether they have changed your assessment. Please let us know if anything remains unclear. Thank you!

As the discussion period is coming to an end, we would like to briefly follow up on our rebuttal. We have added the uncalibrated validation, clarified the control semantics, and provided new causal long-horizon results, as you suggested. We hope these additions have addressed your main concerns. Given your earlier comments, we would really appreciate it if you could reconsider your assessment and your final rating based on our additional experiments and clarifications. Thank you again for your time and constructive feedback!

Reviewer_86mXOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Rate-Adaptive Rollout (RAR) to address temporal-rate coupling in interactive world models. The key component, Stride-RoPE, maps latent frames sampled at different rates to a shared physical-time coordinate. Its core mapping is

s=fbasef,pn=ns,s = \frac{f_{\mathrm{base}}}{f}, \qquad p_n = n s,

so equal physical-time intervals induce equal relative RoPE phases across sampling rates. RAR additionally uses synchronized multi-stride sampling of videos and control signals during training. At inference, the stride can be changed at every autoregressive step, allowing rate-controlled rollout and online rate switching without clearing the KV cache.

The method is evaluated on Doom, Counter-Strike 2, and an FPV drone-racing environment. The experiments cover training rates, unseen interpolation rates, dynamic rate switching, matched-update and matched-time rate-horizon comparisons, and adaptation to the pretrained SANA-WM world model. The results suggest that RAR can preserve relative cross-rate motion scaling and extend the physical horizon under a fixed frame budget.

Critical strengths
  1. Simple and unified formulation. Stride-RoPE provides a shared physical-time coordinate for training and inference without introducing additional learnable parameters.
  2. Broad controlled evaluation. The paper evaluates both training and unseen interpolation rates, as well as smooth and abrupt online rate switching, in three environments.
  3. Potential training-efficiency benefit. Under matched updates and approximately matched wall-clock time, multi-stride training appears to improve lower-rate or long-horizon generation while largely preserving high-rate quality.
  4. Evidence of portability. The adaptation experiment on pretrained SANA-WM suggests that the approach is not restricted to models trained from scratch.
Critical weaknesses
  1. The central physical-time claim is only partially validated. SR normalizes predicted and ground-truth speeds separately, so it measures relative cross-rate scaling but cannot detect a global temporal slowdown or speedup. No absolute state/trajectory error or closed-loop task metric is reported.

  2. The switching evaluation may overstate robustness. The analysis filters cases by LPIPS and measurement reliability, and applies global multiplicative calibration before computing correlations. This tests the shape of the switching curve more than absolute stride accuracy. Training also uses constant strides, while switching creates mixed-stride histories.

  3. Evidence and reproducibility are limited. Comparisons are mainly against the authors' Single and Embed baselines; results outside the training range degrade substantially. The paper does not report enough details on test-set size, seeds, runtime throughput, memory, cache-switching overhead, or coarse-stride control resampling.

Questions for authors
  1. Can you report an absolute physical-time or state-transition metric, together with closed-loop results under matched wall-clock time?
  2. What are the switching results without case filtering and global speed calibration, and how does the model handle mixed-stride histories?
  3. Can you add the key ablations (multi-stride sampling alone, Stride-RoPE alone, and mixed-stride training) and report stronger baseline comparisons?
  4. Please report evaluation-set size, random seeds, control resampling details, runtime throughput, memory, and cache-switching overhead.
4 comments
AuthorsOpenReview ↗

Response to Reviewer 86mX (1/2): Absolute Timing and Uncalibrated Switching

Thank you for recognizing RAR's formulation, evaluation, training efficiency, and portability. We add absolute-speed measurements, uncalibrated switching results, and reproducibility details.

[W1 & Q1] Absolute timing and matched wall-clock results

SR normalizes scene-dependent motion scales to isolate cross-rate scaling, since optical flow depends on scene content and generated trajectories can deviate from GT.

(1) Absolute speed. At each rate, RF=vpred/vGTRF=v_{pred}/v_{GT} compares speeds over the same physical interval. We take the median over all 64 cases without cross-rate normalization (ideal 1). Below is mean ∣RF−1∣|RF-1| over five rates per environment (%, lower is better):

ScopeSingleEmbedRAR
Doom51.5613.065.27
CS239.7910.6212.89
FPV37.3426.746.40
Overall42.9016.818.19
6 interpolation rates30.3526.678.85

RAR has the lowest overall and unseen-rate errors, showing that its physical-time coordinates preserve absolute pacing as well as relative scaling. Per-rate RF is in App. B.

(2) SLAM validation. We use ViPE to estimate GT and generated camera trajectories/speeds on aligned CS2 straight-line walking clips. Their shared first frame provides the scale reference. (Details in App.B) We average the per-case RAR/GT speed ratios (ideal 1; †: unseen):

FPS1624†3248†64
Mean speed ratio0.9690.9301.0771.0630.942

Ratios remain near 1, with no monotonic rate trend or clear training/unseen-rate gap, addressing global slowdown/speedup. At some rates, SLAM ratios are closer to 1 than flow-based RF: flow is affected by depth and non-camera motion, while SLAM provides camera-trajectory evidence. These scene effects also motivate SR.

(3) Wall-clock progression. Timing and motion accuracy are separate. With an accurate throughput estimate, s=fbase/rs=f_{base}/r gives ρ=1 (Eq. 8). Strides are selected online from completion times; switching overhead is at most +0.29% (Q4). RF/SLAM assess execution, and uncalibrated switching tests changing rates.

Our single GPU cannot run the feedback loop at the target rates. We measure completion times, inject waits to vary throughput, and use fixed time scaling, excluding input, display, and network delays. Each of 16 Doom cases runs for about 10 s on this scaled clock: 70→35→20→50→14 FPS, 2 s each. RAR estimates throughput from the last two step intervals and selects stride; the same-model baseline fixes s=1s=1. Both replay recorded controls at physical time, with corresponding recordings as GT.

We report final model-time error and accumulated optical-flow displacement relative to wall-clock GT, without cross-rate normalization or per-case calibration. Values are medians over 16 cases. Displacement ratios have an ideal value of 1; brackets give the IQR.

MethodModel-time error eTe_TWall-clock displacement ratio
Fixed rate (s=1s=1)−4.63 s0.71 [0.64, 0.85]
RAR−0.34 s0.89 [0.84, 1.20]

RAR reduces fixed-rate lag from 4.63 to 0.34 s after 10 s; the residual accumulates during throughput-estimate updates. Displacement improves from 0.71 to 0.89, closer to GT. Thus adaptive strides improve pacing, consistent with the timing/execution analysis.

[W2 & Q2] (a) Switching without filtering or calibration

App. E targets schedule tracking: filtering limits poor generations and unreliable measurements, calibration aligns motion scales, and regression extracts response trends.

The first three columns remove filtering and then regression. Counts: 64→32 (LPIPS)→24 (GT reliability). The last is a separate GT-referenced estimate: generated displacement divided by GT displacement per base-frame interval, using the same interval, with the median over 64 rollouts rather than a mean curve. No scale is fitted. Cells: Pearson r / nRMSE.

Env.SchedulePaper (n=24)No filtering (n=64)+ No regressionGT-referenced, uncalibrated
DoomSmooth0.941 / 0.2190.967 / 0.2820.927 / 0.2910.867 / 0.175
DoomAbrupt0.938 / 0.1850.976 / 0.2110.977 / 0.2130.982 / 0.074
CS2Smooth0.899 / 0.2480.945 / 0.3280.937 / 0.3530.856 / 0.192
CS2Abrupt0.963 / 0.1570.968 / 0.2770.970 / 0.2810.860 / 0.197
FPVSmooth0.998 / 0.1690.994 / 0.2770.988 / 0.3070.997 / 0.108
FPVAbrupt0.970 / 0.1220.973 / 0.2310.974 / 0.2290.980 / 0.118

Without filtering, Pearson rises in 5/6 settings; nRMSE increases to 0.211–0.328, consistent with excluding poorer generations and unreliable measurements. Without regression, Pearson remains 0.927–0.988. GT-referenced, uncalibrated tracking yields Pearson 0.856–0.997 and nRMSE 0.074–0.197. RAR tracks the schedule with no filtering, regression or fitted scale.

AuthorsOpenReview ↗

Response to Reviewer 86mX (2/2): Mixed-Stride Histories, Ablations and Reproducibility

[W2 & Q2] (b) Mixed-stride histories

Stride-RoPE expresses mixed histories in physical time. Stride increments stay within the training range, forming new combinations of learned time differences, analogous to unseen-rate interpolation. The demonstrated switching ability requires no dedicated mixed-stride training.

For non-sink frames, query–key phase depends only on physical-time differences (Eqs. 5, 9). The new frame's position offset sns_n from the latest frame specifies the time advance learned during multi-rate training. Cached states describe the past, independent of future strides; accumulated positions allow correct access, and new entries do not alter existing ones (Fig. 3b). The preceding uncalibrated results support this mechanism.

[W3 & Q3] Ablations and baselines

(1) Requested comparisons. Table 5 provides:

  • Stride-RoPE alone (Single): base-rate training; Stride-RoPE at inference only.
  • Multi-stride alone (Embed): RAR's multi-rate data, frame-index RoPE and FPS conditioning. Here, "alone" means without Stride-RoPE, not without a rate signal.

SR MAE (%, lower is better):

ScopeStride-RoPE alone (Single)Multi-stride alone (Embed)RAR
Doom / CS2 / FPV109.08 / 52.24 / 67.2813.92 / 7.24 / 9.403.54 / 3.54 / 4.02
9 training rates78.477.315.09
6 interpolation rates (†)72.8014.501.62

Training/interpolation groups refer to Embed/RAR; Single uses base-rate training. Single vs. RAR tests multi-rate training with Stride-RoPE; Embed vs. RAR compares rate representations on the same data. Single exceeds 50% error; Embed gives 7.31%/14.50% training/unseen errors versus RAR's 5.09%/1.62%. RAR outperforms both controls, especially at unseen rates.

(2) Baselines. Most interactive world models use one fixed rate; to our knowledge, no multi-rate interactive method is directly comparable. Our controls match the backbone, three-stage pipeline, optimization steps, and other settings; only rate sampling and conditioning differ.

(3) Mixed-stride training. RAR already generalizes from uniform strides to unseen rates and uncalibrated mixed-stride rollout (Q2b). Training with mixed strides may improve control but requires hyperparameter selection and another full three-stage run; we leave it for future work.

(4) Extrapolation. App. D reports out-of-range degradation to delimit the method; improved extrapolation is future work, not a current claim. Within the training-rate interpolation range, s=fbase/rs=f_{base}/r gives ρ=1 and the corresponding rollout rate.

[W3 & Q4] Reproducibility and runtime costs

App. A includes these details; all evaluations use seed 42.

(1) Evaluation settings.

ExperimentStageCasesMetrics
Tabs. 1, 5: multi-rate/comparisonSelf-Forcing64; SR: 32 lowest GT speed-CV casesLPIPS, Drift, DOVER, SR
Sec. 4.2.2, App. E: switchingSelf-Forcing64→32 (LPIPS)→24 (GT reliability)Pearson, nRMSE
Tab. 2, Figs. 8/9/10Bidirectional32LPIPS
Tab. 3: SANA-WMSelf-Forcing64; SR: 32 with the same filteringSR, LPIPS

SR selection uses only GT, not generated outputs; all methods share cases. GT physical speed is identical across rates, so its cross-rate speed CV (coefficient of variation) measures optical-flow noise.

(2) Control resampling.

ControlEnv.Interval representation
Sustained statesDoom, CS2Stride-synchronized sampling
Instantaneous events (fire/click)Doom, CS2Triggered if pressed in the interval
Mouse incrementsDoom, CS2Sum over interval
Camera trajectoriesFPVReference-relative pose at sampled time
Non-integer stridesAllNearest base-rate frame

Cumulative rotation, events, and time-aligned poses are preserved across rates; Stride-RoPE gives interval duration.

(3) Runtime costs. We use one H20, batch 1, and evaluation denoising/cache settings. Each latent step generates 4 frames; latency excludes VAE decoding.

Env.ms/latent stepFPSPeak GiBSwitch vs. ordinary step
Doom801.64.9914.47+0.28%
CS2808.44.9514.47+0.29%
FPV785.35.099.82−0.01%

Cached pre-RoPE keys are rotated at accumulated positions every step, even at fixed rates. Switching changes only new-frame position increments, without cache clearing or historical-feature recomputation. Latency differences are below 0.3%, within measurement noise and consistent with this implementation.

Revisions. Sec. 3.3: control semantics after Eq. 6; mixed histories after Eq. 10. App. A: evaluation/control settings, runtime costs. App. B: RF definitions/results, SR selection, CS2 SLAM validation. App. C: Table 5 ablation labels. App. E: unfiltered/uncalibrated results and filtering counts.

We hope this helps and welcome further discussion.

Thank you again for your thoughtful review. We wanted to briefly follow up on our rebuttal. We have added absolute-speed measurements, uncalibrated switching results, and further details on mixed-stride histories and runtime costs, as you suggested.

We hope these help clarify the main concerns you raised. We'd be glad to hear your thoughts, or address anything that remains unclear, before the discussion closes. Thank you!

As the discussion period is coming to an end, we would like to briefly follow up on our rebuttal. We have conducted additional experiments and provided detailed responses to the reviewers' main concerns. We would really appreciate it if you could take another look and let us know whether any concerns remain. We are happy to clarify anything further before the deadline. Thank you for your time and consideration!

Comments on the paper

1 comment

Dear Reviewers,

Thank you for your thoughtful reviews and constructive suggestions, which have helped us improve our paper!

We have carefully addressed each of your concerns in our point-by-point responses. We would greatly appreciate it if you could take a moment to review our responses and share your thoughts. If you have any remaining concerns or further questions, please let us know. We would be more than happy to provide further clarifications and engage in additional discussions.

Thank you again for your time and valuable feedback!

Best regards,
The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.