acceptodds
Under review as a conference paper at ICLR 2027

RISE: Adaptive Imagination for World Action Models

Abstract

World Action Models (WAMs) improve planning by incorporating future world evolution into action generation, yet existing methods allocate a fixed imagination budget to every scene. We propose RISE (Refining Imagination through SElective Rollout), a system-level adaptive imagination framework that makes sequential Roll/Stop decisions according to the expected planning benefit of continued rollout. At each step, a Latent Evaluator estimates the risk revealed by the current prefix and how much planning could improve if imagination continues, while a Rollout Gate weighs this expected benefit against additional computation cost. Since factual driving logs expose only one realized future, we further construct CounterDrive, a counterfactual dataset with diverse outcomes and risk levels, to enrich future dynamics and provide localized risk supervision. Each retained sample undergoes expert verification and annotation of trajectory validity, incident onset, and causal category, providing a reusable resource for safety-critical world-modeling research. Experiments on NAVSIM and nuScenes show that RISE achieves the best overall planning performance while reducing unnecessary rollout, with additional transfer results supporting its plug-in generality across WAM architectures.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Action Model, Autonomous driving

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_8RXQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes RISE, an adaptive imagination framework for world action models that makes sequential rollout decisions based on predicted planning gains and computational cost. It also introduces CounterDrive, a generated counterfactual driving dataset for future prediction and risk learning. Experiments on NAVSIM and nuScenes demonstrate improved planning performance, while comparisons with fixed rollout depths show a better trade-off between planning quality and latency.

Critical strengths
  1. The paper addresses the useful question of how the benefit of future imagination varies across driving scenes. The experiments show that adaptive rollout improves planning performance while reducing latency compared with the best fixed-depth setting, supporting its practical value.
Critical weaknesses
  1. The distinction from existing adaptive planning methods is not clearly explained. While the fixed-depth comparisons support adaptive allocation, they do not establish the advantage over simpler learned alternatives. This makes it difficult to assess the added value of the proposed sequential stopping mechanism.
  2. The evidence supporting CounterDrive’s reliability is limited. Although human verification is described, the paper does not sufficiently document annotation consistency, preservation of the source scene, or source-scene-level train/test separation.
Questions for authors
  1. What distinguishes the proposed sequential stopping mechanism from prior adaptive imagination and lookahead methods? Could you compare it with a model that predicts the final rollout depth directly from the initial observation, using the same training targets and a comparable computation budget?
  2. Could you clarify CounterDrive’s verification and splitting protocols, including annotator agreement, acceptance/rejection criteria, and checks for unintended changes to scene geometry and ego motion? Are all generated variants of the same source scene kept within a single data split?
5 comments
AuthorsOpenReview ↗

Response to Reviewer 8RXQ (Part 1/2)

We sincerely thank you for your careful review and valuable suggestions. Your comments have been highly beneficial in helping us improve the clarity, rigor, and overall quality of our paper. We have carefully addressed your concerns through additional experiments and detailed clarifications. We hope these revisions satisfactorily resolve your concerns, and we would greatly appreciate it if you could reconsider your rating.

W1/Q1: Comparison with Prior Adaptive World-Action Modeling Methods

MethodDomainAdaptive ComponentDecision SignalDecision Mechanism
FFDC-WAMRobotic manipulationAction execution lengthConsistency between predicted futures and real observationsDynamically determines when to interrupt action execution and replan
RISE (Ours)Autonomous drivingImagination rollout depthFuture Planning Gain, risk, and computation costSequentially decides whether to continue imagination after each predicted latent step

The key distinction between RISE and prior adaptive world-model methods lies in what is adapted and how the adaptation decision is made. For example, FFDC-WAM dynamically adjusts action execution length based on the consistency between imagined futures and real observations, determining when to replan. RISE focuses on adaptive imagination rollout depth during inference, where the Scheduler makes sequential ROLL/STOP decisions based on the expected Future Planning Gain and computation cost. Crucially, each newly predicted latent provides additional information about future world evolution, enabling RISE to reassess the value of continued imagination at every step. To the best of our knowledge, RISE is the first to introduce such adaptive imagination for autonomous driving World Action Models. Our results further demonstrate its effectiveness: RISE improves EPDMS from 90.1 to 90.8 over the best fixed-depth configuration while reducing latency from 318.6 ms to 287.4 ms.

To further validate the benefit of sequential reassessment, we introduce a One-shot Horizon baseline that predicts the final rollout depth directly from the initial observation, using the same all-horizon planning supervision as RISE. The results over three random seeds are summarized below.

MethodAvg. RolloutEPDMS ↑ (Mean ± Std.)Latency (ms) ↓
One-shot Horizon2.0487.784 ± 0.098265.852
RISE (Ours)2.4090.758 ± 0.010287.429

RISE achieves a 2.97-point EPDMS improvement over One-shot Horizon, with substantially lower variance across random seeds. The additional computation is limited to 0.36 rollout steps and 21.6 ms on average. By reassessing planning utility after each predicted latent step, RISE can incorporate newly revealed future information into its stopping decisions, leading to more effective rollout allocation and consistently better planning performance.

We have extended Table 5 in Section 4.3 and clarified the differences from prior adaptive world-action modeling methods in Section 5.1.

AuthorsOpenReview ↗

Response to Reviewer 8RXQ (Part 2/2)

W2/Q2: CounterDrive Construction Details

CounterDrive follows the original training and test splits of NAVSIM and nuScenes, and each generated counterfactual video inherits the split assignment of its source scene. Thus, all variants derived from the same source scene remain within the same split, preventing source-scene overlap between training and test sets. We have clarified the data splitting and annotation protocols in Section 2 (p. 3), with detailed quality-control procedures provided in Appendix G.3 (p. 26).

For annotation, we adopt a sequential collaborative protocol inspired by Chain-of-Talkers (CoTalk), involving 12 annotators in total. Each sample is first annotated by one annotator and subsequently reviewed and supplemented by a second annotator, followed by a designated reviewer. When disagreements arise, a third person joins the discussion until consensus is reached; samples for which consensus cannot be achieved are discarded. Since annotations are produced sequentially rather than independently, conventional inter-annotator agreement statistics cannot be computed from independent duplicate annotations.

Regarding generation quality, our prompts explicitly constrain the camera viewpoint, road geometry, background, and initial traffic configuration. During human verification, we distinguish generation artifacts according to their potential impact on scene understanding and driving decisions. For example, severe distortions of road structures or lane geometry that may affect the model's judgment are treated as critical artifacts. We attempt to regenerate such videos multiple times, and discard them if the distortions persist. In contrast, minor deformations of task-irrelevant objects, such as leaves, are retained when they do not affect scene interpretation or annotation, considering the substantial cost of video generation. Annotators also verify ego-motion consistency and identify incident onset and generation distortions. Samples with unreliable ego-motion or severe visual distortions are excluded.

Finally, we are committed to releasing the complete CounterDrive dataset after the review process. We take full responsibility for its quality and welcome scrutiny, feedback, and suggestions from the research community to further improve the dataset.

Dear Reviewer 8RXQ,

As the discussion period draws to a close, we would greatly appreciate any feedback you may have on our rebuttal and revised manuscript. We would be happy to address any remaining questions or concerns.

Thank you again for your time and consideration!

Best regards,

Authors

Reviewer_HuwcOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes dynamically adjusting the rollout length of world models based on their expected benefits to downstream planning tasks. Rather than using a fixed rollout length, the proposed scheduler balances potential planning improvements with computational costs. Experiments on NAVSIM and nuScenes demonstrate improved planning performance with reduced computational overhead.

Critical strengths
  • The paper tackles an important problem of how world model rollout length affects downstream planning performance
  • The proposed approach demonstrates strong performance on nuScenes and both NAVSIM v1 and v2 benchmarks
Critical weaknesses
  • One of my major concerns is that future planning gains are learned based on a specific planner. Does this mean that the model needs to be adjusted or retrained when the planner changes? Furthermore, since future planning gains are planner-specific, could different planners lead to different conclusions about the effectiveness of optimal rollout length?
  • The paper lacks evaluation results on NavHard.
Questions for authors
  • Please discuss how the choice of planner influences future planning gain estimation and the conclusion about the effectiveness of the optimal rollout length
  • I would like to see additional experiments on NavHard to further evaluate the effectiveness of the proposed adaptive rollout strategy in more challenging driving scenarios
1 comment
AuthorsOpenReview ↗

Response to Reviewer Huwc

We sincerely appreciate your encouraging evaluation and constructive feedback. Your positive assessment has been a great encouragement to us and has further motivated us to strengthen this work. We have carefully addressed your questions and conducted additional experiments to provide stronger evidence. We hope these new results further reinforce your confidence in our contributions, and we would be truly grateful if you could consider raising your rating.

W1/Q1: Planner Dependence of Future Planning Gain

Future Planning Gain is derived from downstream planning performance and may vary across different Planners. To investigate this dependency, we replace the original diffusion Planner with a standard Transformer Planner commonly used in WAMs, consisting of 12 layers, a hidden dimension of 384, and 12 attention heads. The results are summarized below. We also add this analysis to Section 4.4 (Table 9, p. 9).

PlannerEPDMS ↑Avg. RolloutLatency (ms) ↓
Diffusion90.762.40287.429
Transformer (Direct)90.732.41286.120
Transformer (Adapted)90.792.43287.425

(1) Does changing the Planner require retraining?

Transferring RISE requires only lightweight variable-prefix training of the target Planner, without architectural modifications. The original Scheduler can then be directly reused with all its parameters frozen. As shown above, direct transfer achieves 90.73 EPDMS, closely matching the original diffusion Planner's 90.76 EPDMS. Further adapting the Risk Profile head, Future Planning Gain head, and Rollout Gate to the new Planner yields a modest improvement to 90.79 EPDMS. These results demonstrate that additional Scheduler retraining is optional.

(2) Does Planner choice affect the conclusions about rollout length?

Despite the architectural differences, the average rollout depths remain highly consistent across the three configurations (2.40, 2.41, and 2.43). Even after adapting the Scheduler specifically to the Transformer Planner, the average rollout depth changes by only 0.03 steps compared with the original diffusion Planner. Moreover, both Transformer configurations outperform fixed full rollout (89.74 EPDMS), confirming the effectiveness of adaptive imagination with the new Planner. These results suggest that the overall benefit of adaptive rollout and its average computation allocation remain stable across the evaluated Planner architectures.

W2/Q2: Evaluation on NavHard

We further evaluate RISE on NavHard, a more challenging autonomous driving benchmark, and compare it with EponaV2, DriveFuture, and our fixed full-rollout baseline. The results are summarized below and have been added to Section 4.4 (Table 8, p. 9).

MethodNavHard Score ↑Stage 1 RolloutStage 2 Rollout
EponaV236.1––
DriveFuture34.6––
Fixed (h=4h=4)34.84.004.00
RISE37.53.283.04

RISE achieves the highest NavHard score of 37.5, outperforming EponaV2 (36.1), DriveFuture (34.6), and the fixed full-rollout baseline (34.8). Compared with fixed full rollout, RISE improves the score by 2.7 points, while reducing the rollout depth from 4.00 to 3.28 in Stage 1 and 3.04 in Stage 2. These results demonstrate that adaptive imagination remains effective in challenging driving scenarios, improving planning performance while reducing unnecessary rollout computation.

Comments on the paper

7 comments
AuthorsOpenReview ↗

Response to Reviewer 5zbz (Part 1/2)

We sincerely appreciate your thorough review and insightful feedback. Your suggestions have helped us identify important areas for improvement and significantly strengthen our experimental evaluation and analysis. We have made substantial efforts to address your concerns and provide additional supporting evidence, with further validation currently underway. We hope our responses and new results alleviate your concerns, and we would be grateful if you could consider raising your rating.

W1/Q1: Comparison of Stopping Strategies under Matched Computation

We extend Table 5 with three additional baselines and evaluate performance across multiple random seeds. Matched Random adjusts the random stopping distribution to match RISE's average rollout budget. Shuffled Horizon randomly reassigns RISE's selected rollout depths across scenes while preserving their distribution. One-shot Horizon directly predicts the final rollout depth from the initial observation using the same planning-score and computation-cost supervision as RISE.

MethodAvg. RolloutLatency (ms) ↓EPDMS ↑
Random Stop2.03 ± 0.12264.07589.27 ± 0.25
Matched Random2.40 ± 0.01288.01489.63 ± 0.01
Latent Margin2.98 ± 0.05308.53289.72 ± 0.01
Shuffled Horizon2.40 ± 0.01286.08689.73 ± 0.02
One-shot Horizon2.04 ± 0.12265.85287.78 ± 0.10
RISE (Ours)2.40 ± 0.01287.42990.76 ± 0.01

Under matched computation budgets, RISE outperforms Matched Random and Shuffled Horizon by 1.13 and 1.03 EPDMS points, respectively, with nearly identical rollout counts and latency. Compared with One-shot Horizon, RISE improves EPDMS by 2.98 points, demonstrating the advantage of making ROLL/STOP decisions using progressively generated future information. Moreover, RISE achieves a standard deviation of only 0.01 EPDMS across three random seeds, indicating stable performance. Compared with the best fixed-depth baseline (h=3h=3, 90.1 EPDMS), RISE also achieves higher planning performance with lower latency (287.429 vs. 318.554 ms). These results demonstrate that RISE effectively allocates imagination computation across scenes through sequential ROLL/STOP decisions, achieving better planning performance than alternative stopping strategies.

W1/Q2: CounterDrive Generalization and Data Reliability

Regarding generalization to real-world incidents, we further collect real-world accident recordings with synchronized ego-action data covering both pre-incident and post-incident periods. Such data are particularly challenging to obtain due to the rarity of accidents and the limited availability of complete action records. We present two representative cases in Fig. 7 of the revised manuscript (Section 4.4, p. 10): (1) a preceding vehicle suddenly reversing, and (2) a vehicle abruptly cutting into the ego lane. In both cases, the baseline fails to respond to the emerging collision risk in time, following trajectories associated with the recorded accidents. In contrast, RISE, trained with CounterDrive, anticipates the potential collision and generates appropriate avoidance trajectories. These qualitative results provide additional evidence of RISE's ability to recognize and respond to safety-critical situations in real-world driving scenarios, complementing the hazard discrimination results on the held-out CounterDrive test set.

Regarding data splitting and annotation reliability, CounterDrive follows the original training and test splits of NAVSIM and nuScenes. Each generated video inherits the split assignment of its source scene, ensuring that all variants from the same source remain within a single split. For annotation, we adopt a sequential collaborative protocol inspired by CoTalk [1], involving 12 professional annotators. Each sample is annotated by one annotator, supplemented by a second, and independently reviewed by another. Disagreements are discussed with an additional participant, and samples without consensus are discarded. Since our protocol uses sequential annotation, independent duplicate annotations are unavailable for computing conventional inter-annotator agreement statistics. Instead, annotation quality is maintained through staged verification and consensus-based review. We additionally check ego-motion consistency and generation artifacts, rejecting clips with severe scene distortions or unreliable motion.

[1] Chain-of-Talkers (CoTalk): Fast Human Annotation of Dense Image Captions. In EMNLP 25.

AuthorsOpenReview ↗

Response to Reviewer 5zbz (Part 2/2)

W2/Q3: Prediction Fidelity of Risk-Guided Latent Refinement

To examine whether risk-guided latent refinement affects future prediction fidelity, we conduct additional experiments comparing the predicted future representations before and after refinement. Specifically, we measure the accumulated normalized L1 distance between the predicted latents and ground-truth future latents at each rollout depth, together with the corresponding planning performance. We have added the prediction fidelity analysis before and after refinement in Section 4.4 (Fig. 5, p. 9).

Rollout DepthL1 Before ↓L1 After ↓EPDMS Before ↑EPDMS After ↑
h=1h=10.20350.203189.4189.56
h=2h=20.21970.218689.5990.01
h=3h=30.23280.236490.0990.33
h=4h=40.24090.245189.7990.30

As shown above, risk-guided refinement consistently improves planning performance across all rollout depths while introducing only minor changes in prediction error. Specifically, the L1 distance slightly decreases at depths 1 and 2, while increasing by only 0.0036 and 0.0042 at depths 3 and 4, respectively. Meanwhile, EPDMS improves by 0.15–0.51 points across the four depths. Furthermore, our refinement employs only two gradient steps with norm-constrained updates, limiting deviations from the originally predicted future representations. These results provide empirical evidence that refinement improves downstream planning while largely preserving feature-level prediction fidelity.

Q4: DAWN Transfer Details

All Scheduler components are directly transferred with frozen parameters, requiring no retraining. For DAWN transfer, we only need to retrain the DAWN Predictor and a variable-prefix Planner. The Predictor is supervised by ground-truth future latents, while the Planner is trained using ground-truth trajectories across different rollout depths. Both use supervision already available in the original training data, requiring no additional annotations or supervision signals.

Regarding computational overhead, the Scheduler introduces only lightweight evaluation and stopping decisions. The reduced rollout computation lowers the end-to-end inference latency from 951.16 ms to 627.85 ms, achieving a 34.0% latency reduction. Meanwhile, the planning score improves from 89.1 to 90.3 PDMS, demonstrating that RISE can be efficiently transferred to DAWN while improving both planning performance and inference efficiency.

  1. RISE averages 2.40 rollout steps versus 2.04 for One-shot Horizon. How much of the 2.98-point EPDMS gain remains at the same average rollout depth or latency?

  2. Is the reported 90.1 EPDMS the pre-refinement score of 90.09 at depth 3, rounded to one decimal place? Does the 318.554 ms latency come from the same setup? If the other settings match, should we compare RISE with the refined score of 90.33, giving a gain of 0.43 EPDMS?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.