acceptodds
Withdrawn from ICLR 2027

World2Policy: Continual Policy Improvement via Scenario Curriculum in Interactive Gaussian Worlds

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive Gaussian worlds enable photorealistic closed-loop simulation for autonomous driving. However, sustained policy improvement requires addressing two coupled challenges: how to resolve heterogeneous safety constraints during policy learning, and which scenarios should support continual policy update. A monolithic policy often entangles avoidance behaviors across dynamic actors, static obstacles, and road boundaries, while fixed scenario distributions yield diminishing returns as the policy improves. We present **World2Policy**, a co-evolutionary framework that unifies policy optimization with failure-conditioned scenario generation. In the *Policy Optimization Engine*, three constraint-specialized experts decouple safety modeling across foreground dynamics, static hazards, and drivable-area constraints through constrained reinforcement learning, while a privileged selector evaluates their complementary proposals to balance safety, progress, and comfort. In the *Environment Engine*, closed-loop rollouts expose residual failures, which guide the synthesis of physically plausible scenarios near the policy's capability boundary and update the training curriculum for policy optimization. Experiments on the closed-loop HUGSIM benchmark demonstrate that integrating World2Policy with UniAD, VAD, and LTF achieves HD-Score gains of +4.9, +6.1, and +6.9 points, respectively. Crucially, the co-evolutionary loop continually advances the policy's capability and sustains improvements across successive training rounds.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Autonomous Driving, Reinforcement Learning, Curriculum Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_nuN9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper face the problem of continual policy improvement for autonomous driving in interactive Gaussian worlds. It focuses on handling heterogeneous safety constraints without conflicting optimization objectives, and adapting the training scenario distribution as the policy improves. The authors propose World2Policy, a co-evolutionary framework consisting of a Policy Optimization Engine and an Environment Engine. The former trains three constraint-specialized experts using constrained PPO to address different driving risks, and employs a privileged selector to choose trajectories based on safety, progress, and comfort. The latter identifies recoverable failures from closed-loop rollouts and uses CEM to generate physically valid, challenging scenarios for subsequent policy updates.

Experiments on the HUGSIM benchmark show that World2Policy improves the HD-Score of UniAD, VAD, and LTF by 4.9, 6.1, and 6.9 points, respectively. Ablation studies further examine the contributions of multi-expert optimization and failure-conditioned scenario generation.

Critical strengths

The paper addresses an important limitation of closed-loop policy learning: fixed training scenarios may become less informative as the policy improves. The proposed co-evolutionary framework provides a reasonable way to address this issue by generating new scenarios based on the policy's observed failures. The recoverability filtering and physically constrained scenario editing are particularly useful, as they help avoid unrealistic or unsolvable adversarial cases. The results in Table 4 provide evidence that failure-conditioned generation produces more challenging and effective training scenarios than random perturbations.

Critical weaknesses
  1. A major concern is the reliance on privileged trajectory selection during evaluation. As described in Section 5.1, the selector chooses one trajectory from 48 candidates at each decision step. It evaluates candidates using dynamic-actor risk, geometric safety checks, drivable-area compliance, progress, and comfort. These criteria also closely align with the components of the HUGSIM HD-Score. The paper does not clearly specify which privileged signals are available to the selector at test time, or whether it accesses simulator ground-truth states and geometry. If such information is used, the reported improvements may partly arise from an information advantage at inference time rather than the learned policy itself. This raises concerns about evaluation fairness, performance attribution, and real-world applicability. The authors should clarify the selector's inputs and provide comparisons without test-time privileged information, under matched candidate budgets.

  2. Although continual policy improvement is a central claim of the paper, the experiments only demonstrate two rounds of scenario-curriculum updates. This is insufficient to establish whether the gains can be sustained over longer training horizons or whether the policy eventually saturates or regresses. Moreover, the main mechanism ablations are conducted on a 64-scenario subset using only the LTF backbone. The paper also lacks results across multiple random seeds and uncertainty estimates. These limitations make it difficult to assess the stability and generality of the reported improvements. Longer curriculum experiments and statistical evaluations would strengthen the paper's central claim.

Questions for authors

Could the authors report results using an observation-based selector that does not access privileged simulator information? It would also be valuable to compare single-expert and multi-expert policies under the same candidate budget. Such experiments would help disentangle the contributions of expert specialization, candidate diversity, and privileged trajectory selection to the final HD-Score improvement.

Reviewer_RaUVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents World2Policy, a closed-loop framework that combines policy optimization with failure-driven scenario generation in interactive Gaussian worlds.

The policy engine trains three constraint-specialized residual experts using constrained PPO, while a privileged selector chooses among their trajectory proposals. The environment engine mines policy failures and uses CEM to generate challenging but recoverable scenarios for subsequent training rounds.

On HUGSIM, the framework improves the overall HD-Score of UniAD, VAD, and LTF by 4.9, 6.1, and 6.9 points, respectively. Ablation studies further support the benefits of multi-expert optimization and adaptive scenario generation.

Critical strengths
  1. Practical and relevant problem.

Continually improving driving policies through closed-loop interaction is an important problem. Combining policy optimization with adaptive scenario generation is a sensible direction, and the framework is applicable to several pretrained driving models.

  1. Consistent improvements with useful ablations.

The method improves three different base policies on HUGSIM. The ablations also show that failure-conditioned scenario generation is more effective than random perturbation. These results provide useful empirical support for the overall framework.

Critical weaknesses
  1. Limited novelty relative to closely related work.

The main ideas—constrained RL, specialized experts, failure-driven scenario generation, and adaptive curricula—are largely established.

In particular, ScenePilot (ICML 2026) already explores physically solvable scenarios near a policy's failure boundary, while CurricuVLM (2026) adapts scenario curricula to policy weaknesses. AlignADV (2026 preprint) is also conceptually relevant.

The specific combination with multi-expert CPPO and Gaussian simulation is different, but the paper does not clearly establish what new learning principle or empirical insight this combination provides beyond existing approaches.

  1. Insufficient evidence to isolate the key contributions.

The privileged selector may account for a substantial part of the improvement. The paper compares different expert configurations, but does not sufficiently control for the number of trajectory candidates or the additional information used by the selector.

Similarly, the environment engine is compared mainly against random and simpler failure-conditioned generation. A stronger comparison with failure replay or prioritized sampling under matched simulation budgets would help establish the value of generating new scenarios.

Finally, the continual-improvement claim is supported by only two curriculum rounds, and the main ablations use just 64 test scenarios without uncertainty estimates.

Questions for authors

How much of the policy improvement comes from multi-expert learning versus privileged trajectory selection? Could you compare against a single expert generating the same number of candidates (48), using the same selector? What information does the selector require at inference time?

How much does scenario generation improve over simply replaying or prioritizing existing failure scenarios? A comparison under matched rollout and training budgets would be helpful.

How robust are the gains across curriculum rounds and random seeds? Could you clarify the train/validation/test separation and provide uncertainty estimates? Longer training curves would also help support the continual-improvement claim.

Reviewer_WsKsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes World2Policy, a framework that combines multi-expert constrained reinforcement learning with failure-driven scenario generation for closed-loop autonomous driving. The authors evaluate the approach using UniAD, VAD, and LTF on HUGSIM and report consistent improvements in driving scores.

Critical strengths

I find the overall direction interesting, particularly the idea of updating the training scenarios based on the failures of the current policy. The paper is also reasonably well organized and includes several useful ablation studies.

Critical weaknesses

My main concern is the novelty of the proposed framework. Both constrained policy optimization and failure-driven scenario generation have been studied in autonomous driving. The paper combines these components into an iterative training framework, but I am not entirely convinced that the combination introduces a sufficiently new learning mechanism. It would help to explain more clearly what is fundamentally different from existing closed-loop RL and adaptive scenario generation methods, beyond using them together in Gaussian worlds.

The benefit of using multiple specialized experts is not fully established. In Table 3, the single-expert methods use 16 candidate trajectories, whereas the multi-expert approach uses 48. This makes it difficult to determine whether the improvement comes from expert specialization or simply from having more candidates to choose from. I would like to see a comparison with a single expert generating the same number of candidates, using the same selection strategy.

I have some questions about the privileged trajectory selector. The selector plays an important role in the final performance, but it is not sufficiently clear what information is available to it during evaluation. In particular, does it rely on ground-truth actor states or other information provided by the simulator? If so, this could limit the practical applicability of the method. A comparison using only information available to the driving policy would make the results more convincing.

The evidence for continual policy improvement is still somewhat limited. Figure 3 shows improvements over two curriculum rounds, but it is difficult to tell whether this trend would continue with further training. It is also unclear how much of the gain comes from the failure-conditioned curriculum rather than additional training or exposure to more scenarios. A comparison with continued training on a fixed scenario distribution under the same interaction budget would be useful. I would also like to see whether the policy maintains its performance on previously learned scenarios.

The scenario generation process needs further justification. The paper emphasizes generating physically valid and recoverable scenarios near the policy's capability boundary. However, the current validity checks do not necessarily establish that the generated interactions are realistic or representative of actual driving behavior. More details on the recoverability criterion and the CEM search process would be helpful. It would also be interesting to examine whether the generated scenarios improve performance beyond the particular types of failures used to construct them.

Finally, I think the experimental results need a more careful discussion. Although the overall HD-Score improves, the gains are not consistent across all difficulty levels. For example, LTF shows a slight decrease on Extreme scenarios, and some improvements come with reduced comfort. Most of the ablation studies are also conducted on only 64 scenarios, without reported statistical variation. Given that some of the differences are relatively small, reporting results across multiple seeds or confidence intervals would strengthen the conclusions. The computational overhead of sampling and evaluating 48 trajectories at each decision step is also worth discussing.

Questions for authors
  1. In Table 3, the multi-expert approach evaluates 48 candidate trajectories, while the single-expert baselines use only 16. Have you evaluated a single expert with 48 candidates and the same selector? How much improvement can be attributed specifically to expert specialization?

  2. What information does the privileged selector use during inference? Does it have access to simulator ground-truth states, and how would the framework perform with perception-based information only?

  3. Figure 3 reports improvements over two curriculum rounds. Have you tested additional rounds? Does performance continue to improve, and is there any degradation on earlier scenarios as the curriculum becomes more challenging?

  4. How do you separate the benefit of failure-conditioned scenario generation from simply increasing the amount of training data or interaction? Would continued CPPO training on the original scenarios achieve similar improvements under an equal training budget?

  5. How exactly is scenario recoverability verified, and what prevents the CEM search from generating unrealistic but technically valid interactions? Could you provide more details about the scenario acceptance criteria and search budget?

  6. Could you clarify the training, validation, and test split construction, particularly whether scenario generation uses failures from the training distribution or a separate validation set? Also, are the reported results consistent across different training seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.