acceptodds
Withdrawn from ICLR 2027

ShiftWM-Bench: Planning with Frozen Latent World Models under Hidden Within-Episode Dynamics Shifts

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained latent world models are typically evaluated under fixed within-episode dynamics, although changes during execution can invalidate the action-outcome relationships used for planning. We introduce ShiftWM-Bench, a benchmark of four task-specific, state-triggered dynamics changes in manipulation and navigation, with event timing and response parameters hidden from the planner. We study a planning stack with two complementary components and no test-time parameter updates. Proposal Shift combines latent retrieval, cross-entropy method (CEM) initialization, and reliability-weighted action-prior scoring to strengthen candidate search. Latent Belief Recovery infers a distribution over predefined response hypotheses from recent prediction-observation residuals, allocates candidate plans according to this distribution, and ranks them under the most probable hypothesis. The belief model is trained on nominal trajectories and synthetic response perturbations, without real shifted training trajectories. Across 300 scenarios per task and three planner seeds per scenario, Proposal Shift raises four-task mean success from 47.7% with native LeWM to 64.8%, and Recovery further raises it to 72.7%. Controlled post-trigger success gains are established on Cube under particular handoff protocols and control budgets; command-delay gains remain trajectory-dependent, and the current contact-friction approximation can degrade planning. These evaluations measure candidate-search improvements and conditional post-shift benefit separately, characterizing the operating boundary of the complete response-conditioned planner.

Primary area:
datasets and benchmarks
Keywords:
Latent World Models, Model-Based Reinforcement Learning, Visual Planning, Non-Stationary Dynamics

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_bJ6wOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces ShiftWM-Bench, a benchmark for planning with frozen latent world models under hidden within-episode dynamics changes. It considers four task-specific shifts: actuator scaling in Reacher, contact friction in PushT, action-mapping rotation in Cube, and command delay in TwoRoom. The paper also proposes a two-component planning framework. Proposal Shift combines latent retrieval, CEM initialization, and prior-aware scoring to improve candidate search. Latent Belief Recovery infers predefined response hypotheses from recent prediction errors and uses the resulting belief to adjust candidate allocation and scoring without updating the world model. The complete framework improves mean success from 47.7% with native LeWM to 72.7%, with additional evaluations of post-trigger recovery, computational budgets, and compatibility with other planning interfaces.

Critical strengths
  1. Relevant problem formulation. The paper addresses an important limitation of frozen world-model planning: changes in dynamics during execution can invalidate the model's action–outcome predictions. Separating candidate-search performance, event reachability, and post-shift recovery is useful for understanding planning under changing dynamics.
  2. Extensive experimental evaluation. The paper includes paired scenarios, multiple seeds, confidence intervals, matched no-shift comparisons, computational-budget controls, and shared-prefix and snapshot evaluations. It also investigates compatibility with additional planning interfaces and Fast LeWorldModel. The reporting of negative and inconclusive results is appreciated.
Critical weaknesses
  1. Limited scope and generality of the benchmark.

The benchmark consists of only four task–shift pairings, with each shift family assigned to a different task. This confounds task difficulty with shift type and makes it difficult to assess whether the findings generalize across different tasks or dynamics changes. Moreover, the response families and corresponding correction operators are predefined for each task. The planner mainly needs to infer the active response within a known family, rather than adapt to an unfamiliar dynamics change. Some shifts, particularly action-mapping rotation and command delay, admit relatively straightforward action-space corrections. For a benchmark contribution, broader coverage and the ability to distinguish the capabilities of different methods are important. The current evaluation focuses largely on variants of the proposed framework, with limited comparisons against independent adaptation approaches. As a result, the benchmark's broader utility remains insufficiently demonstrated.

  1. Limited evidence of effective post-shift recovery.

The headline improvement from 47.7% to 72.7% combines better candidate search with post-shift adaptation. Proposal Shift alone increases mean success to 64.8%. On Cube, event reachability increases from 7.0% with native CEM to 79.7% with Proposal Shift, indicating that a substantial part of the improvement comes from stronger search rather than recovery. The controlled post-trigger results are considerably less convincing. Recovery improves Cube success by 4.6 percentage points under shared-prefix evaluation, but only by 2.0 points under identical-snapshot handoff with the original 50-step budget, where the confidence interval includes zero. The TwoRoom controlled results are inconclusive, Reacher shows no clear benefit, and PushT exhibits a regression. While Recovery improves full-episode success on Cube and TwoRoom, the evidence for consistent adaptation after a dynamics change remains limited. This weakens the central motivation of the proposed framework.

  1. Modest methodological novelty and unclear contribution of individual components.

Proposal Shift combines established techniques, including nearest-neighbor retrieval, CEM initialization, and action-prior regularization. Recovery relies on a classifier over predefined response hypotheses, manually designed response operators, and belief-conditioned candidate allocation and scoring. The methodological novelty beyond existing prior-guided planning and dynamics-adaptation approaches appears limited. The ablations also do not sufficiently isolate the contributions of belief inference, candidate allocation, response-conditioned priors, and scoring. In particular, the randomized-belief comparison does not establish a statistically significant advantage from accurate belief predictions. It therefore remains unclear whether learned belief inference is necessary or whether much of the benefit comes from searching across multiple predefined response hypotheses. Finally, Recovery introduces substantial computational overhead. Although it outperforms a higher-iteration Proposal baseline under matched predictor-example budgets, the comparison does not establish an advantage under equal wall-clock time. The higher cost further raises questions about the practical value of the relatively limited controlled recovery improvements.

Questions for authors

Q1. Benchmark generality.

Could the authors evaluate multiple shift families within the same task, or the same shift family across different tasks? How representative are these predefined shifts of dynamics changes encountered in practical robotic manipulation?

Q2. Post-shift adaptation.

Given the large contribution of Proposal Shift to event reachability and the limited gains under controlled post-trigger evaluations, what evidence supports Recovery as an effective adaptation mechanism rather than primarily an improvement to candidate search?

Q3. Component contributions.

Could the authors provide more controlled ablations that independently isolate belief inference, candidate allocation, response-conditioned priors, and scoring within the same solver? In particular, is learned belief inference necessary compared with simpler multi-hypothesis planning?

Q4. Computational efficiency and baselines.

How does Recovery compare against stronger search and adaptation baselines under equal wall-clock budgets? Would a simpler residual-based multi-hypothesis planner achieve comparable performance?

Reviewer_1PUtOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ShiftWM-Bench, where a hidden dynamics change is triggered by a physical state during an episode, with one change family per task: actuator scaling in Reacher, contact friction in PushT, action rotation in Cube, and command delay in TwoRoom. It also proposes a planner for frozen LeWM. Proposal Shift uses retrieved actions to initialize and regularize CEM. Latent Belief Recovery classifies recent prediction residuals into predefined response classes and uses this belief to allocate and score CEM candidates.

Critical strengths
  • Planning under dynamics changes during execution is an important problem.
  • The paper includes useful controls, such as matched no-shift runs and a higher-budget baseline.
  • The authors report negative results openly, including the PushT regression.
Critical weaknesses
  1. The benchmark is presented as the main contribution, but it is not positioned against existing dynamics-shift benchmarks such as CARL or the Real-World RL Suite and is mainly used to evaluate the authors' own planner.
  2. The change triggers, such as friction changing after sustained contact or the action mapping rotating once a grasp forms (Table 1), are chosen to force a mid-episode change and lack physical motivation. It does not explain why, for example, calibration drift would start exactly when a grasp forms.
  3. The belief module relies on hand-designed response families for each task (Appendix A.1), which is unclear to scale and already hurts PushT (72.8% to 69.6%).
  4. The evidence that Recovery handles the changed dynamics is weak: under the identical-snapshot handoff at the original 50-step budget, it changes Cube by +2.0 points [-1.7, 5.7] and TwoRoom by -1.0 [-3.2, 1.2].
Questions for authors

Please see the weaknesses above.

Reviewer_bv57OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ShiftWM-Bench for evaluating frozen world-model planners under hidden within-episode dynamics changes. Proposal Shift uses retrieved trajectories to guide CEM search. Latent Belief Recovery learns to classify prediction residuals into predefined response hypotheses, then uses these beliefs to allocate and score candidate plans. The classifier is trained on nominal trajectories with synthetic perturbations.

Critical strengths

The paper studies a relevant failure mode of frozen world-model planners. Paired scenarios, matched no-shift evaluations, and shared-prefix and snapshot controls help distinguish overall planning improvements from post-shift recovery. The reported gains on Cube and TwoRoom, together with the negative results on other tasks, provide useful evidence about the interface’s operating limits.

Critical weaknesses
  1. The value of correctly inferred beliefs is not established. The randomized-belief controls leave Full’s advantage unresolved (Table 14), while the CUSUM comparison changes the solver as well as the inference method. Controlled post-trigger success is also protocol-dependent: Cube improves under shared-prefix evaluation and extended-budget snapshot evaluation, but strict-snapshot success gains at the original budget are inconclusive (Appendices A.4 and H). These results support the complete planner more clearly than recovery driven by accurate response inference.

  2. The synthetic training procedure relies on assumptions that are insufficiently validated under real shifts. Training endpoints are generated using the same response operators used for inference, with nominal prediction residuals added as noise (Appendix A.1). This requires both accurate operators and a prediction-error model that transfers beyond nominal trajectories. Gains are concentrated on tasks whose operators directly represent the intervention; the approximate latent-scaling operators are less successful. With one shift family per task, the evaluation does not separate operator accuracy from task difficulty.

Questions for authors
  1. Within the unchanged Full solver, how does learned belief compare with selecting or weighting hypotheses directly from their prediction residuals? This would clarify the contribution of synthetic classifier training beyond the response operators and mixture search.

  2. At replanning calls that select executed actions, how often does the belief contain post-shift evidence, and how accurate and calibrated is it? The main protocol permits at most two planning calls, with identity belief at the first, so at most one call can use post-shift evidence. The endpoint diagnostics in Table 13 may include updates that never affect a plan.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.