acceptodds
Under review as a conference paper at ICLR 2027

Second Thoughts: Exploring and Planning through Comparative Imagination in World Models

Abstract

World models let agents compare imagined outcomes before acting, but an agent must still decide which possibilities to explore and when a proposed plan should replace its learned policy. We introduce Sibling Disagreement in Imagination (SDI), which connects these decisions through a shared actor reference. SDI compares action sequences imagined from the same model state, using repeated rollouts of the actor's own sequence to measure variation under fixed behavior. This reference scales both the spread across candidate returns, which provides an exploration signal, and a proposed plan's predicted advantage, which governs action acceptance. The agent executes a planned action when its advantage exceeds the reference variability by a specified factor; otherwise, it follows the actor. SDI reuses the base agent's world model, actor, and critic without requiring an additional disagreement ensemble. Exploration and planning guide training data collection, while evaluation uses the learned actor alone. Across Meta-World, Adroit, and DM-Control, SDI achieves a higher observed task-average score than every compared baseline in each suite and matches or exceeds the strongest baseline's performance on of tasks. Component ablations and robustness analyses further examine the roles of exploration, planning exposure, and action acceptance. Paired evaluations from identical simulator states show that the gate improves both recovery from actor failures and preservation of successful behavior relative to ungated planning, while substantially reducing return degradation. Together, these results support comparing imagined alternatives against a shared actor reference as a unified approach to exploring and planning with world models.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, world models, planning in imagination

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_C6JqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

In submission 35058, the authors address two main challenges in world-model-based reinforcement learning. These are: a ) how to explore effectively in latent imagination, and b) how to decide when a plan at decision time should take priority over a learned policy (actor). Their approach, called Sibling Disagreement in Imagination (SDI), brings together exploration and plan acceptance using a shared actor-based reference created at the current latent state ztz_t.

As claimed, SDI links intrinsic exploration and decision-time planning through a unified reference mechanism derived from repeated rollouts of the current actor sequence. This approach eliminates the computational and architectural complexity of training secondary ensemble world models or explicit epistemic-uncertainty heads.

Critical strengths

a) Unified Conceptual Framework without Ensemble Overhead: The proposed SDI cleanly connects intrinsic exploration and decision-time planning through a single, shared reference mechanism (s_\pi(z)) derived from repeated rollouts of the current actor sequence.

b) Evaluation Protocol: SDI uses gated planning solely during training data collection. Evaluation is strictly actor-only without search or gating, ensuring that benchmark gains reflect true policy improvement rather than test-time search compute.

c) Benchmark-Leading Performance: Notwithstanding my comments below, the paper evaluates performance across Meta-World v3, Adroit, and DM-Control (18 pixel-based continuous control tasks). The proposed SDI achieves the highest observed suite-average score in all three domains and matches or exceeds the strongest baseline (including DreamerV3, R2-Dreamer, TD-MPC2, and Plan2Explore) on 15 of 18 tasks.

Critical weaknesses

The paper lacks a quantitative analysis of training time, GPU memory utilization, or latency per environment step relative to actor-only execution (e.g., DreamerV3 / R2-Dreamer) or standard ungated MPPI.

The benchmarks used (Meta-World, Adroit, DM-Control) are standard continuous visual control tasks. However, the paper does not test SDI on environments with multimodal action dynamics, highly non-stationary or stochastic settings (where s_\pi could become very large and stop planning), or on long-horizon tasks with sparse rewards like Crafter or Atari. The authors may wish to consider alternatives to these suggestions, such as Atari tasks.

Actor Mode vs. Stochastic Action: During re-scoring, G_\pi and s_\pi are calculated using the actor's deterministic mode sequence. However, if a plan is rejected, the agent samples a stochastic fallback action a_t \sim \pi(\cdot|z_t) for environment execution. The manuscript should clarify whether evaluating the mode during gating is consistent with the stochastic policy's actual training-time behaviour.

Questions for authors

Please consider the following:

Computational Overhead: What is the training time overhead of SDI compared to the base R2-Dreamer agent and TD-MPC2 on a standard GPU (e.g., NVIDIA A100)? A comparative evaluation against a baseline is essential.

Actor Mode vs. Stochastic Policy Mismatch: During plan acceptance evaluation, G_\pi and s_\pi are measured using repeated rollouts of the actor's mode sequence. Why use the mode rather than sample from the stochastic policy \pi(\cdot|z_t), given that we use stochastic sampling for fallback execution when a plan is rejected?

Sensitivity to Threshold \kappa: How sensitive are the benchmark results (e.g., Meta-World precision tasks) to variations in \kappa \in {0.5, 1.0, 2.0, 3.0, 4.0? Is \kappa=2.0 robust across different reward scales without task-specific tuning? Again, a comprehensive, systematic evaluation of the parameter(s) and setting(s) is required for submission.

Behaviour in Highly Stochastic Dynamics: How does SDI behave when environment dynamics are inherently noisy or stochastic, or changebruptly, i.e. t, whenhe background noisy environment shifts from a benign Gaussian to a Gilbert-Elliot model? Does high stochasticity inflate s_\pi(z), causing the gate to reject all plans and fall back entirely to the actor?

1 comment

Thank you for the positive assessment, for recognizing the unified framework without ensemble overhead, the actor-only evaluation protocol and the benchmark-leading performance, and for the concrete questions, which we address in turn.

Computational overhead (W1, Q1). Table 2 (new, main text) measures every component on one A100-80GB (hand-insert, 16 parallel environments):

  • R2-Dreamer trains at 65 environment steps/s and SDI in its Table 1 configuration at 37 steps/s, i.e. 1.75× the time per step, with 370 ms per planning call for all 16 environments. Gated and ungated MPPI cost the same, because search and re-scoring run before the acceptance decision.
  • The exploration bonus adds 30% (2.05× with both); peak GPU memory rises only from 16.6 to 18.9 GiB.
  • TD-MPC2 in its Table 1 configuration takes 0.70× R2-Dreamer's time per step, so SDI trains at about 2.5× TD-MPC2's cost per step; TD-MPC2, however, also plans at every evaluation step, whereas SDI evaluates with the actor alone, at the base agent's cost. The planner is off for the first 20k steps, so a full run costs between 1× and 1.75× R2-Dreamer's training time.

Actor mode versus a sampled action (W3, Q2). The reference s_π should measure how much imagined return varies when behavior is held fixed. Drawing a fresh actor sample for each repeat would add the policy's own sampling variance, so the threshold would follow the actor's entropy, which changes over training, rather than the model's predictions. The mode is also the action executed at evaluation, and executing a fresh actor sample when a plan is rejected keeps those steps identical to the base agent's training behavior. Section 3.3 now explains this.

Sensitivity to κ and reward scale (Q3). κ = 2 was fixed in advance and used on all 18 tasks without task-specific tuning. Rescaling the reward rescales both the predicted advantage ĝ and κ·s_π, so every acceptance decision is invariant to the reward scale; a raw-margin threshold lacks this property (Figure 3a: one raw threshold executes between 16% and 79% of proposals on tasks whose margins differ by about 20×). The training-level sweep you suggested (Table 4a, hand-insert):

κhand-insert successplans executedoccupancy
0.5.33 [.21, .45].83.97
1.53 [.17, .88].63.88
2 (default).40 [.24, .56].41.92
3.49 [.37, .62].27.97
4.47 [.07, .87].29.97

Success at every κ ∈ {0.5, 1, 3, 4} (.33, .53, .49, .47) lies inside the 95% interval of the default (.40 [.24, .56]), while κ sets how many plans are executed (.83 at κ = 0.5 down to .27–.41 at κ ≥ 2), so the results are not sensitive to κ in this range.

Stochastic and non-stationary dynamics (W2, Q4). Action-independent noise inflates the sibling spread and s_π alike, so the bonus ratio discounts it by construction. We tested sticky actions (p = .25) and Gilbert–Elliott action noise whose σ switches between .05 and .5 (Table 4b):

taskdynamicsR2-DreamerSDI + bonusplans executeds_π
lever-pullclean1.00 [1.00, 1.00]1.00 [1.00, 1.00].368.14
lever-pullGilbert–Elliott.99 [.97, 1.00].99 [.97, 1.00].3217.20
lever-pullsticky (p=.25).92 [.83, .99]1.00 [1.00, 1.00].4314.19
hand-insertclean.16 [.12, .20].18 [.00, .43].4428.04
hand-insertGilbert–Elliott.33 [.23, .43].28 [.06, .51].4757.15

The rule does not stop planning: SDI still executes 32–43% of plans on lever-pull and 36–52% on hand-insert while s_π grows with the noise; both agents solve lever-pull under both noise models, and on hand-insert the two agents' 95% intervals overlap under both conditions.

Other domains (W2). Atari and Crafter have discrete actions, which need a different proposal distribution than MPPI's Gaussian, and strongly multimodal action spaces would need a richer proposal; the revised Limitations (Appendix A) state this scope. A preliminary Crafter run of the bonus alone on DreamerV3 (165k steps) scored below DreamerV3 (3.1 vs 4.5), so we make no claim there and leave discrete-action proposals to future work.

We hope these measurements and explanations address your concerns, and we would be grateful if you would consider raising your score.

Reviewer_ebEgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces SDI (Sibling Disagreement in Imagination) to guide exploration and plan acceptance in world models using a shared actor-repeat reference. By measuring return variation under fixed behavior, SDI normalizes sibling spread for exploration bonuses and scales the predicted advantage required for plan acceptance. The method reuses the base agent without an additional ensemble, using exploration and planning during training while evaluating the actor alone. Across Meta-World, Adroit, and DM-Control, SDI achieves the highest observed task-average score among compared methods, matching or exceeding the strongest baseline on 15 of 18 tasks. Ablations and paired-state experiments provide evidence that gated planning improves policy learning, recovery from actor failures, and preservation of successful behavior relative to ungated planning in the tested settings.

Critical strengths
  • I find the shared actor reference and post-search rescoring design both concrete and well motivated. By reusing the base agent's components without an extra ensemble, SDI uses actor-repeat return variability as a common scale for exploration and gating. Furthermore, fresh rescoring reduces the acceptance decision’s reliance on chance overestimation in the samples used during search, which I think strongly supports the rationale of the gate.
  • I appreciate the empirical evidence provided across policy learning and intervention analyses. The matched-occupancy ablation and paired-state evaluations provide evidence that gated planning improves recovery from actor failures and preserves successful behavior compared to ungated planning in the tested settings.
Critical weaknesses
  • The main experiments and ablations do not specify the number of independent training runs, the aggregation procedure, or measures of uncertainty such as standard deviations or confidence intervals. This absence makes it difficult to assess the stability, reproducibility, and representativeness of the reported performance gains.
  • The current ablations fail to fully isolate the specific contribution of the shared actor reference. Specifically, the exploration controls inadvertently alter planner occupancy, and the gating analysis lacks evaluations at matched acceptance rates, leaving it unclear whether the actor-dependent variability scale provides benefits beyond simply reducing the frequency of planning interventions.
Questions for authors
  • How many independent training seeds were used for each method and task, and how were their results aggregated into the reported scores? Could you provide per-seed results and measures of variability or uncertainty for the main experiments and ablations?

  • What is the effect of removing the exploration bonus, or replacing its actor-repeat normalization, while keeping the base agent, planning schedule, acceptance gate, and training budget unchanged? Are results from these controlled comparisons available?

  • Does the actor-scaled gate outperform a raw-margin or constant-spread gate at matched plan-acceptance rates? Could you provide realized-return and success comparisons to clarify whether the actor reference improves selection quality beyond simply reducing intervention frequency?

3 comments

Thank you for the detailed review, and for recognizing the shared actor reference and post-search re-scoring as concrete and well motivated. Your questions led to some of the most informative new analyses in the revision.

Runs, aggregation and uncertainty (W1, Q1). Each entry is the mean endpoint over independent training runs that differ only in the seed: success over the final evaluations at the task's budget, or DM-Control return at a common depth; suite means weight tasks equally (Section 4). Table 1 now shows a 95% bootstrap interval for every entry (resampling runs, then the evaluations forming each run's score), Table A2 adds the standard deviation and number of runs, Tables A3–A4 list every run's endpoint, and Section 5.1 gives intervals for the suite means. We added replication runs so that every SDI entry and the strongest baseline on every task rest on multiple independent runs. Only runs that reached the budget count, and no run is excluded for its outcome (Appendix G lists two same-seed duplicate launches, excluded by launch order). With these intervals, SDI is competitive with or exceeds every baseline on 16 of 18 tasks, and its Meta-World suite mean exceeds R2-Dreamer's with a 95% interval of [.01, .18] for the difference.

Removing or re-normalizing the bonus at a fixed schedule (Q2). Table 3 compares these with the same planner schedule, acceptance rule (κ = 2) and budget, so planner occupancy is matched (.95–.97). On the three precision tasks, SDI without the bonus reaches .29 [.20, .40], SDI with the bonus .23 [.12, .36], and the bonus with raw sibling spread in place of the actor normalization .17 [.03, .33]; the bonus without the planner stays at .06. Planning exposure therefore drives the gains, and the actor normalization gives the bonus a higher mean than the raw spread. Table 1 now evaluates SDI without the bonus.

Actor-scaled rule at matched acceptance rates (W2, Q3). On 9,000 paired decisions (18 checkpoints, five Meta-World tasks), we held each rule to the SDI rule's own acceptance rate at every checkpoint and measured the realized 8-step gain of the planned action over the actor's from the same restored simulator state:

Rule (same acceptance rate)P(realized gain > 0)mean realized gain
Actor-scaled rule (κ = 2).276−.047
Random selection.293 [.284, .302]−.043
Raw margin.294−.071
Constant per-task spread.294 (same selections as raw margin)−.071

At the same rate the selectors pick beneficial plans about equally often, but raw-margin and constant-spread selection lose about 50% more return per executed plan than the actor-scaled rule. The actor reference's main benefits lie elsewhere (Section 5.4, Figure 3): (i) a unitless threshold, as κ = 2 executes 39–52% of proposals on four of five tasks whose margins differ by about 20×, where one raw threshold executes 16% to 79%; and (ii) protection during sustained planning, as over 50-step continuations the rule loses 13 of the actor's 746 successes versus 31 when every plan is executed, and cuts the return loss relative to the actor from 16.7 to 3.6. At the training level (Table 3), the constant-spread threshold reaches a three-task mean of .18 [.15, .22] and random acceptance .20 [.15, .26], against .23 [.12, .36] for the same configuration with the actor-scaled threshold. The paper now states this role precisely: the rule sets how much the agent plans on a reward-scale-free threshold and protects the actor's successes.

We hope these controlled comparisons address your concerns, and we would be grateful if you would consider raising your score.

Thank you again for your thoughtful review. If anything remains unclear, or if a specific additional check would help your assessment, please let us know and we will try to address it before the discussion deadline. We would be grateful for any further feedback, and for an updated assessment if your concerns are resolved.

Having reviewed the authors' responses, I find that the newly added uncertainty quantification and well-controlled ablations adequately address my concerns regarding reproducibility and the isolation of the shared actor reference. I am therefore raising my rating to 3 (Weak acceptance).

Reviewer_LcVNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The agent executes a planned action when its advantage exceeds the reference variability by a specified factor; otherwise, it follows the actor. SDI reuses the base agent's world model, actor, and critic without requiring an additional disagreement ensemble. Across Meta-World, Adroit, and DM-Control, SDI achieves a higher observed task-average score than the baseline

Critical strengths

Extensive empirical experiments. Quite impressive results on harder tasks

Critical weaknesses

If the rollout achieves improvement it is important to quanitfy the uncertainty. There are only task average results if there were variation across seeds one could judge it.

Questions for authors

Is the improvement robust across seeds?

Could the gains simply come from using the planner less often, rather than from the proposed acceptance rule?

Could you report planning alone, planning with the gate, planning with the exploration bonus, and full SDI?

2 comments

Thank you for the review, and for recognizing the extensive experiments and the strong results on the harder tasks. We address each question below.

Robustness across seeds (W1, Q1). Yes, the improvement is robust across seeds. Table 1 now reports every entry with a 95% bootstrap interval that resamples runs and then the evaluations forming each run's score, Section 5.1 gives intervals for the suite means, and Tables A3–A4 list every run's endpoint. We added replication runs so that every SDI entry and the strongest baseline on every task rest on multiple independent runs. With these intervals, SDI has the highest average score in every suite, is competitive with or exceeds every baseline on 16 of 18 tasks, and its Meta-World suite mean exceeds the strongest baseline's with a 95% interval of [.01, .18] for the difference. On the precision tasks, the three-task mean rises from .08 [.03, .14] for R2-Dreamer to .29 [.20, .40].

Do the gains come from planning less often? (Q2) No; planning less often lowers performance. Halving planner availability (occupancy .97 → .47, Table 3) lowers the three-task mean from .29 [.20, .40] to .16 [.12, .19], while executing every plan at full availability raises it to .53 [.35, .71]. The gains come from planning during data collection. The acceptance rule sets how much the agent plans with one unitless threshold for every task and protects what the actor already does well: during sustained planning it keeps 98.3% of the actor's successes (95.8% when every plan is executed) and cuts the return loss from 16.7 to 3.6 (Figure 3b). Section 5.4 now explains how κ sets this planning budget at no extra training cost.

Four-way comparison (Q3). Table R1 reports the configurations you asked for on the three precision tasks, at matched planner availability; Table 3 in the paper reports the same comparison with occupancy and the share of executed plans.

Table R1.

Configurationhand-insertassemblydisassembleacting steps executing a plan
R2-Dreamer (no planning).13 [.02, .27].00 [.00, .00].12 [.00, .24].00
Planning, every plan executed.49 [.31, .67].75 [.37, .97].35 [.00, .75].97
SDI: planning + acceptance rule (Table 1).52 [.40, .63].24 [.00, .49].12 [.00, .30].43
Planning + bonus, every plan executed.35 [.06, .64].25 [.00, .73].25 [.00, .74].97
SDI + exploration bonus.40 [.24, .56].02 [.00, .06].27 [.00, .56].40
SDI + bonus, random acceptance at the rule's rate.61 [.46, .77].00 [.00, .00].00 [.00, .00].43
SDI + bonus with raw sibling spread.10 [.01, .24].40 [.00, .89].00 [.00, .00].44
SDI + bonus, constant-spread threshold (slow running average).55 [.45, .65].00 [.00, .00].00 [.00, .00].25

Success at the full budget: mean with its 95% bootstrap interval, computed as in Table 1 (resampling runs, then the evaluations that form each run's score). Submission runs are pooled with discussion-phase seeds of identical configurations.

Every configuration that plans exceeds R2-Dreamer's three-task mean, and the exploration bonus adds no consistent benefit at this schedule, so Table 1 now evaluates SDI without it.

We hope these results address your concerns, and we would be grateful if you would consider raising your score.

Thank you again for your thoughtful review. If anything remains unclear, or if a specific additional check would help your assessment, please let us know and we will try to address it before the discussion deadline. We would be grateful for any further feedback, and for an updated assessment if your concerns are resolved.

Comments on the paper

2 comments

Thank you all for the careful and constructive reviews. We are glad that the reviewers found the shared actor reference concrete and well motivated (ebEg, C6Jq), valued that it unifies exploration and plan acceptance without an extra ensemble (C6Jq, ebEg), appreciated the actor-only evaluation that measures true policy improvement (C6Jq), and found the results on the harder tasks impressive (LcVN). The requested analyses made the paper substantially stronger. A separate comment lists every change in the revised PDF; the main additions are:

  • Uncertainty and replication (LcVN, ebEg). Every entry of Table 1 now carries a 95% bootstrap interval (resampling runs, then the evaluations that form each run's score); Section 5.1 gives intervals for the suite means; Tables A3–A4 list every run's endpoint. We added replication runs so that every SDI entry and the strongest baseline on every task rest on multiple independent runs.
  • Controlled ablations (LcVN, ebEg). Table 3 compares planning alone, planning with the acceptance rule, the exploration bonus, random acceptance at the rule's own rate and a constant-spread threshold, all at matched planner availability.
  • Compute (C6Jq). Table 2 measures training time, planner latency and GPU memory against R2-Dreamer and TD-MPC2 on an A100.
  • Robustness (C6Jq). Table 4 sweeps κ and tests sticky actions and non-stationary Gilbert–Elliott noise.
  • What the actor reference provides. A redesigned Figure 3 shows one threshold across reward scales, protection of the actor's successes, and gains that follow planning exposure.

Main results. SDI has the highest average score in every suite and is competitive with or exceeds every baseline on 16 of 18 tasks (the 95% interval of its difference from the strongest baseline includes zero or lies above it). Against R2-Dreamer, the strongest baseline in each suite: Meta-World .69 [.65, .73] vs .60 [.53, .67], a difference that holds across runs; Adroit .48 [.42, .53] vs .45 [.40, .50]; DM-Control 545 [471, 615] vs 530 [462, 597]. On the three precision tasks, SDI raises the three-task mean from .08 [.03, .14] to .29 [.20, .40], 3.6 times the base agent, and the ablations show that this gain comes from planning during data collection. The actor reference gives that planning one unitless threshold: κ = 2 serves all 18 tasks without tuning, and during sustained planning the rule keeps 98.3% of the actor's successes (95.8% when every plan is executed) and cuts the return loss relative to the actor from 16.7 to 3.6.

The author-reviewer discussion period will end soon. Please review and acknowledge the authors’ responses. If your concerns have been addressed, please reconsider your score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.