- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
In submission 35058, the authors address two main challenges in world-model-based reinforcement learning. These are: a ) how to explore effectively in latent imagination, and b) how to decide when a plan at decision time should take priority over a learned policy (actor). Their approach, called Sibling Disagreement in Imagination (SDI), brings together exploration and plan acceptance using a shared actor-based reference created at the current latent state .
As claimed, SDI links intrinsic exploration and decision-time planning through a unified reference mechanism derived from repeated rollouts of the current actor sequence. This approach eliminates the computational and architectural complexity of training secondary ensemble world models or explicit epistemic-uncertainty heads.
Critical strengths
a) Unified Conceptual Framework without Ensemble Overhead: The proposed SDI cleanly connects intrinsic exploration and decision-time planning through a single, shared reference mechanism (s_\pi(z)) derived from repeated rollouts of the current actor sequence.
b) Evaluation Protocol: SDI uses gated planning solely during training data collection. Evaluation is strictly actor-only without search or gating, ensuring that benchmark gains reflect true policy improvement rather than test-time search compute.
c) Benchmark-Leading Performance: Notwithstanding my comments below, the paper evaluates performance across Meta-World v3, Adroit, and DM-Control (18 pixel-based continuous control tasks). The proposed SDI achieves the highest observed suite-average score in all three domains and matches or exceeds the strongest baseline (including DreamerV3, R2-Dreamer, TD-MPC2, and Plan2Explore) on 15 of 18 tasks.
Critical weaknesses
The paper lacks a quantitative analysis of training time, GPU memory utilization, or latency per environment step relative to actor-only execution (e.g., DreamerV3 / R2-Dreamer) or standard ungated MPPI.
The benchmarks used (Meta-World, Adroit, DM-Control) are standard continuous visual control tasks. However, the paper does not test SDI on environments with multimodal action dynamics, highly non-stationary or stochastic settings (where s_\pi could become very large and stop planning), or on long-horizon tasks with sparse rewards like Crafter or Atari. The authors may wish to consider alternatives to these suggestions, such as Atari tasks.
Actor Mode vs. Stochastic Action: During re-scoring, G_\pi and s_\pi are calculated using the actor's deterministic mode sequence. However, if a plan is rejected, the agent samples a stochastic fallback action a_t \sim \pi(\cdot|z_t) for environment execution. The manuscript should clarify whether evaluating the mode during gating is consistent with the stochastic policy's actual training-time behaviour.
Questions for authors
Please consider the following:
Computational Overhead: What is the training time overhead of SDI compared to the base R2-Dreamer agent and TD-MPC2 on a standard GPU (e.g., NVIDIA A100)? A comparative evaluation against a baseline is essential.
Actor Mode vs. Stochastic Policy Mismatch: During plan acceptance evaluation, G_\pi and s_\pi are measured using repeated rollouts of the actor's mode sequence. Why use the mode rather than sample from the stochastic policy \pi(\cdot|z_t), given that we use stochastic sampling for fallback execution when a plan is rejected?
Sensitivity to Threshold \kappa: How sensitive are the benchmark results (e.g., Meta-World precision tasks) to variations in \kappa \in {0.5, 1.0, 2.0, 3.0, 4.0? Is \kappa=2.0 robust across different reward scales without task-specific tuning? Again, a comprehensive, systematic evaluation of the parameter(s) and setting(s) is required for submission.
Behaviour in Highly Stochastic Dynamics: How does SDI behave when environment dynamics are inherently noisy or stochastic, or changebruptly, i.e. t, whenhe background noisy environment shifts from a benign Gaussian to a Gilbert-Elliot model? Does high stochasticity inflate s_\pi(z), causing the gate to reject all plans and fall back entirely to the actor?
Thank you for the positive assessment, for recognizing the unified framework without ensemble overhead, the actor-only evaluation protocol and the benchmark-leading performance, and for the concrete questions, which we address in turn.
Computational overhead (W1, Q1). Table 2 (new, main text) measures every component on one A100-80GB (hand-insert, 16 parallel environments):
Actor mode versus a sampled action (W3, Q2). The reference s_π should measure how much imagined return varies when behavior is held fixed. Drawing a fresh actor sample for each repeat would add the policy's own sampling variance, so the threshold would follow the actor's entropy, which changes over training, rather than the model's predictions. The mode is also the action executed at evaluation, and executing a fresh actor sample when a plan is rejected keeps those steps identical to the base agent's training behavior. Section 3.3 now explains this.
Sensitivity to κ and reward scale (Q3). κ = 2 was fixed in advance and used on all 18 tasks without task-specific tuning. Rescaling the reward rescales both the predicted advantage ĝ and κ·s_π, so every acceptance decision is invariant to the reward scale; a raw-margin threshold lacks this property (Figure 3a: one raw threshold executes between 16% and 79% of proposals on tasks whose margins differ by about 20×). The training-level sweep you suggested (Table 4a, hand-insert):
Success at every κ ∈ {0.5, 1, 3, 4} (.33, .53, .49, .47) lies inside the 95% interval of the default (.40 [.24, .56]), while κ sets how many plans are executed (.83 at κ = 0.5 down to .27–.41 at κ ≥ 2), so the results are not sensitive to κ in this range.
Stochastic and non-stationary dynamics (W2, Q4). Action-independent noise inflates the sibling spread and s_π alike, so the bonus ratio discounts it by construction. We tested sticky actions (p = .25) and Gilbert–Elliott action noise whose σ switches between .05 and .5 (Table 4b):
The rule does not stop planning: SDI still executes 32–43% of plans on lever-pull and 36–52% on hand-insert while s_π grows with the noise; both agents solve lever-pull under both noise models, and on hand-insert the two agents' 95% intervals overlap under both conditions.
Other domains (W2). Atari and Crafter have discrete actions, which need a different proposal distribution than MPPI's Gaussian, and strongly multimodal action spaces would need a richer proposal; the revised Limitations (Appendix A) state this scope. A preliminary Crafter run of the bonus alone on DreamerV3 (165k steps) scored below DreamerV3 (3.1 vs 4.5), so we make no claim there and leave discrete-action proposals to future work.
We hope these measurements and explanations address your concerns, and we would be grateful if you would consider raising your score.