acceptodds
Under review as a conference paper at ICLR 2027

Disagreement in Imagination: A Cross-Timescale Consistency Signal for Robust Model-Based Control

Abstract

Planning inside a learned world model is sample-efficient but fragile: prediction errors compound with rollout depth, and execution-time disturbances cause plans to drift from reality. Every standard detector of model error such as filtering residuals, reconstruction losses requires an observation to compare against, and inside imagination no observation exists. We study a quantity that does not need one —the disagreement between two predictors of the same imagined step, one coarse and trained over abstract steps, one fine and trained over primitive steps. This cross-timescale disagreement is computable from predictions alone, needs no future observation, and is directional: it states how the fine predictor would revise the coarse one. It is consumed at three clocks: as a consistency penalty during planning, as a training regularizer that sharpens the abstract model, and, once the imagined step is realized, as the residual an online controller cancels. On 80 tasks across DMControl, Meta-World, and MyoSuite under combined observation and action noise, HiWM achieves the highest IQM among evaluated methods with a margin that grows monotonically with noise intensity. Ablations on a completed factorial confirm super-additive coupling between the feedback pathway and the controller. Freezing the fast predictor at its initial observation as the configuration closest to a standard posterior, loses half the benefit, and two prediction rates without upward flow perform worse than a single rate. The mechanism transfers zero-shot to a Franka Panda robot, where disagreement is 2.4 larger than in simulation and the controller's corrections scale proportionally.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, world models, imagination-based planning, prediction disagreement, continuous control

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_ELTPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This study proposes HiWM that compares a coarse and a fine world model to qualify unreliable plans. HiWM uses the disagreement between these models during imagined rollouts to guide training and planning. The coarse model predicts abstract states over action segments, while the fine model steps over primitive actions conditioned on the abstract state. An additional controller corrects action errors. Results show improved noise robustness across standard tasks (DMC, Meta-World, MyoSuite) and a Franka Panda robot.

Critical strengths
  1. I find the core idea of comparing predictions across temporal resolutions to flag unreliable parts of imagined rollouts without future observations or an ensemble, simple and useful. HiWM only needs the disagreement between the two models as the consistency signal.

  2. The empircal evaluation is broad. The study reports results across 80 tasks with uncertainty estimates, clean controls and ablations and a physical robot test.

Critical weaknesses
  1. Important parts of the proposed method are unclear. The planner and controller use different error quantites and their identification as the same functional is not clear. The former uses an abstract KL divergence while the latter uses L2 error. Eq. 3 as written appears to fit observed deviations rather than cancel them. Also, its not clear how the belief net qq is trained as it appears to be stop-gradiented everywhere.

  2. Super-additive coupling is not substantiated. In Table 2, in fact the claim appears reversed. The combined gain is smaller than the sum of the individual ones, and we see negative interaction on all four tasks.

Questions for authors
  1. Please specify how the belief qq is trained and how the controller update corrects execution errors.

  2. How do you reconcile the super-additivity claim?

Reviewer_BwRPOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes a method of detecting model error in model-based RL without observing the true consequence of proposed actions. It works by comparing the predictions of two models, one trained on a coarse time scale and another trained on a fine time scale. The difference between the two is used as a regularizer to train the abstract model, as a penalty for planning, and as a residual for an online corrective controller. The paper evaluated performance on DMC, Meta-World and MyoSuite in comparison to various baselines, and evaluated the model trained in MuJoCo zero-shot on a real Franka Panda robot.

Critical strengths
  • The paper evaluated performance on a real robot (Franka Panda)
  • The paper evaluated performance on both directly observable states and only image observations
Critical weaknesses
  • Incorrect positioning of the paper

    • The abstract and the introduction makes it seem as if the paper is the first to not require access to observations to detect model errors. However, various prior methods do not require access to observations. The paper itself mentions ensemble-based approaches from 2018 and 2020 in the related work section, which do not require access to observations either.
  • Overclaim of novelty

    • The paper claims that prior methods for detecting model errors that don't require access to observations use "read-only diagnostics" (L107); however, more sophisticated approaches do in fact train models in a way that allows for detection of model errors. For example, see Aghabozorgi et al., "WIMLE: Uncertainty-aware world models with IMLE for sample-efficient continuous control", ICLR 2026.
  • Lack of comparison to the state-of-the-art

    • The methods the paper compares to are somewhat old, e.g., DC-MPC from 2024, TD-MPC2 from 2023, TD-MPC from 2022, DreamerV3 from 2025 and Director from 2022. The paper did not compare to the latest methods, such as WIMLE from ICLR 2026.
    • The results seem to be worse than those of WIMLE. For example, on DMC, the proposed method achieves 0.84 compared to WIMLE's 0.871 (Table 2 on pg. 22), and on MyoSuite, the proposed method achieves 0.72 compared to WIMLE's 0.980.
    • Moreover, the paper doesn't report the number of iterations used to achieve the reported result, which is important since sample efficiency is one of the most important metrics for model-based RL.
  • The exposition of the method is confusing, incomplete and inconsistent with the description of the method in the abstract and the introduction. As a result, it's unclear how the method works.

    • It's unclear what family of distributions q(zs∣d^t+hs,z^t+hf)q(z^s | \hat{d}_{t+h}^{s}, \hat{z}^{f}_{t+h} ) and p(zs∣d^t+hs,z^t+hf)p(z^s | \hat{d}_{t+h}^{s}, \hat{z}_{t+h}^{f}) belong to, and what their parameters are. Furthermore, the difference between qq and pp isn't explained. In particular, how would p(zs∣d^t+hs,z^t+hf)p(z^s \vert \hat{d}_{t+h}^{s}, \hat{z}_{t+h}^{f}) differ from q(zs∣d^t+hs,z^t+hf)q(z^s \vert \hat{d}_{t+h}^{s}, \hat{z}_{t+h}^{f})?
    • How is At+h−1A_{t+h-1} different from a^t+h−1\hat{a}_{t+h-1}?
    • Since there is a stop gradient on q(zs∣d^t+hs,z^t+hf)q(z^s | \hat{d}_{t+h}^{s}, \hat{z}^{f}_{t+h} ), how will it be learned?
    • Why does GhG_h not depend on tt?
    • What's the policy, i.e., how do you sample actions given states?
  • The phrasing is weird and unclear.

    • L21: "It is consumed at three clocks" - it's unclear what "consumed" and "clocks" mean in this context
    • L68: "The plan-time penalty and the execution-time controller are approximately additive on their own" - it's unclear what "approximately additive on their own" means in this context
    • L79: "reported as a complete factorial over the signal’s consumers" - it's unclear what "factorial" and "consumers" mean in this context
  • Various mathematical notations are left undefined or defined much later than they are first used, so it was a struggle to try to understand what they mean.

    • In eq. 4, LdynL_{\mathrm{dyn}}, LrewL_{\mathrm{rew}} and LtermL_{\mathrm{term}} are not defined. While I assume LdynL_{\mathrm{dyn}} may be the same as the loss term used DreamerV3, I am not sure what the other two terms do.
Questions for authors
  • How many environments steps were used to achieve the reported results? How does the number of environment steps compare to the baselines?
  • What are the definitions of LrewL_{\mathrm{rew}} and LtermL_{\mathrm{term}}?
Reviewer_pS1eOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes HiWM, a hierarchical world-model-based control framework designed to improve robustness under observation and action noise. The central idea is to measure cross-timescale disagreement between a coarse abstract predictor and a fine-grained predictor without requiring future observations. This disagreement is incorporated into model-based planning as a consistency penalty and into model training as a regularizer. An online corrective controller is further introduced to compensate for execution errors. The authors report improvements over several model-based reinforcement learning baselines across DMControl, Meta-World, MyoSuite, and Franka Kitchen, together with real-world experiments on a Franka Panda robot. The paper also presents component ablations, prediction-error analyses, and robustness evaluations under dynamics shifts. The research question is important, and using cross-timescale consistency as an internal reliability signal is a potentially valuable direction. However, I have serious concerns about the interpretation of the paper's central mathematical results, the validity of its claimed unified mechanism, and the consistency of its experimental reporting. Several headline claims are not adequately supported by the presented evidence.

Critical strengths

Important problem and a potentially useful methodological perspective. Evaluating the reliability of imagined trajectories without access to future observations is an important challenge in model-based reinforcement learning. The proposed use of disagreement between predictors operating at different temporal resolutions offers an interesting way to obtain internal consistency information during planning. The potentially novel contribution lies in using cross-timescale disagreement as a training target and planning criterion, rather than merely introducing another uncertainty estimate. Broad experimental scope and several targeted analyses. The paper reports results across diverse continuous-control environments, including high-dimensional locomotion, robotic manipulation, compositional tasks, and physical hardware. Several experiments are relevant to the proposed mechanism. The stale-evidence ablation (Table 4), isolated noise-source analysis (Table 9), and long-horizon prediction-error analysis (Figure 2 and Appendix L) attempt to distinguish the contributions of different components and establish the conditions under which the method is useful.

Critical weaknesses

The central claim of super-additive synergy is contradicted by the conventional factorial interaction effect. This is my most serious concern. In Section 4.3 and Table 2, the authors claim that the feedback pathway and corrective controller exhibit positive super-additive coupling. However, the reported results correspond to negative interaction effects under the standard factorial definition. For Dog Run, Table 2 reports: Full model: 0.84 No controller: 0.78 No feedback: 0.68 Neither component: 0.56 Under the standard factorial definition, the interaction effect is obtained by taking the full-model performance, subtracting both single-component performances, and then adding the performance without either component. For Dog Run, this gives negative 0.06 (0.84 minus 0.78 minus 0.68 plus 0.56). Equivalently, introducing the feedback pathway alone improves performance by 0.22, while introducing the controller alone improves performance by 0.12. Their combined improvement is only 0.28, smaller than the sum of their individual improvements (0.34). The same calculation gives negative interaction effects for all four tasks: Dog Run: −0.06 Humanoid Walk: −0.06 Assembly: −0.02 Hand Pose: −0.04 The paper instead defines its synergy index in terms of the performance drop when both components are removed, minus the drops from removing each component separately, normalized by the full-model score. By substitution, this quantity is exactly the negative of the conventional factorial interaction, divided by the full-model score. Therefore, a positive value of the paper's synergy index is mathematically equivalent to a negative conventional factorial interaction. This is not an arithmetic error in the proposed index, but a fundamental problem with its interpretation as positive super-additive performance gains. The results are consistent with partially compensatory effects: when one mechanism is removed, the other can still provide some of the missing functionality. Indeed, Appendix C.4 states that corrections from one mechanism reduce the burden on the other, which is consistent with overlapping or partially substitutable contributions. Furthermore, the increasing positive S values in Table 11 correspond to increasingly negative normalized interaction effects under the conventional definition. I acknowledge that the feedback factor also bundles the upward pathway with its associated planning penalty, rather than isolating a single independent operation. However, this does not resolve the sign inconsistency in the reported factorial comparison. Because super-additive coupling is repeatedly highlighted in the abstract, introduction, and conclusion as a central contribution, this issue directly undermines one of the paper's principal scientific claims. The claimed single-signal coupling across planning, training, and execution is not established by the mathematical formulation. The paper repeatedly claims that one cross-timescale disagreement signal operates at three stages: planning, training, and execution. In Section 3.3, the authors explicitly state:The same functional, evaluated on a transition that actually happened, is an execution residual. However, the equations do not establish this equivalence. During planning and training, the proposed disagreement is the KL divergence from the fast-model-conditioned abstract belief (q) to the abstract prior (p). This is a distribution-level quantity defined in the abstract latent space. In contrast, the corrective controller receives latent-state prediction discrepancies and is optimized using an L2 prediction-error objective. These quantities differ in both representation space and mathematical definition. The KL disagreement is computed between distributions in the abstract latent space, whereas the execution residual involves differences between fast latent-state representations. Moreover, while the belief distribution q can provide a direction for updating the abstract prior, the scalar KL divergence alone does not directly specify an action correction. The paper does not provide a mathematical derivation connecting these quantities, nor does it demonstrate experimentally that the controller directly consumes G or directional information derived from q and p. Consequently, the existing results may support a useful combination of consistency-regularized planning and residual control. They do not establish the stronger claim that a single directional signal directly drives all three stages. This distinction is central to assessing the paper's novelty. Without a rigorous connection, the proposed framework appears closer to combining two related but distinct error-handling mechanisms than to the claimed unified single-signal architecture. Multiple inconsistencies in experimental coverage and quantitative reporting undermine confidence in the evaluation. Cross-checking the main paper and appendices reveals several concrete discrepancies. (a) Incomplete DMControl task reporting. Appendix M, Table 27, claims to report results for all 30 DMControl tasks and supports the statement that HiWM achieves the best performance on 26 out of 30 tasks. However, the table contains only 24 tasks: 5 easy, 9 medium, and 10 hard. Furthermore, HiWM achieves the highest reported score on every one of these 24 tasks. The paper also states that the four exceptions are characterized by low-dimensional observations and short effective planning horizons, while all explicitly reported low-dimensional tasks favor HiWM. The six missing tasks and the identities of the four exceptions must be provided to reconcile these statements and verify the claimed 26/30 result. (b) Inconsistent baseline evaluation coverage. Appendix B.3 states that Puppeteer and HWM are evaluated on 12 representative tasks with three seeds, whereas Director is evaluated on 80 tasks with five seeds. Nevertheless, Table 1 presents aggregate benchmark comparisons across these methods without adequately clarifying whether the results are calculated over identical task sets. The appendix also generally describes evaluation using five seeds, which is inconsistent with the stated three-seed protocol for these hierarchical baselines. Without a clearly matched evaluation protocol, the aggregate IQM comparisons and conclusions about hierarchical baselines are difficult to assess fairly. (c) Contradictory computational-cost reporting. Appendix P, Table 30, reports HiWM training time as 8–10 days. However, Appendix Q states that simulation experiments require 8–10 hours per task on a single A100 GPU. These figures differ by approximately a factor of 24, making the reported computational cost unclear. (d) Inconsistent real-robot trial counts. Appendix J specifies 20 trials per method per task across three tasks, with two policy seeds alternated across trials. Given the three methods reported in the main hardware comparison, the stated protocol accounts for 180 trials in total (three tasks, three methods, and 20 trials per method per task). However, Appendix Q reports approximately 360 real-robot trials without explaining the additional experiments. The use of two alternating policy seeds does not itself double the stated number of trials. (e) Incorrect quantitative statement in Section 4.3. The text claims that the no-feedback configuration performs worse than TD-MPC2 on three of four representative tasks. However, Table 2 shows worse performance on only two of the four tasks when compared with the listed TD-MPC2 reference. Individually, some of these discrepancies could be reporting or transcription errors. Nevertheless, their combined presence makes it difficult to verify several prominent experimental claims, particularly given the emphasis placed on evaluation breadth and robustness. The algorithm specification leaves critical aspects of execution and belief learning unresolved. Algorithm 1 contains ambiguities that directly affect reproducibility and the interpretation of the proposed mechanism. (a) The source of the controller's prediction residual is unclear. The controller in Section 3.3 is defined using the current fast latent state, a predicted fast latent state, and the reference action. However, Algorithm 1 does not clearly explain how the predicted state used for online correction is preserved and updated across control steps. Line 13 initializes a predicted fast state within the planning rollout, while line 28 uses the current and predicted fast states to compute the corrective action. The paper does not explicitly distinguish the planning-time initialization from a prediction carried over from the previous execution step. It is therefore unclear how the controller obtains the intended execution-time prediction discrepancy during inference. This ambiguity does not necessarily imply that the implementation is incorrect, but the required state propagation is missing from the algorithmic specification. (b) The next abstract state is referenced before it is available.** Line 27 computes the next fast-state prediction using the next abstract state, although the corresponding environment observation is obtained only after action execution, and the next observed abstract state is encoded in line 35. If line 27 is intended to use a predicted abstract state rather than an observation-derived state, the notation and execution sequence must be corrected. As written, the algorithm does not clearly specify a causally executable procedure. (c) The learning mechanism for the fast-conditioned belief q is insufficiently specified.** The disagreement regularizer explicitly applies a stop-gradient operation to the fast-conditioned belief (q) before computing the KL divergence against the abstract prior (p). Thus, this regularizer does not directly propagate gradients through q. However, Equation (4) and Algorithm 1 do not clearly specify an independent learning objective or parameter-sharing mechanism that trains the fast-conditioned belief distribution. Appendix A.4 introduces cross-level alignment modules, but their precise relationship to the parameterization and optimization of q is not established. This is particularly consequential because q serves as the central training target for the abstract prior. Its ability to provide informative predictions cannot simply be assumed. The authors should explicitly specify how q is constructed, which parameters it uses, and through which loss terms those parameters are updated. These issues concern the functionality of the central mechanism rather than minor implementation details. The disagreement signal is insufficiently validated as a general reliability measure, and important performance contributions remain entangled. (a) Cross-timescale agreement does not necessarily imply prediction accuracy.** Section 4.4 and Appendix F report correlations between imagined disagreement and disagreement recomputed using realized latent states. These results are useful, but realized disagreement is still a model-relative quantity. It is not equivalent to an independently measured error between a predicted trajectory and the actual environment dynamics. Two predictors can agree while both making incorrect predictions. This common-mode failure is particularly relevant here because Equation (1) conditions the fine predictor on the abstract prediction. The two predictors are therefore structurally dependent rather than independent estimators. Their errors may be correlated, allowing disagreement to remain low even when both predictions are inaccurate. The paper does not adequately investigate this failure mode or establish the calibration of G against actual future prediction errors. Consequently, the claim that cross-timescale disagreement acts as a general model-mismatch detector is stronger than what the presented analyses establish. (b) The exploration bonus is not independently isolated. Appendix A.5 introduces an exploration objective that adds a positive bonus proportional to disagreement. This constitutes an additional use of the disagreement signal beyond the three mechanisms emphasized in the main paper. Because disagreement-guided exploration can affect the data distribution and subsequent model quality, it may contribute to the observed performance improvements. However, the main credit-assignment experiments do not clearly isolate the contribution of this exploration bonus. The use of positive disagreement during exploration and negative disagreement during planning is not inherently contradictory, since these objectives serve different purposes. Nevertheless, the contribution of exploration should be separated from the proposed planning-training-execution coupling. (c) Additional simple controls would strengthen the causal attribution. The evaluation includes a flat backbone with an identical corrective controller, which is a useful comparison. However, simple alternatives such as observation filtering, noise-augmented training, or online bias estimation would help determine how much of the robustness improvement requires the proposed cross-timescale mechanism. I do not regard the absence of every possible baseline as a decisive weakness. The more important issue is that several mechanisms remain entangled while the paper makes strong claims about their individual and synergistic contributions.

Questions for authors

Interpretation of super-additive coupling The reported synergy index is algebraically equivalent to the negative of the conventional factorial interaction effect after normalization. All four tasks in Table 2 have negative conventional interactions. What precise definition of super-additivity is intended? Why should these results be interpreted as super-additive performance gains rather than sub-additive gains or partially compensatory component effects? Please also clarify whether the conclusions drawn from the increasing synergy indices in Table 11 remain valid under the conventional interaction interpretation. Mathematical relationship between the claimed shared signals What is the exact mathematical relationship between the KL-based disagreement G used in planning and training and the fast latent-state residual used by the corrective controller? Does the controller explicitly consume G, q, p, or their directional information? If not, what establishes the claim that the same functional operates across all three stages? Training of the fast-conditioned belief How is q parameterized and trained? Given the stop-gradient applied to q in the KL regularizer, which other objective provides the learning signal for its parameters? Is q constructed from the alignment head introduced in Appendix A.4, or does it have an independent parameterization? Algorithm execution and state propagation How is the predicted fast latent state used by the controller maintained across execution steps? Why does Algorithm 1, line 27, reference the next abstract state before the corresponding observation is available? Please provide an unambiguous, causally executable description of the controller update and inference procedure. Completeness and comparability of experimental results Why does Table 27 report only 24 of the claimed 30 DMControl tasks? Which four tasks constitute the reported exceptions to HiWM's best-performance count? Were all aggregate baseline comparisons in Table 1 computed on identical task sets? How were the 12-task, three-seed evaluations for Puppeteer and HWM incorporated? Computational cost and hardware trial counts Is the correct training time 8–10 days or 8–10 hours per task? How does the approximately 360 hardware-trial count in Appendix Q reconcile with the 180 trials described by the main hardware protocol? Independent contribution of exploration What is the performance of HiWM when the disagreement-based exploration bonus is removed while all other components remain unchanged? How much of the reported robustness and sample-efficiency improvement can be attributed specifically to the proposed planning and training mechanisms? Reliability under correlated model errors How does G behave when the coarse and fine predictors make similar incorrect predictions? Can the authors provide a direct analysis of the relationship between G and ground-truth future prediction error, rather than disagreement recomputed from realized latent states? Overall Assessment The paper addresses an important research problem and presents an interesting approach to measuring internal consistency across temporal scales in learned world models. The reported empirical improvements are potentially meaningful, and several ablations provide useful evidence that multi-rate prediction and online correction can improve robustness. However, the current submission contains fundamental problems that directly affect its headline scientific claims. Most importantly, the claimed positive super-additive coupling is based on an index whose sign is opposite to the conventional factorial interaction effect. The reported results therefore support sub-additive gains under the standard interpretation, contrary to the paper's central claim of super-additive performance improvements. In addition, the proposed single-signal mechanism is not established by the equations: the KL disagreement used in planning and training is distinct from the latent prediction residual used for execution correction. The learning pathway for the fast-conditioned belief and the execution-time state propagation are also insufficiently specified. These issues are compounded by multiple inconsistencies in experimental reporting, including missing per-task results, potentially unmatched baseline comparisons, contradictory training-time figures, and unexplained hardware trial counts. I acknowledge that some reporting discrepancies and algorithmic ambiguities may be correctable. Nevertheless, the concerns are not limited to presentation. They affect the interpretation of the central mechanism, the attribution of reported performance gains, and the ability to verify the empirical evidence. The work may contain a valuable underlying idea, but the current manuscript does not establish its main methodological claims with sufficient rigor and internal consistency. I recommend clear rejection because the deficiencies affect both the central theoretical interpretation and the reliability of the empirical validation. Addressing them would require a substantial clarification and re-evaluation of the paper's principal contributions, rather than minor revisions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.