acceptodds
Under review as a conference paper at ICLR 2027

Lucid Dreaming for World Models: Learning to Doubt Imagination and Decide by Trust

Abstract

World models enable agents to learn and plan in imagination, but predictions beyond their experience can become unreliable and mislead decisions. Existing uncertainty estimates derived from predictions can remain overconfident on unfamiliar state-action pairs. We propose the Lucid World Model (LucidWM), which learns doubt from experience and propagates trust through imagination. By integrating Subjective Logic into categorical latent transitions, LucidWM distinguishes predicted outcomes from their evidential support and assigns each transition a degree of doubt. The complement of this doubt defines transition-level trust, which accumulates multiplicatively along imagined trajectories to reweight returns for policy learning and guide action selection. Uncertainty estimation requires no additional parameters or forward passes. Evaluated on four base world models against seventeen uncertainty readouts, LucidWM detects environmental changes and signals uncertainty during action-corrupted rollouts. In a controlled navigation case study, acting on trust reduces the number of steps required to reach the goal from 362 to 190. Fifteen demonstration videos show how LucidWM doubts its dreams and acts on that doubt. Code and videos are available at https://anonymous.4open.science/r/LucidWM.

Primary area:
reinforcement learning
Keywords:
world models, model-based reinforcement learning, uncertainty estimation, evidential learning, subjective logic

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_K6Y7OpenReview ↗
Rating
4(+2)
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a new way to train world models, rooted in Subjective Logic, which gives the model inherent, principled means to reason about epistemic uncertainty.

Critical strengths
  1. The idea seems sound (my familiarity with Subjective Logic is unfortunately limited).

  2. The significance of the achievement, assuming the results stand (see weaknesses section), is tremendous.

  3. The presentation is for the most part very clear and serves the contributions very well.

Critical weaknesses

I'll preface this section with two statements:

First, in my opinion both the paper and its contributions are excellent and should be published, and I would like us together throughout this rebuttal to bring the paper to the point where I can easily give a score of 4.

Second, my familiarly with Subjective Logic, which underpins the entire theoretical support for this work, is unfortunately limited, which influences my review and I think should influence the weight of my related comments (and more importantly, the lack of comments about the soundness of the underlying approach).

Below are the issues I think we need to address before we can increase the score:

Evaluation and statistical significance:

  1. The main metric used to evaluate the contributions is AUROC. I've never heard of it, and it is not explained or even cited. Please make sure the metric used is sufficiently well explained in the body of the paper to understand the results, and that the choice of metric is motivated. This is one of the most major items: I am currently unable to evaluate the performance of the suggested method essentially at all.

  2. There is almost no metric of stat. signif. for any of the presented results. Please make sure that whenever relevant (everywhere, I think?) each evaluation is supported by a principled stat. signif. metric, such as a proper CI over the mean.

  3. Tables are the single worst way to present results. If (if!) the best metric to present is indeed a single score, please consider using bar charts instead, for example. If these results can be evaluated during training for example, consider plotting complete training curves.

  4. I would have liked to see RL experiments that show that this metric is sufficient to drive exploration in sparse-reward RL tasks. I think this is one of the most important proof-of-concepts for uncertainty estimation. I understand that this is an additional significant layer of complexity. Whether this is necessary or not will, to me, depend strongly on the re-presented results section and the explanation of the metric.

Theoretical motivation:

Due to my limited familiarity with SL, it's hard for me to distinguish to what extent the different loss terms etc. are directly theoretically motivated by SL, are approximations to quantities that are fully motivated, or are heuristics inspired by SL. I think that it's important that even a reader that is not fully familiar with the theoretical basis can know whether the approach is fully or partially supported by said theoretical basis. Please make sure that every for every contribution (I'm thinking eqs. 4-8, 10) it is made clear and explicit to what extent each it is supported by SL ("we approximate X with Y", "we track X with Y", "inspired by X, we use Y", etc).

Evidence in cont. spaces:

Despite limited familiarity with SL, the idea makes a lot of sense to me in discrete spaces. However, the underlying state space, and the action space - which is itself the same as the environment's I believe? - may very well be continuous. Reasoning about evidence in continuous spaces is.. nontrivial, because we (may) never see the same transition twice. This is never explicitly addressed. Please discusses this (first in the rebuttal - I would like to understand how this is addressed, and then the resulting discussion in the paper).

Additional minor items and questions:

  1. What is a "Self ensemble"?

  2. In the "Deciding what to do" section it's not clear to me whether a concrete method to decide what to do (presumably meaning, a way to select actions? during planning or in the environment?) is proposed, or whether it's just pointed out that J can be used (in different ways) to direct action selection. Please make it clear whichever of the two it is. It's better if it's the first, but I think the contribution remains sufficiently significant to publish even if not.

  3. There are some small issues (in my opinion) with notation and definition of terms:

    1. First, reward, transition, values etc are all discussed and used, but a proper MDP definition is never given. I think that giving a proper definition of an MDP and then the WM in light of it will both improve the presentation period, as well as help address the issues below.

    2. Second, the WM is defined as not-having a reward or value head (I don't like it, but I can live with it). But then all the followup quantities appear to be defined with respect to reward and value heads. Including the uncertainty, which is "for WMs", as I understand it. So which is it? Is there a reward and value, or not? Personally, I would recommend going the direction that states that in principle there are reward and value models, even if it is possible to ommit them and only learn latent transition dynamics. Once the MDP is fully defined, it will all become very natural to present coherently I think.

    3. I don't love the choice of symbols. sts_t is a latent state. Ok, sure (although - probably not great, because sts_t should be the true state of the MDP?). But then, what is ztz_t? It's a predicted latent state. It's still a latent state. Wouldn't it make more sense to call it s^t\hat s_t? Why are observations xtx_t and not for example oto_t? Are observations (and thus POMDPs with their additional layer of complexity) necessary? I would suggest the following: 1. Define an MDP or POMDP properly with all the standard symbols, which will make the presentation clearer. 2. If ztz_t is used for latent-states, than the encoder can output ztz_t and the transitions z^t\hat z_t (an attempt to learn what the encoder would predict). Alternatively, to use hth_t for the encoder's output and ztz_t for the dynamics output, and explain that in principle both try to predict the same latent state, but we use different letters to denote that they come out of different predictors.

  4. I think that section 3 should be called something more informative. Perhaps - "evidence grounded model learning," or something that actually describes the method. Often I would suggest using the title of the paper, but in this case the title is a bit marketing-oriented, so maybe something more informative would be better here.

  5. It's not clear to me that Figure 1 belongs before the abstract. In fact, the significance of the contribution, to me, became clear in the introduction rather than from the Figure.

  6. The presentation of the paper is very clear in my opinion, despite the entire paper's point being epistemic uncertainty estimation while the word "epistemic" doesn't even appear. This is not so much a weakness as much as its a genuine achievement, in my opinion. However, I would suggest at least referring to these terms (epistemic, aleatoric) and contrasting SL with the standard ideas / ways to quantify epistemic / aleatoric uncertainty. Could be part of a related work section.

Questions for authors

Please see the weaknesses section.

13 comments
AuthorsOpenReview ↗

Response to Reviewer K6Y7 (1/5)

Response to Reviewer K6Y7 (1/5)

Thank you for the careful and encouraging review, for your confidence in the contribution, and for offering to work with us toward a score of 4. We answer each point below in your order. Several of them were covered only in the appendix, where they were easy to miss, and the revision brings the key ones forward to where they first matter. In the revised PDF, every change to the text is marked in red and every new table or figure has a red caption; a revision record at the very end of the paper (Appendix G) lists the changes by review point and then sets the submitted and the revised wording of each changed passage side by side, so that you can compare them directly. For a direct look at what the method does, the anonymous repository linked in the abstract, unchanged since the submission deadline, shows the doubt rising before a fall and inside a changed room, next to the readouts we compare against.

T1. Evaluation 1, AUROC.

Thank you for raising this point. You are right that the submitted version only named AUROC, with its chance level in the caption of Table 1, and neither defined nor cited it. Following your suggestion, Section 4.2 now defines and cites AUROC where the text first uses it (Fawcett, 2006), and Appendix E.4 describes it in detail. AUROC, the area under the receiver operating characteristic curve, is the probability that a random positive example scores higher than a random negative one, so 0.5 is chance and 1 is perfect separation. It is to detection what accuracy is to classification, and it is the standard score for detecting inputs a model has not seen (Hendrycks and Gimpel, 2017). In our test, positives are frames of the new room and negatives are frames of the familiar maze. Because it needs no threshold, AUROC compares readouts on different scales directly. On DreamerV3 in Table 1(A), for example, our doubt ranks a new-room frame above a familiar frame in 77 of 100 random pairs, while none of the eleven other readouts exceeds chance, the best only 48 of 100. The other three bases show the same picture: there the doubt scores 0.83, 0.96 and 0.82, the best readout on each (Table 1(A), Figure 23). A value below 0.5, shown in grey, means that a readout, read in the direction its authors define, judges the new room more familiar than the maze. This direction is fixed in advance for every readout, since an agent that had to act on a readout could not choose its direction after the fact. Test (B) of Table 1 uses a second score, the lift, which Section 4.2 now also spells out: a readout's value when every imagined action is random, divided by its value under the true actions, so that 1 means no response.

T2. Evaluation 2, statistical significance.

Thank you for this suggestion. In the submitted version, Figures 7 and 21(c) show 95% bootstrap intervals and Figure 20 repeats the analysis of Figure 7 over ten runs, but, as you note, Table 1 reported means only. Following your suggestion, every cell of Table 1 now reports the mean and standard deviation over five training seeds, and Appendix F.14 gives, for every row, the 95% interval of our margin over the strongest other readout. Ours has the higher mean in all 20 rows, and in 18 of them the 95% interval of its margin lies entirely above zero, so in these 18 rows the lead is significant at the 5% level. Part (A) compares separately trained models, so these intervals are Welch t-intervals, and part (B) computes all readouts on the same model and seed, so they are paired t-intervals. On the two exceptions, R2-Dreamer on Crafter and EMERALD on pendulum, the margin changes sign across seeds, and Appendix F.14 marks them as ties. Appendices F.2 and F.13 add Wilson intervals, the standard 95% intervals for percentages, to the fall and veto rates. In each of the five seeds, for example, the doubt warns before 69% to 79% of the falls and the entropy before 4% to 7%. The learning and decision results are now reported in the same way: Crafter over three training seeds (Table 10), and the veto over 60 paired walks with a 95% interval (Table 12).

AuthorsOpenReview ↗

Response to Reviewer K6Y7 (2/5)

Response to Reviewer K6Y7 (2/5)

T3. Evaluation 3, presentation.

Thank you for this suggestion. In the submitted version, Table 1 is the only table in the main text, every other result is shown as a figure, and two results are plotted over training. Figure 6 follows the prediction error and the doubt of one run across its checkpoints, with all six checkpoints and the entropy in Figure 18. Figure 17(b) tracks the AUROC of the maze test in Table 1(A) over 38 checkpoints of one DreamerV3 run, from 0.1M to 3.8M frames. The doubt stays above chance at every checkpoint, while the same readout on the unmodified base stays below chance. New in the revision, following your suggestion, Appendix F.14 shows Table 1 as a bar chart (Figure 23), with ours next to the strongest other readout on every row as five-seed means with 95% intervals. The caption of Table 1 now points to this figure. The standard deviation now printed under every mean shows how much each row varies across seeds. We have kept Table 1 in the main text, since it gives the exact value of every readout on every row, and added the bar chart in Appendix F.14.

T4. Evaluation 4, exploration.

Thank you for this suggestion. We agree that exploration is an important use of uncertainty, as in Plan2Explore with ensemble disagreement (Section 2). This paper uses the doubt the other way round. An exploration bonus steers the agent toward states the model is unsure about, whereas we use the doubt to discount what the model imagines in those states, both in what the agent learns from and in what it decides. Section 4.5 tests whether the steps that trust discounts are the wrong ones, Section 4.6 tests deciding, and Appendix F.12 tests learning. Table 1 also runs a standard check of the doubt itself, whether it flags what the model has not experienced (Hendrycks and Gimpel, 2017), and the doubt passes it on all four world models, in all 20 rows. In Crafter, an open world whose rewards come from unlocking achievements, LucidWM reaches an evaluation return of 8 after 0.41M steps on average over three training seeds, where its base needs 0.88M (Table 10). You kindly noted that whether these experiments are needed would depend on the re-presented results and the explanation of the metric; the revision now defines and cites AUROC and adds the spread across seeds and the 95% intervals of Appendix F.14 (T1 to T3). Whether the doubt also works as an exploration bonus in sparse-reward tasks is an interesting question that we would like to study in future work.

AuthorsOpenReview ↗

Response to Reviewer K6Y7 (3/5)

Response to Reviewer K6Y7 (3/5)

T5. Theoretical motivation.

Thank you for this request, which will help readers new to SL. In the submitted version, Section 3 names the source of each equation where it is introduced, and Table 3 in Appendix B sets Eqs. 4 to 7 and 10 beside their standard forms. Following your suggestion, Table 3 now has a row for Eq. 8 and a column that states how each equation relates to SL. Equation by equation:

  • Eq. 4, SL applied directly. It is the mean of an SL opinion with a uniform base rate (Eq. 3); our two choices are to read the logits as evidence through a softplus, as is common in evidential deep learning, and to floor the doubt at the standard head's ε\varepsilon, so that the head never trusts a transition more than the standard world-model head does.
  • Eq. 5, adapted from evidential learning, never more confident than the SL count at its optimum. It adapts the KL regulariser of evidential deep learning (Sensoy et al., 2018), pulling all evidence toward the vacuous opinion of SL, the opinion with no evidence. In an idealised model with one opinion per scene–action pair, Proposition 4 shows that at its optimum the doubt is never lower than the doubt of SL's count; the two nearly coincide when some outcome was never observed (0.24 against 0.20 in the example of Appendix C.4), and otherwise its doubt stays higher. Section 4 tests the shared network.
  • Eq. 6, SL applied directly. It is the cumulative fusion of SL (Jøsang, 2016), which adds the evidence of the two sources, and the observation evidence is read from the observation alone, so that the two sources stay apart as the operator assumes.
  • Eq. 7, our construction, compounding trust as SL trust discounting does. It lets the share of the λ\lambda-return follow the trust τ=(1−uˉ)/(1−ε)\tau=(1-\bar u)/(1-\varepsilon), rescaled so that τ=1\tau=1 recovers the λ\lambda-return. Reading each τk\tau_k as the probability that step kk is reliable and the rollout as a chain of referrals, Proposition 8 shows that the trust it compounds equals the transitive trust discounting of SL, in which trusts multiply along the chain.
  • Eq. 8, exact. It rewrites Eq. 7 with no approximation (Lemma 2).
  • Eq. 10, our construction for acting. It is Eq. 7 with λ=1\lambda=1 and the critic's share set to zero before the last step, a design choice that weights rewards by the same compounded trust.

T6. Evidence in continuous spaces.

Thank you for this question, which goes to the heart of how the evidence is counted. In the submitted version, Appendix C.4 touched on this only briefly, at its end, in the network case, where “the evidence of every pair is one function e(st,at)e(s_t,a_t) of shared parameters”, and contrasted it with Posterior Networks (Charpentier et al., 2020), which keep counts over continuous inputs with a learned density that our head does not need. Following your suggestion, Section 3.1 now answers the question in brief where the evidence discipline is introduced, and Appendix C.4 now gives the full explanation below under its own heading, “Continuous scenes and actions”. Yes, the actions are the environment's own, continuous in DeepMind Control and discrete in the maze and Crafter (Section 4.1). First, what is counted is discrete: the evidence is over the next latent state, GG categorical variables of KK classes each (G=K=32G=K=32 in DreamerV3), exactly as in the discrete case. Second, the pair (st,at)(s_t,a_t) does stay continuous, but no pair needs to be seen twice, because e(st,at)e(s_t,a_t) is the output of one network that all pairs share, not a count stored per pair. The dynamics loss holds the evidence up on the pairs the agent has experienced, since a sharp posterior cannot be fitted without it (Proposition 3), and the shared parameters carry it to similar states and actions, so a new pair receives the evidence of the experience around it. The discipline (Eq. 5) pulls the evidence down at every training pair, so the network defaults to low evidence and keeps it high only where the data hold it up; the experiments below show that away from the data the evidence falls and the doubt rises. With one entry per pair, as in a table, Propositions 3 and 4 prove this; for the network, the experiments show it. On walker, where states and actions are continuous, imagined rollouts with random actions end up twice as far from the training states as any training state is from its nearest neighbour, and by the end nine in ten have doubt above the 95th percentile measured on the training states (Figure 4). The same model trained without the discipline drifts as far and its doubt stays flat, as Section 4.3 now states. Table 1(B) replaces the actions by random ones on its three DeepMind Control tasks, and the doubt rises for all four world models.

Reviewer_FKUZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper tackles the problem of recognizing unreliable imaginations in world model. The authors do so by learning with Subjective Opinions within the world model training. The method was tested across multiple benchmarks, across multiple world model on how well it can predict uncertainties, with very limited study on how it benefits downstream control.

Critical strengths
  • Approach seems sound, by also predicting the belief based on the evidence it has
  • Tested on a large variety of baselines, across several world models and various simplified baselines
Critical weaknesses
  • (major) Experimental settings can be more realistic for the “before decision” result. Currently the setup is, at different times, random actions are given. However, this seems very unrealistic for an actual world model rollout. Instead, I think the usage of this result is to gauge how well the proposed mechanism senses a poor world model rollout. So instead, the authors can do an experiment, where it compares the actual simulated rollout and the predicted world model rollout, and categorize ones that are relatively correct versus incorrect, and see if the proposed method can capture this difference better.

    • (major) Lack of testing with downstream control experiments: Currently there is limited testing of how this mechanism benefits control. However, this is exceptionally important to ensure that the mechanism’s improvement is actually in what is most necessary. Currently the experiment seems to only have one test, and compared only to base model. It will be important to compare across multiple tasks, over a significant amount of trials, and with a highest performant baseline from previous experiments to highlight the performance.
    • Evaluation: It would be good to separate the method’s ability to handle both aleatoric and epistemic uncertainty. Currently the experiments capture uncertainty due to lack of evidence. However, there is also a category where it is just too difficult to model (aleatoric), how well does the method compare to this?
    • 4.3, it is hard to understand where this projection is coming from. Which experiment, which baseline?
  • Writing is hard to understand. I strongly suggest a heavy rewrite, and review by a native english speaker to check for clarity. This hinders understanding of the novelty, approach and experimental setup.

    • Intro: (i) Doubt concerns support for the input, not just the shape of the output.
    • Experiments: “We report the lift p1/p0”. What does this mean?
    • Experiments: We run the same weights and seed twice through a dead end whose deep half was repainted after training: every wall is where it was, so a policy that learned the map walks in as before. Once the agent ignores its doubt (base); once it acts on trust with the veto of Sec. 3.3 (ours). The veto is one rule: when doubt holds above the alarm line for three frames, trust in the course collapses, the agent turns, and its policy resumes. For the first 24 steps the two are one agent. At step 24 the veto fires; ours turns, leaves the dead end and reaches the goal, an armour, at step 190, while base circles the room until step 262 and arrives at step 362 (Fig. 8): the same policy, deciding 238 steps earlier and arriving in half the steps. Over its whole walk the entropy of base crosses its own line for a single frame, so the same rule would never have fired. On both altered maps, 143 of 145 vetoes fired inside a changed room, though the agent is never told where the changes are (App. F.13).
Questions for authors
  • Experimental settings can be more realistic for the “before decision” result. Currently the setup is, at different times, random actions are given. However, this seems very unrealistic for an actual world model rollout. Instead, I think the usage of this result is to gauge how well the proposed mechanism senses a poor world model rollout. So instead, the authors can do an experiment, where it compares the actual simulated rollout and the predicted world model rollout, and categorize ones that are relatively correct versus incorrect, and see if the proposed method can capture this difference better. How would LucidWM perform in this experiment?
    • Currently there is limited testing of how this mechanism benefits control. However, this is exceptionally important to ensure that the mechanism’s improvement is actually in what is most necessary. Currently the experiment seems to only have one test, and compared only to the base model. It will be important to compare across multiple tasks, over a significant amount of trials, and with a highest performant baseline from previous experiments to highlight the performance. How would LucidWM perform in this experiment?
7 comments
AuthorsOpenReview ↗

Response to Reviewer FKUZ (1/3)

Response to Reviewer FKUZ (1/3)

Thank you for the candid review, and for recognising the soundness of the approach and the breadth of the comparison across world models. Your two major points each asked for an experiment, and the revision adds both; we answer every point below in your order. In the revised PDF, every change to the text is marked in red and every new table or figure has a red caption; a revision record at the very end of the paper (Appendix G) lists the changes by review point and then sets the submitted and the revised wording of each changed passage side by side, so that you can compare them directly. For a direct look at what the method does, the anonymous repository linked in the abstract, unchanged since the submission deadline, shows the doubt rising before a fall and inside a changed room, next to the readouts we compare against.

F1. Imagined rollouts against real ones.

Thank you for this suggestion; we agree that what matters is whether the doubt senses a poor rollout. In the submitted version, Section 4.5 runs this comparison: from 480 real states per run, the model imagines 15 steps under the actions that were actually taken, and every imagined step is compared with what really happened (Figure 7; ten runs in Appendix F.9). Following your suggestion, a new Appendix F.15 sorts these imagined rollouts as you describe: in each run, the 20% of rollouts with the largest prediction error, averaged over their 15 steps, are called wrong and the rest right, and each readout is scored by its AUROC at telling them apart (0.5 is chance). Each rollout is scored by the trust compounded over its 15 steps and, for the entropy, by its sum over them. Trust scores 0.75 on the three tasks of Figure 7(a) and 0.69 over all ten runs, against 0.53 and 0.50 for the entropy, and it is above the entropy in 9 of the 10 runs. On the three runs from pixels the entropy even points the wrong way (0.29 to 0.42): the rollouts it is least sure of tend to be the more accurate ones. The stricter the line for wrong, the clearer the gap: with only the worst 10% called wrong, trust reaches 0.81 on the three tasks, while the entropy is 0.55. Test (B) of Table 1 complements this: random actions are not how an agent acts, but they produce futures that are known to leave the model's experience, to a degree we can set (Appendix F.5). So LucidWM does capture the difference you describe: its trust tells which imagined rollouts have gone wrong, where the entropy of the same predictions does not.

F2. Downstream control.

Thank you for raising this; we agree that the doubt is worth most where it changes what the agent does. In the submitted version, the agent acts on trust in two ways and in two environments: it learns from trust in Crafter, and it decides by trust in the maze. Following your request for more trials, the revision repeats both over three training seeds (Appendices F.12 and F.13). In Crafter, LucidWM, which learns from the trust-weighted return of Eq. 7, reaches an evaluation return of 8 after 0.41M steps on average, where its base needs 0.88M, and over the last 100 evaluation episodes it scores 15.3% against 10.1% on the official Crafter score, ahead on every seed. In the maze, the veto of Section 3.3 turns the agent back where the world has changed: on three trained agents (on the repainted dead end, 20 paired walks per agent in which the veto fires; on the opened map, 100 walks per agent), 92% and 99% of the vetoes fall in the changed region, against chance levels of 56% and 79%. On the opened map, where the goal has moved into the newly opened branch, the agent that acts on trust spends 51% of each walk in the room where the goal used to be, against 65% when it ignores its doubt, a drop of 14 points (95% interval 11 to 17) over 300 paired walks. On the dead end, where the agent is also kept from re-entering the region where its veto fired, it reaches the goal 61 steps sooner on average than without the veto (95% interval 41 to 81) over the 60 paired walks. The walk of Section 4.6 is one of these pairs and is unchanged by this rule. On three continuous-control tasks, the full method keeps the return of its base (Appendix F.16). As for the costliest baselines, the ensembles, Appendix F.1 compares the alarm that triggers the veto over ten walks through the opened door: the doubt raises it on eight, while the deep and self-ensembles, at three times the cost, and the entropy raise it on none, so on these walks a veto driven by them would not have fired. So, in the experiments we have, trust already changes what the agent learns and how it acts once the world has changed, at no added cost.

AuthorsOpenReview ↗

Response to Reviewer FKUZ (2/3)

Response to Reviewer FKUZ (2/3)

F3. Aleatoric and epistemic uncertainty.

Thank you for this suggestion; the distinction you draw is the right one to test. In the submitted version, Section 4.4 already probes both kinds within one imagined rollout, alongside the tests of missing evidence in Table 1. We imagine the same 24 steps of cheetah from six checkpoints of one run, and for the first step, which the model can learn with more experience, the doubt halves while the error falls 24.6-fold. Step 24 lies past the point where the dynamics turn chaotic, so no amount of training makes it right, and there the doubt stays the highest of the steps shown at every checkpoint (Figure 6, Appendix F.7). So the doubt falls where more experience removes the error and stays the highest where the outcome is too difficult to model, whereas the entropy of the same predictions, averaged over the 24 steps, rises from 0.79 to 1.01 as the error falls (Appendix F.7). Following your suggestion, Section 2 now names the two kinds, and a new paragraph in Appendix C.4 shows how SL holds both in one Dirichlet, read in one forward pass: the uncertainty mass on the epistemic side, and the direction of the evidence, the spread of the predicted outcome, on the aleatoric side. Proposition 4 shows how the two meet in the doubt: it nearly coincides with the count doubt of Eq. 11 when some outcome was never observed and stays higher when the outcome is spread over every class, so noise that reaches every outcome keeps the doubt up; the doubt is therefore close to epistemic uncertainty but not identical to it.

F4. The projection in Section 4.3.

Thank you for pointing this out. In the submitted version, Table 6 in Appendix E.1 lists the run behind Figure 4, walker on DreamerV3, but Section 4.3 did not say what the base is or how the plane is drawn. The base is the same head trained without the evidence discipline of Eq. 5, an ablation of ours and not the unmodified DreamerV3 of Table 1, so the comparison isolates what the discipline does. The plane is linear and chosen to show the drift, not a plain PCA or a t-SNE embedding: its horizontal axis points from the mean of the visited states to the mean of where the random-action rollouts end, and its vertical axis is the first principal component of the visited states, made orthogonal to the horizontal axis. The distances quoted in Section 4.3 are measured in the full state, not on this plane. Following your suggestion, Section 4.3 now names the base and points to a new paragraph in Appendix E.4 with these details.

F5. Writing.

Thank you for this candid assessment. Following your suggestion, the revision rewrites in plain terms the three passages you quoted, as described in F6 to F8 below. It also defines these terms where they are introduced: AUROC and the lift in Section 4.2, the self-ensemble in Section 4.1, the decision process in Section 2 and Appendix A, and the base of Figure 4 in Section 4.3. Beyond these, the method section is clearer in four places: Section 3 now carries a descriptive title, Section 3.1 says how evidence is counted when no pair of scene and action recurs, and Section 3.3 states what Theorem 1 does and does not claim and names both uses of the score JJ. Section 2 now names the two kinds of uncertainty and the core parts of the world model, and the introduction and Section 4.6 give the sample unit of each number they report. Changes to the text are marked in red, and Appendix G lists them by review point and shows each changed passage before and after.

F6. Point (i) of the introduction.

Thank you; we agree that the heading was too compressed. It now reads “Doubt depends on the experience behind a prediction, not on how sharp it is.” The sentences after it give the reason: a familiar scene and a familiar action can be combined in a way the model has never experienced, and the model may still predict the outcome sharply (Figure 2), so the sharpness of a prediction alone cannot show that experience supports it.

F7. The lift.

Thank you for flagging this. Section 4.2 now spells the lift out in words: for each readout, the lift ρ1/ρ0\rho_1/\rho_0 is its value when every action fed to imagination is random, divided by its value under the true actions. A lift of 1 therefore means that the readout does not respond, and a readout passes when its lift is above 1.

AuthorsOpenReview ↗

Response to Reviewer FKUZ (3/3)

Response to Reviewer FKUZ (3/3)

F8. The veto paragraph in Section 4.6.

Thank you; Section 4.6 now describes the test step by step, in the order it runs. The maze has a dead end that was sealed during training; at test time it is opened and its deep half repainted, and we start the same trained agent twice deep inside it. In one run the agent ignores its doubt; in the other it uses the veto, which turns the agent around when the doubt stays above its alarm line for three frames in a row. The two runs are identical until the veto fires at step 24; the agent with the veto then leaves the dead end and reaches the goal at step 190, while the other keeps circling in the dead end and reaches the goal at step 362.

Reviewer_KVMrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how discrete-latent world models can distinguish spread-out predictions from transitions that lack empirical support. The authors propose LucidWM, which reinterprets the outputs of a categorical transition head as evidence in Subjective Logic: the direction of the evidence determines the state prediction, the total evidence determines doubt, and a standard categorical head emerges as the special case of constant doubt. During training, a regularization term pulls unsupported evidence back toward a vacuous prior and cumulative evidence fusion replaces direct posterior overwriting; during imagination and planning, doubt is converted into a multiplicative trust weight along trajectories to discount unreliable imagined returns and veto untrusted action sequences. Experiments across multiple world-model backbones and continuous-control, maze-navigation, and open-world environments evaluate out-of-experience detection, imagination-error filtering, and decision making.

Critical strengths
  1. Unfamiliar transitions and spread-out predictions are cleanly separated into two quantities without adding an extra network. Proposition 1 shows that the uniform mixture term ε\varepsilon in a standard categorical head is equivalent to assuming one fixed amount of evidence for every transition. The proposed head lets total evidence vary with the state and action while reading the predicted direction from the same outputs, so the model can express high doubt even when its categorical prediction is sharp. Figure 13 verifies this on the same prediction: predictive entropy alarms on 0 of 20 consecutive falls, whereas doubt alarms before every fall.

  2. Out-of-experience detection is consistent across four independent world-model code bases. After a sealed room is opened in the maze, detection AUROC on all four bases (0.77, 0.83, 0.96, and 0.82) exceeds both the same formula on the unmodified model and predictive entropy. When actions are replaced by random actions in continuous control and Crafter, the response ratio is the highest entry in all 16 rows. Across 100 episodes in the original maze, adding the mechanism does not degrade goal reaching.

Critical weaknesses
  1. The headline decision and learning numbers in the abstract come mainly from single sequences, whereas the multi-seed gains are much more modest. The 20/20 fall warnings come from consecutive falls along one cheetah trajectory (median lead of four steps); the 190-versus-362-step result is a single dead-end walk with one set of weights and one seed; and 143/145 counts veto events landing inside the altered region (127 of them on the opened map), not 145 independent episodes. The multi-seed action evaluation is in Appendix F.11: over 50 seeds on Crafter, selecting actions by trust lowers deaths from 4.08 to 3.80 per 1,000 steps (a 6.9% relative drop) and raises median life from 236 to 253 steps. The Crafter learning curve in Appendix F.12 (15.6% versus 9.0%, and 0.35M versus 0.96M steps to return 8) reports episode-bootstrap intervals without stating the number of independent training seeds, and no full environment-return table is given for continuous control. Theorem 1 establishes a value fixed point inside the imagined model rather than a higher environment return.

    Suggestion: State the sample unit for 20/20, 190 versus 362 steps, and 143/145 in the abstract and main text; report multi-seed arrival steps and success rates for the repainted dead end; and state the number of training seeds behind the Crafter learning curve.

  2. Table 1 omits seed dispersion, and several comparison readouts move strongly in the wrong direction on EMERALD. Table 1 reports five-seed means without standard deviations or intervals. Several margins are small, such as 1.15 versus 1.14 (Laplace) on R2-Dreamer's Crafter row and 0.82 versus 0.75 (unmodified base and entropy) on OC-STORM's maze row. At the same time, on EMERALD's walker row, MC dropout, Laplace, and snapshot give response ratios of 0.05, 0.06, and 0.16 under corrupted actions, and 0.28, 0.29, and 0.25 on cheetah (where 1 is no response and below 1 is the opposite direction). If those implementations are not responding properly on that base, counting them among seventeen opponents inflates the breadth of the lead. In addition, when filtering imagination error across depths, most of the gain comes from horizon depth itself; within a fixed depth, trust removes about 11% more error on average.

    Suggestion: Add five-seed standard deviations or intervals to Table 1; check and explain why the multi-forward baselines fall far below 1 on EMERALD; and distinguish within-depth from across-depth error filtering when summarizing Figure 7.

Questions for authors

Questions whose answers could change the assessment:

  1. Can each cell of Table 1 report the five-seed mean and standard deviation, or a seed-level interval? Do the R2-Dreamer Crafter lift (1.15 versus 1.14) and the OC-STORM maze AUROC (0.82 versus 0.75) still favor Ours across seeds? On familiar data, is the variance of EMERALD's walker-row MC-dropout, Laplace, and snapshot readouts, whose lifts are 0.05, 0.06, and 0.16, sane? Are the cheetah-row lifts 0.28, 0.29, and 0.25 the same phenomenon?
  2. How many independent trajectories and seeds produce the twenty falls? Besides the one 190-versus-362-step dead-end walk, how many paired walks are there, and what are their arrival times and success rates? How many trajectories produce the 143/145 veto counts?
  3. How many training seeds support the Crafter learning curve and the 15.6% versus 9.0% score? On walker, cheetah, and pendulum, what are the evaluation returns with the discipline and the trust-weighted return relative to the standard Dreamer return?

Future work, not requested as a condition for changing this rating:

  • Test the veto on longer planning horizons or a physical robot.
  • Check, in the settings of Jürgens et al. (ICML 2024), whether this doubt falls in regions of high aleatoric uncertainty.
5 comments
AuthorsOpenReview ↗

Response to Reviewer KVMr (1/4)

Response to Reviewer KVMr (1/4)

Thank you for the precise and thorough review, and for recognising that the doubt separates unfamiliar transitions from spread-out predictions without an extra network, on four independent code bases. We answer each point below in your order. In the revised PDF, every change to the text is marked in red and every new table or figure has a red caption; a revision record at the very end of the paper (Appendix G) lists the changes by review point and then sets the submitted and the revised wording of each changed passage side by side, so that you can compare them directly. For a direct look at what the method does, the anonymous repository linked in the abstract, unchanged since the submission deadline, shows the doubt rising before a fall and inside a changed room, next to the readouts we compare against.

K1. The twenty falls.

Thank you for asking for the sample unit. The twenty falls come from twenty different trajectories: they are the first falls of twenty consecutive episodes of one training seed on cheetah from pixels. Following your question, Appendix F.2 now counts the first fall of every training episode of each of five seeds, 892 falls in all (Table 9): the doubt warns before 69% to 79% of the falls in each seed, and the entropy before 4% to 7%, close to its own rate of alarms on quiet frames. The doubt's alarm stays on into the fall in almost every case, whereas the entropy's few alarms come about fifteen steps before the fall and do not last until it. Figure 1 shows one of these falls as an example.

K2. The 190 against 362 steps.

Thank you; you are right that the 190-versus-362-step result is one walk: the abstract presents it as a case study, and Section 4.6 runs the same trained agent twice from the same start with the same seed to show how the veto acts. Following your suggestion, Appendix F.13 now repeats this walk on three trained agents, each with 20 paired walks from deep inside the repainted dead end, with and without the veto from the same start, counting the walks in which the veto fires. After a veto the agent is also kept from re-entering the region where it fired; the walk of Section 4.6 is one of these pairs and is unchanged by this. With the veto, the median arrival is 196 steps against 235 without it, 61 steps sooner on average (95% interval 41 to 81), sooner in 48 of the 60 pairs and later in 12; the agent reaches the goal in all 60 walks with the veto and in 58 without it, and Table 12 lists each agent.

K3. The 143 of 145 vetoes.

Thank you; you are right that 143 of 145 counts vetoes, not independent episodes. They come from one trained agent over 80 walks, 50 on the opened map and 30 on the repainted dead end, 64 of which contain at least one veto. Since the agent spends much of each walk in the changed region, the count must be read against chance: had each veto fired at a random moment of its own walk, about 108 of the 145 would have fallen there, against the 143 observed. Following your comment, Appendix F.13 now states this sample unit and repeats the count on three trained agents (on the repainted dead end, 20 paired walks per agent in which the veto fires; on the opened map, 100 walks per agent): 92% and 99% of the vetoes fall in the changed region, against chance levels of 56% and 79% (Table 11).

K4. The size of the multi-seed gains.

Thank you for raising this. In the submitted version, the experiment in this appendix varies one thing only: at every step the same trained agent imagines the same eight futures, and we rank them either with trust or without it. Trust has little room to act here, since it can only pick among eight futures that the agent's own policy proposes. Even so, deaths drop by the 6.9% that you cite, with no retraining and no extra forward pass. The gains across seeds are larger where trust does more than pick. In learning, LucidWM reaches an evaluation return of 8 after 0.41M steps, against 0.88M for its base, over three training seeds (K5); with the veto, the agent reaches the goal 61 steps sooner on average over the 60 paired walks (K2).

K5. The training seeds behind the Crafter curve.

Thank you for asking about the training seeds. In the submitted version, the learning curve and the score of Appendix F.12 come from one training seed per agent. Following your question, Appendix F.12 now states that Figure 21(b, c) shows seed 0, and reports three training seeds per agent, listed seed by seed in Table 10. Across these seeds, LucidWM reaches an evaluation return of 8 after 0.41 ± 0.06M steps, against 0.88 ± 0.09M for its base (mean ± standard deviation over seeds). Over the last 100 evaluation episodes of training, it scores 15.3 ± 0.9% against 10.1 ± 1.1% on the official Crafter score, and it is ahead on every seed on both measures.

AuthorsOpenReview ↗

Response to Reviewer KVMr (2/4)

Response to Reviewer KVMr (2/4)

K6. Returns in continuous control.

Thank you for asking for the returns. In the submitted version, Appendix F.10 shows that the return on walker stays level across weights of the discipline (Fig. 20c). Following your question, Appendix F.16 now reports the evaluation return of the full method, with both the discipline and the trust-weighted return, against that of standard DreamerV3. It covers walker, cheetah and pendulum from proprioception, with three training seeds per task (Table 16). The full method reaches 947.5 ± 11.9 against 931.8 ± 19.3 on walker, 724.7 ± 46.1 against 688.9 ± 50.4 on cheetah, and 753.0 ± 39.2 against 741.0 ± 76.9 on pendulum (mean ± standard deviation over seeds). LucidWM is built to quantify uncertainty, not to raise the return, so the question on these tasks is whether the doubt costs any return. We see none: the mean return of the full method is in fact higher on all three tasks, though within the spread across seeds. In Crafter, a harder task, LucidWM also learns faster than its base (K5).

K7. The scope of Theorem 1.

Thank you; this is the reading we intend. In the submitted version, Section 3.3 presents Theorem 1 as a guarantee that the reweighting does not change what the critic converges to, and part (i) names this fixed point as the value of the policy in the imagined model. Part (ii) ties the trust-weighted return to the true value: for a critic at that value, the bias of this return is bounded by the model's one-step errors, each weighted by the trust that reaches its depth. The theorem makes no claim about the return in the environment, which we measure instead: it is kept in continuous control (K6), and learning is faster in Crafter (K5), each over three training seeds. Following your comment, the text after Theorem 1 now states this scope and points to these two results (Appendices F.12 and F.16).

K8. The sample units in the abstract and main text.

Thank you for this suggestion. In the submitted version, of the three numbers the abstract reports only the 190-versus-362 walk, as a case study, and Section 4.6 already gives its unit (K2). Following your suggestion, the introduction now gives the unit of both numbers it reports: the twenty falls come from consecutive episodes of one training seed, and the 190 against 362 steps are one paired walk. Section 4.6 now says that the 143 of 145 are vetoes over 80 walks of one agent. The introduction also points to the counts over several seeds, in Appendix F.2 for the falls (K1) and Appendix F.13 for the walks and the vetoes (K2, K3).

K9. Seed dispersion in Table 1.

Thank you for this suggestion. Every cell of Table 1 now reports the mean over five training seeds, with its 95% confidence half-width below it. Appendix F.14 adds, for every row, the 95% interval of our margin over the strongest other readout (Table 13), and shows the same comparison as bars (Figure 23). Part (A) compares separately trained models, so its intervals are Welch t-intervals; part (B) computes all readouts on the same model and seed, so its intervals are paired t-intervals. Ours has the higher mean in all 20 rows, and the interval of its margin lies above zero in 18. On the other two, R2-Dreamer on Crafter and EMERALD on pendulum, the interval includes zero, and we report them as ties; K10 discusses the first.

AuthorsOpenReview ↗

Response to Reviewer KVMr (3/4)

Response to Reviewer KVMr (3/4)

K10. The two small margins of Table 1.

Thank you for pointing to these two rows. On R2-Dreamer in Crafter (Table 13), the lift over five seeds is 1.15 ± 0.05 for ours against 1.14 ± 0.15 for Laplace; the 95% interval of the margin runs from −0.16 to +0.19, so we now report this row as a tie. The margin is small because the test has little to detect on this row: with every action replaced by a random one, the prediction error of imagination rises only 1.03-fold here, against 1.5- to 6.6-fold on the three continuous-control tasks of the same base. Imagination hardly leaves experience, so a readout that tracks experience should hardly rise: the doubt rises 1.15-fold here, against 1.5- to 3.1-fold on those three tasks. Laplace matches it on this row only with twenty forward passes per step, where the doubt adds none. On OC-STORM in the maze, the AUROC is 0.82 ± 0.04 for ours against 0.75 ± 0.05 for the unmodified base and 0.75 ± 0.08 for the entropy; the 95% interval of the margin runs from +0.01 to +0.13 against the base and from −0.03 to +0.17 against the entropy, so ours has the higher mean, clearly against the base and within the spread of the entropy. The margin is small because, for OC-STORM in the maze, the failure that the doubt repairs does not occur: the entropy of the unmodified model is already higher in the new room than on familiar frames (0.76 against 0.60), so its entropy and the same formula on it already point the right way. In Table 1 this failure appears on the two recurrent bases, where the same formula on the unmodified model reads the new room as familiar (0.24 and 0.37), and not on the two transformer bases (0.72 and 0.75). A likely reason is how the state after an observation is formed: the recurrent bases infer it from the frame together with their recurrent state, which carries the confidence built on the familiar walls into the new room, while the transformer bases encode each frame on its own, so an unseen room is encoded less sharply. Where the test corrupts actions instead, the same formula on unmodified OC-STORM stays near 1 (1.03 to 1.16) and ours reaches 2.3 to 5.6.

K11. The multi-forward readouts on EMERALD.

Thank you for flagging these cells; we checked the three readouts as you suggest. On familiar data their values are of the usual order: on walker under true actions, the spread is 0.015 for MC dropout, 0.002 for Laplace and 0.056 for the snapshot ensemble, against 0.029, 0.010 and 0.076 on the unmodified EMERALD, and no value underflows. Their reversal under random actions is not an implementation fault. Test (B) reads every readout from the same model and the same imagined rollouts as ours, so that the comparison is paired, and under random actions this model's prediction spreads to almost uniform: its entropy rises from 2.17 to 3.44, against a maximum of 3.47. Perturbed passes over an almost uniform prediction return nearly the same distribution, so their disagreement falls exactly where the doubt rises, which gives a lift below 1. Two checks support this: under true actions the spread of MC dropout is already lower where the doubt is higher (rank correlation −0.69), and across five variants of this model with different regulariser weights the lift of MC dropout falls from 0.53 to 0.04 as the doubt under random actions rises from 0.76 to 0.99. Cheetah is the same phenomenon in milder form: the entropy reaches 3.20, and the lifts are 0.25 to 0.29. Read on the unmodified EMERALD instead, MC dropout and Laplace have lifts of 0.98 to 1.03 on the three tasks and the snapshot ensemble 0.89 to 1.57, so under either reading ours is the highest on these rows (2.06 to 3.67).

K12. The count of seventeen readouts.

Thank you; the breadth of the lead should not rest on these cells, and it does not. Table 13 measures the lead on each row against the strongest other readout, and on EMERALD's walker, cheetah and pendulum rows that readout is Mahalanobis, Mahalanobis and RND (2.08, 2.58 and 1.87), not one of the three. Read on the unmodified EMERALD, the three stay below these values (K11), so on these rows no margin and no outcome in Table 13 depends on how they are read. Nor are they weak opponents elsewhere: one of the three is the strongest other readout on four of the sixteen rows of test (B), on R2-Dreamer and OC-STORM, where MC dropout reaches 4.37. We therefore keep them among the seventeen, and count the lead row by row in Table 13.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Request for feedback on our responses

Dear Reviewers,

We appreciate your time and effort in reviewing our ICLR submission.

We noticed that some of the comments may have arisen from misunderstandings of our methodology and contributions. We have made every effort to clarify these points and address your concerns in our rebuttal and the revised paper.

We would greatly appreciate it if you could take another look at our responses and let us know if any concerns remain before the discussion period ends on 18 November.

Best wishes,

The Authors

AuthorsOpenReview ↗

To all reviewers: point-by-point replies and an updated paper

Dear Reviewers,

Following further suggestions from Reviewer K6Y7, we have made two updates.

Replies. Under each review there is now a reply in which every comment of that review is quoted in short and answered briefly. These replace our earlier replies, which we leave in place so that the threads stay intact.

Paper. The main results are now a bar chart with 95% confidence intervals in the main text (Figure 4), and Table 1 has moved to the appendix, where every cell carries its complete interval. In the tables, bold marks the largest entry and every entry that is not significantly below it. The learning experiments are now in the main text (Section 4.6, Figure 9). With two figures added to the main text, figure numbers after Figure 3 have shifted; the new replies use the new numbers.

We would be glad to answer anything further. Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.