- Rating
- 4(+2)
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a new way to train world models, rooted in Subjective Logic, which gives the model inherent, principled means to reason about epistemic uncertainty.
Critical strengths
-
The idea seems sound (my familiarity with Subjective Logic is unfortunately limited).
-
The significance of the achievement, assuming the results stand (see weaknesses section), is tremendous.
-
The presentation is for the most part very clear and serves the contributions very well.
Critical weaknesses
I'll preface this section with two statements:
First, in my opinion both the paper and its contributions are excellent and should be published, and I would like us together throughout this rebuttal to bring the paper to the point where I can easily give a score of 4.
Second, my familiarly with Subjective Logic, which underpins the entire theoretical support for this work, is unfortunately limited, which influences my review and I think should influence the weight of my related comments (and more importantly, the lack of comments about the soundness of the underlying approach).
Below are the issues I think we need to address before we can increase the score:
Evaluation and statistical significance:
-
The main metric used to evaluate the contributions is AUROC. I've never heard of it, and it is not explained or even cited. Please make sure the metric used is sufficiently well explained in the body of the paper to understand the results, and that the choice of metric is motivated. This is one of the most major items: I am currently unable to evaluate the performance of the suggested method essentially at all.
-
There is almost no metric of stat. signif. for any of the presented results. Please make sure that whenever relevant (everywhere, I think?) each evaluation is supported by a principled stat. signif. metric, such as a proper CI over the mean.
-
Tables are the single worst way to present results. If (if!) the best metric to present is indeed a single score, please consider using bar charts instead, for example. If these results can be evaluated during training for example, consider plotting complete training curves.
-
I would have liked to see RL experiments that show that this metric is sufficient to drive exploration in sparse-reward RL tasks. I think this is one of the most important proof-of-concepts for uncertainty estimation. I understand that this is an additional significant layer of complexity. Whether this is necessary or not will, to me, depend strongly on the re-presented results section and the explanation of the metric.
Theoretical motivation:
Due to my limited familiarity with SL, it's hard for me to distinguish to what extent the different loss terms etc. are directly theoretically motivated by SL, are approximations to quantities that are fully motivated, or are heuristics inspired by SL. I think that it's important that even a reader that is not fully familiar with the theoretical basis can know whether the approach is fully or partially supported by said theoretical basis. Please make sure that every for every contribution (I'm thinking eqs. 4-8, 10) it is made clear and explicit to what extent each it is supported by SL ("we approximate X with Y", "we track X with Y", "inspired by X, we use Y", etc).
Evidence in cont. spaces:
Despite limited familiarity with SL, the idea makes a lot of sense to me in discrete spaces. However, the underlying state space, and the action space - which is itself the same as the environment's I believe? - may very well be continuous. Reasoning about evidence in continuous spaces is.. nontrivial, because we (may) never see the same transition twice. This is never explicitly addressed. Please discusses this (first in the rebuttal - I would like to understand how this is addressed, and then the resulting discussion in the paper).
Additional minor items and questions:
-
What is a "Self ensemble"?
-
In the "Deciding what to do" section it's not clear to me whether a concrete method to decide what to do (presumably meaning, a way to select actions? during planning or in the environment?) is proposed, or whether it's just pointed out that J can be used (in different ways) to direct action selection. Please make it clear whichever of the two it is. It's better if it's the first, but I think the contribution remains sufficiently significant to publish even if not.
-
There are some small issues (in my opinion) with notation and definition of terms:
-
First, reward, transition, values etc are all discussed and used, but a proper MDP definition is never given. I think that giving a proper definition of an MDP and then the WM in light of it will both improve the presentation period, as well as help address the issues below.
-
Second, the WM is defined as not-having a reward or value head (I don't like it, but I can live with it). But then all the followup quantities appear to be defined with respect to reward and value heads. Including the uncertainty, which is "for WMs", as I understand it. So which is it? Is there a reward and value, or not? Personally, I would recommend going the direction that states that in principle there are reward and value models, even if it is possible to ommit them and only learn latent transition dynamics. Once the MDP is fully defined, it will all become very natural to present coherently I think.
-
I don't love the choice of symbols. is a latent state. Ok, sure (although - probably not great, because should be the true state of the MDP?). But then, what is ? It's a predicted latent state. It's still a latent state. Wouldn't it make more sense to call it ? Why are observations and not for example ? Are observations (and thus POMDPs with their additional layer of complexity) necessary? I would suggest the following: 1. Define an MDP or POMDP properly with all the standard symbols, which will make the presentation clearer. 2. If is used for latent-states, than the encoder can output and the transitions (an attempt to learn what the encoder would predict). Alternatively, to use for the encoder's output and for the dynamics output, and explain that in principle both try to predict the same latent state, but we use different letters to denote that they come out of different predictors.
-
-
I think that section 3 should be called something more informative. Perhaps - "evidence grounded model learning," or something that actually describes the method. Often I would suggest using the title of the paper, but in this case the title is a bit marketing-oriented, so maybe something more informative would be better here.
-
It's not clear to me that Figure 1 belongs before the abstract. In fact, the significance of the contribution, to me, became clear in the introduction rather than from the Figure.
-
The presentation of the paper is very clear in my opinion, despite the entire paper's point being epistemic uncertainty estimation while the word "epistemic" doesn't even appear. This is not so much a weakness as much as its a genuine achievement, in my opinion. However, I would suggest at least referring to these terms (epistemic, aleatoric) and contrasting SL with the standard ideas / ways to quantify epistemic / aleatoric uncertainty. Could be part of a related work section.
Questions for authors
Please see the weaknesses section.
13 comments
Response to Reviewer K6Y7 (2/5)
Response to Reviewer K6Y7 (2/5)
T3. Evaluation 3, presentation.
Thank you for this suggestion. In the submitted version, Table 1 is the only table in the main text, every other result is shown as a figure, and two results are plotted over training. Figure 6 follows the prediction error and the doubt of one run across its checkpoints, with all six checkpoints and the entropy in Figure 18. Figure 17(b) tracks the AUROC of the maze test in Table 1(A) over 38 checkpoints of one DreamerV3 run, from 0.1M to 3.8M frames. The doubt stays above chance at every checkpoint, while the same readout on the unmodified base stays below chance. New in the revision, following your suggestion, Appendix F.14 shows Table 1 as a bar chart (Figure 23), with ours next to the strongest other readout on every row as five-seed means with 95% intervals. The caption of Table 1 now points to this figure. The standard deviation now printed under every mean shows how much each row varies across seeds. We have kept Table 1 in the main text, since it gives the exact value of every readout on every row, and added the bar chart in Appendix F.14.
T4. Evaluation 4, exploration.
Thank you for this suggestion. We agree that exploration is an important use of uncertainty, as in Plan2Explore with ensemble disagreement (Section 2). This paper uses the doubt the other way round. An exploration bonus steers the agent toward states the model is unsure about, whereas we use the doubt to discount what the model imagines in those states, both in what the agent learns from and in what it decides. Section 4.5 tests whether the steps that trust discounts are the wrong ones, Section 4.6 tests deciding, and Appendix F.12 tests learning. Table 1 also runs a standard check of the doubt itself, whether it flags what the model has not experienced (Hendrycks and Gimpel, 2017), and the doubt passes it on all four world models, in all 20 rows. In Crafter, an open world whose rewards come from unlocking achievements, LucidWM reaches an evaluation return of 8 after 0.41M steps on average over three training seeds, where its base needs 0.88M (Table 10). You kindly noted that whether these experiments are needed would depend on the re-presented results and the explanation of the metric; the revision now defines and cites AUROC and adds the spread across seeds and the 95% intervals of Appendix F.14 (T1 to T3). Whether the doubt also works as an exploration bonus in sparse-reward tasks is an interesting question that we would like to study in future work.
Response to Reviewer K6Y7 (3/5)
Response to Reviewer K6Y7 (3/5)
T5. Theoretical motivation.
Thank you for this request, which will help readers new to SL. In the submitted version, Section 3 names the source of each equation where it is introduced, and Table 3 in Appendix B sets Eqs. 4 to 7 and 10 beside their standard forms. Following your suggestion, Table 3 now has a row for Eq. 8 and a column that states how each equation relates to SL. Equation by equation:
- Eq. 4, SL applied directly. It is the mean of an SL opinion with a uniform base rate (Eq. 3); our two choices are to read the logits as evidence through a softplus, as is common in evidential deep learning, and to floor the doubt at the standard head's , so that the head never trusts a transition more than the standard world-model head does.
- Eq. 5, adapted from evidential learning, never more confident than the SL count at its optimum. It adapts the KL regulariser of evidential deep learning (Sensoy et al., 2018), pulling all evidence toward the vacuous opinion of SL, the opinion with no evidence. In an idealised model with one opinion per scene–action pair, Proposition 4 shows that at its optimum the doubt is never lower than the doubt of SL's count; the two nearly coincide when some outcome was never observed (0.24 against 0.20 in the example of Appendix C.4), and otherwise its doubt stays higher. Section 4 tests the shared network.
- Eq. 6, SL applied directly. It is the cumulative fusion of SL (Jøsang, 2016), which adds the evidence of the two sources, and the observation evidence is read from the observation alone, so that the two sources stay apart as the operator assumes.
- Eq. 7, our construction, compounding trust as SL trust discounting does. It lets the share of the -return follow the trust , rescaled so that recovers the -return. Reading each as the probability that step is reliable and the rollout as a chain of referrals, Proposition 8 shows that the trust it compounds equals the transitive trust discounting of SL, in which trusts multiply along the chain.
- Eq. 8, exact. It rewrites Eq. 7 with no approximation (Lemma 2).
- Eq. 10, our construction for acting. It is Eq. 7 with and the critic's share set to zero before the last step, a design choice that weights rewards by the same compounded trust.
T6. Evidence in continuous spaces.
Thank you for this question, which goes to the heart of how the evidence is counted. In the submitted version, Appendix C.4 touched on this only briefly, at its end, in the network case, where “the evidence of every pair is one function of shared parameters”, and contrasted it with Posterior Networks (Charpentier et al., 2020), which keep counts over continuous inputs with a learned density that our head does not need. Following your suggestion, Section 3.1 now answers the question in brief where the evidence discipline is introduced, and Appendix C.4 now gives the full explanation below under its own heading, “Continuous scenes and actions”. Yes, the actions are the environment's own, continuous in DeepMind Control and discrete in the maze and Crafter (Section 4.1). First, what is counted is discrete: the evidence is over the next latent state, categorical variables of classes each ( in DreamerV3), exactly as in the discrete case. Second, the pair does stay continuous, but no pair needs to be seen twice, because is the output of one network that all pairs share, not a count stored per pair. The dynamics loss holds the evidence up on the pairs the agent has experienced, since a sharp posterior cannot be fitted without it (Proposition 3), and the shared parameters carry it to similar states and actions, so a new pair receives the evidence of the experience around it. The discipline (Eq. 5) pulls the evidence down at every training pair, so the network defaults to low evidence and keeps it high only where the data hold it up; the experiments below show that away from the data the evidence falls and the doubt rises. With one entry per pair, as in a table, Propositions 3 and 4 prove this; for the network, the experiments show it. On walker, where states and actions are continuous, imagined rollouts with random actions end up twice as far from the training states as any training state is from its nearest neighbour, and by the end nine in ten have doubt above the 95th percentile measured on the training states (Figure 4). The same model trained without the discipline drifts as far and its doubt stays flat, as Section 4.3 now states. Table 1(B) replaces the actions by random ones on its three DeepMind Control tasks, and the doubt rises for all four world models.
Response to Reviewer K6Y7 (1/5)
Response to Reviewer K6Y7 (1/5)
Thank you for the careful and encouraging review, for your confidence in the contribution, and for offering to work with us toward a score of 4. We answer each point below in your order. Several of them were covered only in the appendix, where they were easy to miss, and the revision brings the key ones forward to where they first matter. In the revised PDF, every change to the text is marked in red and every new table or figure has a red caption; a revision record at the very end of the paper (Appendix G) lists the changes by review point and then sets the submitted and the revised wording of each changed passage side by side, so that you can compare them directly. For a direct look at what the method does, the anonymous repository linked in the abstract, unchanged since the submission deadline, shows the doubt rising before a fall and inside a changed room, next to the readouts we compare against.
T1. Evaluation 1, AUROC.
Thank you for raising this point. You are right that the submitted version only named AUROC, with its chance level in the caption of Table 1, and neither defined nor cited it. Following your suggestion, Section 4.2 now defines and cites AUROC where the text first uses it (Fawcett, 2006), and Appendix E.4 describes it in detail. AUROC, the area under the receiver operating characteristic curve, is the probability that a random positive example scores higher than a random negative one, so 0.5 is chance and 1 is perfect separation. It is to detection what accuracy is to classification, and it is the standard score for detecting inputs a model has not seen (Hendrycks and Gimpel, 2017). In our test, positives are frames of the new room and negatives are frames of the familiar maze. Because it needs no threshold, AUROC compares readouts on different scales directly. On DreamerV3 in Table 1(A), for example, our doubt ranks a new-room frame above a familiar frame in 77 of 100 random pairs, while none of the eleven other readouts exceeds chance, the best only 48 of 100. The other three bases show the same picture: there the doubt scores 0.83, 0.96 and 0.82, the best readout on each (Table 1(A), Figure 23). A value below 0.5, shown in grey, means that a readout, read in the direction its authors define, judges the new room more familiar than the maze. This direction is fixed in advance for every readout, since an agent that had to act on a readout could not choose its direction after the fact. Test (B) of Table 1 uses a second score, the lift, which Section 4.2 now also spells out: a readout's value when every imagined action is random, divided by its value under the true actions, so that 1 means no response.
T2. Evaluation 2, statistical significance.
Thank you for this suggestion. In the submitted version, Figures 7 and 21(c) show 95% bootstrap intervals and Figure 20 repeats the analysis of Figure 7 over ten runs, but, as you note, Table 1 reported means only. Following your suggestion, every cell of Table 1 now reports the mean and standard deviation over five training seeds, and Appendix F.14 gives, for every row, the 95% interval of our margin over the strongest other readout. Ours has the higher mean in all 20 rows, and in 18 of them the 95% interval of its margin lies entirely above zero, so in these 18 rows the lead is significant at the 5% level. Part (A) compares separately trained models, so these intervals are Welch t-intervals, and part (B) computes all readouts on the same model and seed, so they are paired t-intervals. On the two exceptions, R2-Dreamer on Crafter and EMERALD on pendulum, the margin changes sign across seeds, and Appendix F.14 marks them as ties. Appendices F.2 and F.13 add Wilson intervals, the standard 95% intervals for percentages, to the fall and veto rates. In each of the five seeds, for example, the doubt warns before 69% to 79% of the falls and the entropy before 4% to 7%. The learning and decision results are now reported in the same way: Crafter over three training seeds (Table 10), and the veto over 60 paired walks with a 95% interval (Table 12).