- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper investigates why latent uncertainty cannot track long-horizon error. The paper provides theoretical arguments on the above question. They also show empirically that the correlation between true reward error and ensemble disagreement reduces as the rollout horizon grows. Across two world models, DreamerV3 (DMC, Atari benchmarks) and TD-MPC2 (DMC benchmarks), the paper provides a detailed empirical analysis on long-horizon error. They propose a linear calibrated probe to predict rollout's reward error.
Critical strengths
The paper asks a fundamental question of why latent uncertainty cannot track long-horizon error. They provide theoretical arguments on this question. Furthermore, the paper proposes a linear probe to predict rollout's reward error. The empirical evaluation is strong, across many task of DMC and Atari benchmarks, on two world models DreamerV3 and TD-MPC2.
Critical weaknesses
-
The correctness of theorem 4 as stated is not clear. The proof in the appendix is not sufficiently detailed to validate its accuracy. As stated in the Appendix, " sup δt = ∞, which is doubtful on DMC tasks with bounded state spaces".
-
Presentation: The paper's presentation can be improved. (For e.g., Figure 1 size). The paper's key takeaways are distribution throughout in many places and hence are difficult to evaluate.
-
Minor corrections: "Two one-hot vectors over C classes differ in at most one position" in proof of Proposition 2. Though two one-hot vectors differ in at most two positions.
Questions for authors
-
Can the authors provide a detailed and expanded proof for Theorem 4 instead of brief version provided in appendix. It is not clear why many claims hold, such as the bound on U_t. The paper states in appendix that the theorem requires "supt δt = ∞, which is doubtful on DMC tasks with bounded state spaces". Why does the true dynamics not being globally stable imply unboundedness? Please expand on the precise meaning of "Ut cannot order trajectories".
-
The quality of the Figure 1 should be improved include size and resolution.
-
Doesn't the linear probe also satisfy properties in Theorem 4?
-
DreamerV3 (H=15) and TD-MPC2 both use much shorter rollout horizon. Perhaps a more interesting study would be for world models where long rollouts are actually used in practice.