- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies intrinsic motivation for an agent that must identify which world it inhabits, where worlds are sampled from a known distribution and formalized as controlled hidden Markov models with unknown parameters θ. Information gain (IG) is the natural objective: it can be written as the gap between a “first take” (surprise under the prior predictive) and a “double take” (expected surprise under the posterior after observing the outcome), and it ignores noisy TVs by construction. However, computing it requires an explicit posterior over θ, which autoregressive world models do not maintain.
The paper makes three contributions. First, a construction shows that the posterior predictive alone cannot determine IG, so any next-frame predictor trained on interaction histories needs additional structure from the simulator. Second, the authors propose Double Take. A world model is trained to predict one rollout given a second rollout from the same world as support. At scoring time, the just-observed transition is disguised as part of the support history, and the reward measures how much more likely the observation becomes. This reward equals the order-2 Rényi divergence between two double-counted posteriors, and is zero iff IG is zero. Third, the authors introduce a suite of Craftax-based environments that isolate distinct exploration challenges, and measure gathered world information in bits relative to an ideal Bayesian observer.
Double Take outperforms RND, ICM, E3B, E3B×RND, BYOL-Explore, and BYOL-Hindsight across the suite. It matches an oracle PPO agent on DualLab, LilyPond, and BallPit. On LabTower it collects ~1,067 bits, versus at most ~12 for the baselines. It visits more CraftCamp rooms than both the oracle and the strongest baseline, and covers more terrain in Craftax-classic and Craftax-full. After pretraining on small procedural mazes, it transfers best to out-of-distribution maze families up to 201×201, with an advantage that grows with maze size.
Critical strengths
- Clean, principled idea.
The paper reduces information gain to quantities a standard autoregressive world model can compute. It gives an exact identity between the reward and a Rényi-2 divergence, and a non-identifiability argument for why paired same-world rollouts are needed. Together these give a coherent account of how the method handles stochastic distractors without heuristic latent-space design.
- Targeted evaluation with convincing results.
Information is measured in bits against an ideal Bayesian observer, not through coverage proxies. Each environment isolates a known failure mode of curiosity methods: static noise, interactive noise, long horizons, and sparse information. Gains are largest exactly where baselines fail (BallPit, LabTower). In OOD transfer, the advantage grows with maze size.
Critical weaknesses
- Gap between theory and the practical estimator.
The paper only shows DT and IG agree in sign (zero iff zero), not in magnitude or ranking. The double-counting approximation (Eq. 5) is not analyzed. The double-take query places the target inside the support history, an input never seen in training, so the Appendix A guarantees do not apply and reward reliability rests on OOD generalization. The claim that a simulator is provably necessary also outpaces the informal argument in Section 3.
- Privileged information and limited scope.
The method requires same-world resets that baselines do not receive. No ablation separates the effect of the objective from the effect of the paired data. Performance on sparse, one-shot information is weak (0.4 vs. 8.3 bits of potion information on CraftCamp). The paper reports that untrained policies attain the highest Craftax achievement scores, which undercuts the claimed link to environment reward.
Questions for authors
- Does the per-step DT reward actually track true information gain, especially on noise?
The theoretical guarantees in Appendix A (DT = D₂ ≥ 0, and DT = 0 iff IG = 0) hold only if q is the exact double-counted posterior predictive. However, the double-take query q(o | h₁=(h,a), h₂=(h,a,o)) places the query’s own target inside the support history. This configuration never occurs during training: self-prefix supports stop before the target, and peer supports are independent rollouts that essentially never coincide with h₁. The authors acknowledge that this input is out-of-distribution, so the guarantees do not carry over to the trained model.
One concrete worry is that the model learns to copy aligned support content, which would assign spurious reward to noise that happens to match at scoring time. The end-to-end results in Table 1 suggest the reward behaves sensibly, but they do not demonstrate this directly. Since the true per-step IG is exactly computable in DualLab, LilyPond, and BallPit, the paper would be considerably strengthened by evidence that:
- DT is positively related to the true per-step IG.
- DT stays close to zero on stochastic elements such as the pond and monster dynamics, rather than only being outweighed by reward from maze regions.
- The raw DT values in practice respect the non-negativity implied by D₂ ≥ 0, and the [0, 8] clipping is not masking systematic violations.
2 comments
Thanks for the answer, I will remain my score.
DT reward tracks info gain
In Appendix I we confirm that learned DT reward distinguishes positive from null info steps with high accuracy, and generally ranks noisy below informative steps. Appendix A now also analyzes the double-counting approximation (Eq. 7, formerly Eq. 5): With only “hard” evidence, counting the history twice leaves the posterior unchanged, so an ideal predictor's DT equals IG exactly (Eq. 20).
In more stochastic scenarios, an ideal DT need not preserve IG’s ranking of informative steps, though DT's learned reward correlates with IG more than any baseline’s (Appendix I).
A caveat is sparse, one-shot information that the world model fails to learn in the first place, such as AlchemyLab’s potion effects (Appendix H). In Appendix F.3 we explore one way of addressing this problem by augmenting shared-world data.
In Appendix I, we see that clipping does not mask systematic violations. Raw scores often dip below zero, but only slightly: 97–99% of negative scores are within 0.01 bits of zero, and together they amount to 0.5–6.2% of positive score mass. A small positive background reward remains on uninformative steps, but the reward still tells information-revealing steps apart from zero-information ones.
Training for scoring time
We’ve clarified that the scoring situation is not strictly out-of-distribution, but it is an outlier in complex environments. In Appendix F we study a number of implementation techniques that mitigate this by changes to the structure of the same-world pairing during training.
We study this in Appendix J, and find copying to be near-zero for DT and its peer-prefix variant, unlike forked variants, for which copying some transitory state is in fact optimal.
Shared-world structure
We’ve added a formal proof of the non-identifiability of distinct environment generators under generic next-frame prediction in Lemma 1 and clarified the text.
Baselines also train on same-world rollout pairs, but are blind to their structural relationship. Similarly depriving DT of this information by scrambling support worlds hurts its ability to reveal procedural information (Appendix F.1).
Agent performance and behavior
DT collects more CraftCamp potion information than E3B×RND, the strongest overall baseline (Appendix D.2, Table 4).
In general, some performance gaps remain between intrinsic RL approaches and oracle agents trained on ground truth procedural information, but DT is the strongest learned approach overall across domains (Table 1).
We’ve updated the text to clarify that information gain is our core success metric. Incidentally, in Appendix G we evaluate DT in several new domains in which information gain itself appears more aligned with extrinsic reward (Tetris, XLand-MiniGrid, CoinRun).