acceptodds
Under review as a conference paper at ICLR 2027

Double Take: Intrinsically Motivating Agents to Model Many Worlds

Abstract

We consider the problem of an agent with prior beliefs over a distribution of possible worlds, who wants to infer the identity of the new world in which it finds itself at runtime. The key is to take the right kind of exploratory actions. Supposing the agent has some implicit belief about the current world, we show that a surrogate for information gain can be computed by training a model to predict multiple possible outcomes of an action given some observed history in the world, then asking the model how much more likely an outcome becomes after observing it once. The benefit of such a "double take" is that first looks are informative only insofar as they reveal the hidden structure of the environment; by construction, they cannot help predict noisy TVs. However, assuming a black-box environment, we cannot sample alternate outcomes conditioned only on an agent's observations if the environment includes noisy hidden state. To get around this, we instead perform an entire "second take" of agent interaction in the same world, and train a model to better predict the second take using information from the first. This teaches the model to transmit shared-world structure and discard rollout-specific transients. We can then simulate the effect of an instantaneous double take by replaying the first look to the model as a transition within a supposed first take, and having it make new predictions using the shared-world structure this disguised observation confers. We use procedurally-generated reinforcement learning environments to model the problem of information gathering over many worlds, measuring the amount of information gathered about the world by the resultant agents using an ideal Bayesian observer. Our methods outperform intrinsic-RL baselines in terms of world-information-gathering, and often discover behavior correlating with high environment reward. They may also provide a general recipe for collecting relevant data for training interactive next-frame-prediction world models.

Primary area:
reinforcement learning
Keywords:
active learning, world models, reinforcement learning, procedural content generation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8HLYOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies intrinsic motivation for an agent that must identify which world it inhabits, where worlds are sampled from a known distribution and formalized as controlled hidden Markov models with unknown parameters θ. Information gain (IG) is the natural objective: it can be written as the gap between a “first take” (surprise under the prior predictive) and a “double take” (expected surprise under the posterior after observing the outcome), and it ignores noisy TVs by construction. However, computing it requires an explicit posterior over θ, which autoregressive world models do not maintain.

The paper makes three contributions. First, a construction shows that the posterior predictive alone cannot determine IG, so any next-frame predictor trained on interaction histories needs additional structure from the simulator. Second, the authors propose Double Take. A world model is trained to predict one rollout given a second rollout from the same world as support. At scoring time, the just-observed transition is disguised as part of the support history, and the reward measures how much more likely the observation becomes. This reward equals the order-2 Rényi divergence between two double-counted posteriors, and is zero iff IG is zero. Third, the authors introduce a suite of Craftax-based environments that isolate distinct exploration challenges, and measure gathered world information in bits relative to an ideal Bayesian observer.

Double Take outperforms RND, ICM, E3B, E3B×RND, BYOL-Explore, and BYOL-Hindsight across the suite. It matches an oracle PPO agent on DualLab, LilyPond, and BallPit. On LabTower it collects ~1,067 bits, versus at most ~12 for the baselines. It visits more CraftCamp rooms than both the oracle and the strongest baseline, and covers more terrain in Craftax-classic and Craftax-full. After pretraining on small procedural mazes, it transfers best to out-of-distribution maze families up to 201×201, with an advantage that grows with maze size.

Critical strengths
  1. Clean, principled idea.

The paper reduces information gain to quantities a standard autoregressive world model can compute. It gives an exact identity between the reward and a Rényi-2 divergence, and a non-identifiability argument for why paired same-world rollouts are needed. Together these give a coherent account of how the method handles stochastic distractors without heuristic latent-space design.

  1. Targeted evaluation with convincing results.

Information is measured in bits against an ideal Bayesian observer, not through coverage proxies. Each environment isolates a known failure mode of curiosity methods: static noise, interactive noise, long horizons, and sparse information. Gains are largest exactly where baselines fail (BallPit, LabTower). In OOD transfer, the advantage grows with maze size.

Critical weaknesses
  1. Gap between theory and the practical estimator.

The paper only shows DT and IG agree in sign (zero iff zero), not in magnitude or ranking. The double-counting approximation (Eq. 5) is not analyzed. The double-take query places the target inside the support history, an input never seen in training, so the Appendix A guarantees do not apply and reward reliability rests on OOD generalization. The claim that a simulator is provably necessary also outpaces the informal argument in Section 3.

  1. Privileged information and limited scope.

The method requires same-world resets that baselines do not receive. No ablation separates the effect of the objective from the effect of the paired data. Performance on sparse, one-shot information is weak (0.4 vs. 8.3 bits of potion information on CraftCamp). The paper reports that untrained policies attain the highest Craftax achievement scores, which undercuts the claimed link to environment reward.

Questions for authors
  1. Does the per-step DT reward actually track true information gain, especially on noise?

The theoretical guarantees in Appendix A (DT = D₂ ≥ 0, and DT = 0 iff IG = 0) hold only if q is the exact double-counted posterior predictive. However, the double-take query q(o | h₁=(h,a), h₂=(h,a,o)) places the query’s own target inside the support history. This configuration never occurs during training: self-prefix supports stop before the target, and peer supports are independent rollouts that essentially never coincide with h₁. The authors acknowledge that this input is out-of-distribution, so the guarantees do not carry over to the trained model.

One concrete worry is that the model learns to copy aligned support content, which would assign spurious reward to noise that happens to match at scoring time. The end-to-end results in Table 1 suggest the reward behaves sensibly, but they do not demonstrate this directly. Since the true per-step IG is exactly computable in DualLab, LilyPond, and BallPit, the paper would be considerably strengthened by evidence that:

  • DT is positively related to the true per-step IG.
  • DT stays close to zero on stochastic elements such as the pond and monster dynamics, rather than only being outweighed by reward from maze regions.
  • The raw DT values in practice respect the non-negativity implied by D₂ ≥ 0, and the [0, 8] clipping is not masking systematic violations.
2 comments

DT reward tracks info gain

Does the per-step DT reward actually track true information gain, especially on noise?

In Appendix I we confirm that learned DT reward distinguishes positive from null info steps with high accuracy, and generally ranks noisy below informative steps. Appendix A now also analyzes the double-counting approximation (Eq. 7, formerly Eq. 5): With only “hard” evidence, counting the history twice leaves the posterior unchanged, so an ideal predictor's DT equals IG exactly (Eq. 20).

In more stochastic scenarios, an ideal DT need not preserve IG’s ranking of informative steps, though DT's learned reward correlates with IG more than any baseline’s (Appendix I).

A caveat is sparse, one-shot information that the world model fails to learn in the first place, such as AlchemyLab’s potion effects (Appendix H). In Appendix F.3 we explore one way of addressing this problem by augmenting shared-world data.

The raw DT values in practice respect the non-negativity implied by D₂ ≥ 0, and the [0, 8] clipping is not masking systematic violations.

In Appendix I, we see that clipping does not mask systematic violations. Raw scores often dip below zero, but only slightly: 97–99% of negative scores are within 0.01 bits of zero, and together they amount to 0.5–6.2% of positive score mass. A small positive background reward remains on uninformative steps, but the reward still tells information-revealing steps apart from zero-information ones.

Training for scoring time

The double-take query places the target inside the support history, an input never seen in training.

We’ve clarified that the scoring situation is not strictly out-of-distribution, but it is an outlier in complex environments. In Appendix F we study a number of implementation techniques that mitigate this by changes to the structure of the same-world pairing during training.

One concrete worry is that the model learns to copy aligned support content, which would assign spurious reward to noise that happens to match at scoring time.

We study this in Appendix J, and find copying to be near-zero for DT and its peer-prefix variant, unlike forked variants, for which copying some transitory state is in fact optimal.

Shared-world structure

The claim that a simulator is provably necessary also outpaces the informal argument in Section 3.

We’ve added a formal proof of the non-identifiability of distinct environment generators under generic next-frame prediction in Lemma 1 and clarified the text.

The method requires same-world resets that baselines do not receive.

Baselines also train on same-world rollout pairs, but are blind to their structural relationship. Similarly depriving DT of this information by scrambling support worlds hurts its ability to reveal procedural information (Appendix F.1).

Agent performance and behavior

Performance on sparse, one-shot information is weak (0.4 vs. 8.3 bits of potion information on CraftCamp).

DT collects more CraftCamp potion information than E3B×RND, the strongest overall baseline (Appendix D.2, Table 4).

In general, some performance gaps remain between intrinsic RL approaches and oracle agents trained on ground truth procedural information, but DT is the strongest learned approach overall across domains (Table 1).

The paper reports that untrained policies attain the highest Craftax achievement scores.

We’ve updated the text to clarify that information gain is our core success metric. Incidentally, in Appendix G we evaluate DT in several new domains in which information gain itself appears more aligned with extrinsic reward (Tetris, XLand-MiniGrid, CoinRun).

Thanks for the answer, I will remain my score.

Reviewer_sHSGOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper tackles the problem of how agents infer the identity of new worlds they find themselves in. They introduce a model based on the concept of "double-takes" that aims to distinguish the difference in types of surprise -- e.g. events that are useful/teach you something vs events that are just randomness. This is effectively trained by sampling multiple independent visits (two really) and trying to train the model to predict observation given partial information about other visits. They introduce an intrinsic reward that promotes exploration in unfamiliar environments, while ignoring noise. Experiments show strong performance in exploration and info-gathering.

Critical strengths
  • **Strong empirical results showing improved exploration/info-gathering abilities and meaningful contribution. ** The proposed method does well to improve over existing baselines and approach oracle performance, showing the merit of the technique. The empirical setup is well designed, and it achieves the goals set out and demonstrates meaningful contribution to this field. The method of double-take, or using independent trajectories of the same world is clear and effective method for separating relevant information from noise. The deigned reward promotes actionable exploration, which is well demonstrated in the empirical study.
Critical weaknesses
  • Isolating importance of same-world pairing. It would be useful to determine the importance of same-world pairing in the double-take regime, and therefore prove some fundamental assumptions made in the paper.
  • Computational cost. What is the relative cost of this method to baselines, computationally, and would baselines be more competitive if given a similar budget. This would help isolate the value of the method of simply more computation sampling more trajectories say.
Questions for authors
  1. Following the first weakness, what happens when the supporting trajectory is sourced from an new world (assuming other training params are fixed let's say).
  2. What evidence that data collected by double take actually leads to better world models can the authors present, if any? This is a central motivation and it would be useful to include discussion on this.
2 comments

Importance of same-world pairing

what happens when the supporting trajectory is sourced from an new world

In Appendix F.1 we see that information gathering becomes substantially weaker. This empirically illustrates our argument that learning to gather world-information requires training on some shared-world structure.

Relative computational cost

What is the relative cost of this method to baselines, computationally, and would baselines be more competitive if given a similar budget.

In Appendix E, we now measure each method’s training speed on the same H100 and give baselines additional environment steps to match DT’s estimated training time. DT remains strongest overall across domains.

Does DT-collected data produce better world models?

What evidence that data collected by double take actually leads to better world models can the authors present, if any?

We now study this in Appendix H: compared to a shadow world model trained alongside E3B×RND, the DT world model is better in LabTower and CraftCamp at using support rollouts to predict world-information revealed for the first time in the current history.

Thanks for the response, I maintain my rating.

Reviewer_aQsUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes to teach an AI explorer to seek observations that reveal its surroundings while ignoring distractions. the main approach is to train a prediction model using two separates visits to the same underlying world and reward the explorer when seeing a new observation helps the model assign a higher probability to that observation during the second visit. Empirical study shows that the method encourages exploration and transfers to unfamiliar much larger mazes although improvements in control tasks are limited.

Critical strengths

The proposed idea seems to be intuitive and promising to distinguish learnable world structure from unpredictable distractions.

Critical weaknesses
  1. I think a first doubt is that whether there is really a clear cut/distinctions between learnable world structures and distractions. For example, the randomly moving objects could still follow some fixed physics dynamics rules, and they could be still relevant and not really distracting, as the concept of distracting could be highly relevant to the decision task. This might also partially explains that why the control performance gain is limited.

  2. Eq 1 and 2 incorrectly equate the entropy decrease from a particular observation with KL divergence; these agree only after averaging over observations. Even after correcting this, the proposed surrogate can prefer an action with lower true expected information gain, there is no justification on this discrepancy.

  3. There is mismatch between implemented training distribution and the theoretical one. The derivation assumes conditionally independent histories whereas Appendix C trains on a mixture of independent peer trajectories and the agent's own history. i don't see a necessity for such mismatch between theory and experiments design.

  4. Broader usefulness is not convincing: the potion discovery performance in CraftCamp is weak and the absence f downstream world model training experiments make the general dynamics learning benefits unestablished.

  5. The gains are not convincingly separated from differences in training resources, and scalability beyond the structured grid observations remain largely untested.

Questions for authors

see weaknesses.

1 comment

Scaling beyond grids

scalability beyond the structured grid observations remain largely untested.

We now apply Double Take (DT) to new domains including tabletop robotics and platformer games (Appendix G):

  • On a simulated Franka robot arm, DT learns a policy that reveals the hidden physical properties of objects (Appendix G.5).
  • In CoinRun (Cobbe et al., 2020), DT learns policies that control a player in a procedurally-generated side-scrolling platformer, running and jumping in order to discover more of the hidden level layout (Appendix G.3).

Learnable structure and informative stochasticity

whether there is really a clear cut/distinctions between learnable world structures and distractions.

The distinction we make is between stationary vs. transitory dynamics. We address stationary dynamics (how the world works), but don’t target transitory dynamics (what’s the temporary hidden state right now). Intuitively, a biologist cares about how animals work in general, but doesn’t necessarily care about the exact state of every animal at every time. We don’t think there’s a universal right answer here, just different algorithms that solve different problems. DT solves this particular problem.

For example, the randomly moving objects could still follow some fixed physics dynamics rules

Indeed, if the randomly moving objects actually obeyed unknown persistent rules that varied across sampled worlds, then uncovering those rules would be of value, and is rewarded by DT. This is exactly our stationary vs. transitory uncertainty dichotomy.

Appendix G.1 gives new experiments that tease out exactly this distinction.

Formulation of information gain as reward

Eq 1 and 2 incorrectly equate the entropy decrease from a particular observation with KL divergence; these agree only after averaging over observations.

Thanks, we’ve edited these equations to clarify the distinction. In the limit, during training, we end up averaging over all observations.

Even after correcting this, the proposed surrogate can prefer an action with lower true expected information gain, there is no justification on this discrepancy.

Yes, the paper now states this: DT’s proxy is nonzero iff information gain is nonzero, therefore it does not stop exploring until there is no more information to gain (Appendix A, Eqs. 16–19).

Independence of same-world support rollouts during world model training

The derivation assumes conditionally independent histories whereas Appendix C trains on a mixture of independent peer trajectories and the agent's own history.

Empirically, we found that training with a copy of the current history in the support slot helps bring the scoring-time prediction further in distribution. New experiments explore two new alternatives:

  1. Independent same-world support rollouts, exactly matching the theory. This variant retains strong exploration in LabTower, CraftCamp and Craftax, but degrades potion identification in AlchemyLab (Appendix F.2).
  2. “Forking” the rollout a variable number of steps in the past. This leads to much stronger potion information collection in AlchemyLab, but degrades performance in CraftCamp (Appendix F.3).

Compute budget

The gains are not convincingly separated from differences in training resources

The original comparison matches environment frames. We measure each method’s training speed on the same H100 and give baselines additional environment steps to match DT’s estimated training time in Craftax/Lab environments. DT remains strongest overall across domains and no baseline overtakes it in any domain (Appendix E).

World model learning

the absence of downstream world model training experiments make the general dynamics learning benefits unestablished.

Appendix H: we compare DT’s world model with an equivalent model trained online alongside E3B×RND, the strongest baseline. In LabTower and CraftCamp, DT’s model is much better at using a same-world support visit to predict procedural information that the current visit has not yet revealed.

potion discovery performance in CraftCamp is weak

Potion discovery is weak relative to the oracle, but DT exceeds E3B×RND on potion information as well as total world information (Appendix D.2, Table 4). “Forking” history per above also improves potion performance in AlchemyLab (Appendix F.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.