acceptodds
Under review as a conference paper at ICLR 2027

Considering Context: When World Models Need Context Encoders

Abstract

Methods for generalization in model-based reinforcement learning typically assume that an agent cannot recover the latent context governing the environment dynamics from its own experience, and therefore supplies it externally. We formalize and test this assumption with predictive sufficiency, which quantifies what access to the context adds to next-step prediction under the visitation distribution an agent induces, and separates that quantity into a history-recoverable part, a residual requiring the true context, and the deficit added by a finite model. We classify context-aware algorithms by the predictive risk their conditioning set can target and demonstrate across environments of increasing identification difficulty that the headroom does not follow the MDP class. The same task under different priors leaves predictive headroom in one setting and nothing distinguishable from zero in another, where the agent's behavior implicitly identifies the context and any benefit of such a mechanism cannot be attributed to missing information. Where headroom persists, the learned state exposes it only partially, and adding the true context still lowers the risk. Our contribution is a practical criterion for matching contextual mechanisms to the information available to them, estimated from the ordinary trained agent without a reference policy.

Primary area:
reinforcement learning
Keywords:
model-based reinforcement learning, contextual reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_osotOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary
  • The paper quantifies how much directly observing the context improves next-state prediction beyond only conditioning on the current state, the entire history, or a subset between the two. It formalises predictive sufficiency, which simply asks the model to preserve transition information rather than the context itself.
  • The main point is that headroom (the potential improvement in next-state prediction from observing the true context under the agent’s observed trajectories) can collapse simply because the agent’s visitation reveals the true context. If headroom remains, this can either be due to finite model deficit, or because the headroom cannot be closed by history, and so requires the true context.
  • Unlike prior work, which asks how to infer and adapt to hidden context gained from experience, this paper asks whether the information supplied to the inference mechanism actually contains useful information that is not already available to the predictor.
Critical strengths
  • Clean conceptual framework: unlike prior work, which asks how to infer and adapt to hidden context gained from experience, this paper asks whether the information supplied to the inference mechanism actually contains useful information that is not already available to the predictor.
  • Experimental design: the experimental design is good and interesting; I particularly like Table 2 which clearly shows the connection to related work.
  • Implication for downstream context-inference benchmarks: simply introducing a hidden dynamics parameter does not automatically create a meaningful context-inference benchmark.
Critical weaknesses
  • Presentation and Clarity: I think that both of these could be improved. While the theoretical framework is very interesting, the paper could be made much easier to follow. Firstly, I think that the introduction could be made less abstract and verbose. The theoretical framework section could be restructured so that the paragraph headings are consistent (e.g., organising the paragraphs around the hierarchy of conditioning information), rather than some introducing assumptions, others new quantities or ideas. Figure 1 is not very clear (I understood it once reading the paper, but I think that it could either be improved or removed).
  • Practical utility: the theoretical framework uses Bayes-optimal risks, but the practitioner only has finite fitted probes. This means that if, for example, a history probe does not improve prediction to the same level as the oracle (e.g., as required for P2), we do not know whether the history contains no useful additional information, or whether the probe has simply failed to extract that useful information. The oracle itself is also fitted, which can cause R(qO)>RO\mathcal{R}(q_O)>\mathcal{R}_O
Questions for authors
  • The abstract describes the main contribution as a practical criterion for matching contextual mechanisms to the information available to them, but in practice (please correct me if I am wrong) this criterion is implemented through differences in held-out losses of finite probes conditioned on different information sets. How should a practitioner distinguish absence of predictive information in a conditioning set from failure of the corresponding finite probe to extract that information (i.e. if R(qH)>R(qO)\mathcal{R}(q_H) > \mathcal{R}(q_O) how do we know if RH>RO\mathcal{R}_H > \mathcal{R}_O (history genuinely lacks information), or RH=RO\mathcal{R}_H = \mathcal{R}_O (the information is present, but the finite probe failed to extract it)?
  • Since estimating headroom requires ground-truth context labels, does this mean that the criterion is only applicable in simulation? Is this only intended as a benchmark-design diagnostic?
3 comments

We thank the reviewer for the time dedicated to reviewing our paper and for recognizing our conceptual framework, along with its experimental support and downstream implications. We address the remaining raised points below:

W1 - On improving paper's presentation

We are thankful for those suggestions. In the revised manuscript, we will re-organize Sec 3 to preview the hierarchy presented in Table 2, and improve Fig 1 so its panels state the two mechanisms directly. We believe that an illustrative cartoon in Fig 1 has its value in communicating the motivating idea before diving into theory, so we will commit to improving rather than removing it.

W2 - On using practical finite probes for assessing Bayes-optimal risks

We agree with the observation that the inference from probes is one-sided, and we will make this notion more clear in the revision. In essence, a history probe that matches the oracle probe justifies that history carries the information, up to the oracle probe's own fitting error, that our calibration study characterizes. Moreover, the probes are used comparatively rather than as point estimates of Bayes risks. The quantity that bounds the residual from above with the probes already fitted is the gap between the best finite-family history probe and the oracle probe. In revision we will report it together with history recovery and the permutation null, so that a reader can see where this one-sidedness binds.

Q1 - On separating absence of the predictive information from failure of the finite probes

Our criterion is indeed implemented through held-out losses of finite probes. We'd like to highlight however that this is the direction in which finite probes are informative, because no difference shows that history carries the information, while an open gap is an upper bound on the deficit, since the history probe's error can only increase it. As such, the criterion's decisions follow from gaps being small, so they depend on the bound rather than on the exact value. While an open gap can't distinguish between the cases, a practitioner can then enlarge the history probe's family (i.e., the finite-family reference of Sec 5) to see whether the gap closes, which changes only the history probe's error. In revision, we will make the bound interpretation explicit and also exemplify it by reporting the corresponding quantities as per the W2 answer.

Q2 - On estimating headroom without ground-truth labels

We note that logged context is needed only for the oracle quantities, such as the probe headroom, residual, decodability, and the null. At the same time, history recovery, the finite-model deficit, and the drift metrics are computed from the agent's own data without labels. As such, they answer whether history carries predictive information under the trained agent's visitation, which is the part of what our criterion asks. The diagnostic is therefore applicable wherever a context label can be logged, which includes not only benchmarks, but also simulators and similar systems. At the same time, its label-free part applies everywhere. We appreciate the opportunity to clarify this distinction and we will make those aspects explicit in Sec 5 in the revision.

We thank the reviewer again for their careful reading and the opportunity to improve our paper, and remain happy to answer any follow-up questions.

Thank you very much for your response. I wonder whether the authors might consider framing the paper more explicitly as a theoretical contribution, with the practical diagnostic presented as an illustration of the framework and its limitations.

Whether I increase my score depends on seeing the revisions to the manuscript regarding scope and clarity. I am concerned that, without either a convincing practical contribution or a clearly communicated theoretical framework, the paper's impact may be limited.

W1 Please could you update the manuscript so that I can see the changes to Figure 1 and the reorganised Section 3?

Q1 I do understand the value of the idea, but I still feel that the proposed solution of enlarging the history probe family does not fundamentally resolve the ambiguity raised in my original question: if the gap remains open, it is still unclear whether history lacks predictive information or whether the finite probe fails to extract it.

Q2 I understand that some components of the diagnostic can be applied without context labels, but my concern remains: fully determining whether a contextual mechanism has additional predictive information to exploit still requires access to ground-truth context. I think it would be beneficial to clarify this limitation and provide concrete examples of real-world applications in which the full criterion could be used.

Q3 Reading through the paper again, I have an additional question regarding the claimed upper bound on the residual. Writing the fitted risks as R^H=RH+eH\widehat R_H=R_H+e_H and R^O=RO+eO\widehat R_O=R_O+e_O, the observed gap satisfies R^H−R^O=Ihist+eH−eO\widehat R_H-\widehat R_O=I_{\mathrm{hist}}+e_H-e_O, and therefore only upper-bounds the residual when eH≥eOe_H\geq e_O, which does not appear to be guaranteed. What justifies this upper-bound interpretation? More generally, how can we distinguish differences in the predictive information available to the probes from differences arising from their respective function approximation, estimation, or optimisation errors?

We thank the reviewer for engaging with our rebuttal and for the follow-up questions. The uploaded PDF incorporates the agreed restructuring, addresses the scope and clarity points, as well as positions our work as an analytics contribution with practical illustration. On the remaining points:

Q1

We agree with the point that a larger history probe family doesn't fully solve the raised ambiguity, because decisions follow from closed gaps and an open gap is a bound, both up to eOe_O, whereas enlarging the family only changes eHe_H. However, we note that since RH≤RMR_H\le R_M, a history probe whose held-out risk separates above the memoryless probe's indicates eH>eMe_H>e_M without θ\theta. The table below illustrates this in the identifiable cRSSM-D row. In revised Sec 5 we state the decision rule linking these measurements to the criterion's branches.

Q2

Indeed, full determination requires labels in general, since headroom is a functional of the mixing distribution over kernels, which unlabeled trajectories need not identify (finite-state mixtures under strong assumptions are a special case outside our setting). For instance, low headroom follows from weak kernel separation in our Resistance and from visitation-mediated identification in Wind, and Sec 4 separates those two with θ\theta. Also, in the case of single-step episodes, one kernel (zero headroom) is indistinguishable from a mixing distribution over distinct kernels (headroom > 0). However, for Bayes-optimal predictors the label-free history gap still lower-bounds headroom and is tight when the episode's past determines the kernel at (st,at)(s_t,a_t).

As to concrete examples, the full criterion applies in settings where world modeling is typically used and the context is available per episode. For instance, in real-life physical systems such as robots with measured payload and in simulated physical systems such as games, where θ\theta is known by design. Furthermore, gaming setups have applications for non-physical systems, where the transition kernel captures the underlying game mechanics and parameters, such as level difficulty, are logged in telemetry. Other randomized simulators are applicable as well.

Q3

Sec 5 states Gθ=Iemp+eM−eOG_\theta=I_{\mathrm{emp}}+e_M-e_O. Our earlier replies clarify both error terms separately: the history-to-Oracle match holds up to the Oracle probe's own fitting error eOe_O, and the history probe's error eHe_H can only widen the gap. The bound doesn't require the eH≥eOe_H\ge e_O assumption: with QXQ_X the fitted probes' population risks and eX=QX−RX≥0e_X=Q_X-\mathcal R_X\ge0, Ihist≤QH−QO+eOI_{\mathrm{hist}}\le Q_H-Q_O+e_O, and a closed population gap gives Ihist≤eOI_{\mathrm{hist}}\le e_O. The table below reports retained held-out minima, which is distinct from the validation-selected history-only gap. In the calibration rows the fitted residual exceeds the exact filter residual by 0.115±0.0250.115\pm0.025 in that environment's units. The equal-information pairs show how much fitting error alone contributes, but they do not pin down eOe_O by itself. The calibration rows give a direct comparison of the fitted gap with the closed-form reference, which shows how much the probes attenuate it.

CellMethodHeadroom R^M−R^O\widehat R_M-\widehat R_ONull R^M−R^perm\widehat R_M-\widehat R_{\mathrm{perm}} (θ\theta permuted across episodes)R^rec−R^M\widehat R_{\mathrm{rec}}-\widehat R_MMin over window, learned state, learned state + window, recurrent probes −R^O-\widehat R_O
IdentifiablecRSSM-D−0.14±0.16-0.14\pm0.16−0.05±0.18-0.05\pm0.180.82±0.200.82\pm0.20−0.01±0.03-0.01\pm0.03
HardDALI-S2.10±1.762.10\pm1.76−0.01±0.24-0.01\pm0.240.19±0.490.19\pm0.490.62±0.230.62\pm0.23
Calibrationexact2.51±0.052.51\pm0.0500−2.49±0.03-2.49\pm0.030.0170.017
Calibrationfitted probes1.20±0.061.20\pm0.06−0.04±0.02-0.04\pm0.02−1.06±0.06-1.06\pm0.060.132±0.0250.132\pm0.025
Reviewer_oApVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper is an analysis/diagnostic paper. This paper asks when a world model really needs a context encoder in contextual MDPs. From my understanding, the main idea is: the agent’s own trajectory may already reveal the hidden context. If so, a context encoder adds no new information. To explore this, the paper separates structural variation (I_struct) from predictive headroom (I_emp). The paper also split headroom into a part that history can recover and a residual that needs the true context. Experiments include a controlled diagnostics (MiniGrid, Point-Mass) with known kernels, as well as CARL/RWRL Walker with DreamerV3-based methods, measured by probes.

Critical strengths
  • Clear and useful distinction between I_struct and I_emp. The idea that “the agent’s behavior already does context inference” is interesting.
  • Section 4 is well designed. The matched-query quantity separates two effects, and the reset interventions show which state variables carry the context.
  • Honest discussion of limitations. Code is provided.
Critical weaknesses
  1. Prediction accuracy is not linked to policy quality. The paper presents its measure as a practical criterion for deciding whether a context module is needed. But people care about control, and the results show that headroom and return do not match. In the Identifiable setting, headroom is about zero, yet cRSSM-D gets clearly higher return (862 vs. 712). In the Hard setting, headroom is large, yet all returns are within about 2%. So the criterion does not tell us whether context will help the agent act better.

Several points:

  • Prop. 2 bounds value error with a supremum over all (s, a), while I_emp is an average under visitation. A small average does not imply a small worst case, so Prop. 2 does not connect the measured quantity to return.
  • Low headroom can mean two different things: the state reveals the context, or the policy simply avoids context-sensitive regions (like the agent along the wall in Fig. 1c). The paper does not separate these cases for trained agents.
  • Dreamer trains its actor on 15-step imagined rollouts that partly leave the data distribution. One-step error on the agent’s own data may not be the quantity that matters.
  1. Limited theoretical novelty. I_emp is a conditional mutual information. Prop. 1 follows from the entropy chain rule (the proof is one line), and Prop. 2 is a standard simulation lemma. The idea that history contains information about the context is well known from Bayes-adaptive MDPs and POMDPs.
  2. The criterion is post hoc. It requires a trained agent and depends on its policy, so it cannot guide design choices before training. If one has to train the agent anyway, directly comparing returns is simpler.
Questions for authors
  • Can you show any correlation between headroom (or residual context gain) and return differences? If not, in what sense is the criterion practical?
  • Is low headroom caused by the state identifying θ\theta, or by the policy avoiding context-sensitive regions? Could you measure headroom under a fixed exploratory policy or on imagined rollout states?
  • Why only one-step, on-policy prediction? What does headroom look like for multi-step or imagined rollouts?
  • How does this work relate to value equivalence and objective mismatch?

[1]. Lambert, N., Amos, B., Yadan, O., & Calandra, R. . Objective Mismatch in Model-based Reinforcement Learning.

[2]. Grimm, C., Barreto, A., Singh, S., & Silver, D. The Value Equivalence Principle for Model-Based Reinforcement Learning.

3 comments

We thank the reviewer for their constructive reading and for recognizing our central idea of visitation-mediated identification, supported by the analysis and controlled diagnostics. Below we address the raised questions and outline the planned revision.

Q1 - Correlation between the headroom and return

We agree that the present results do not support such a correlation and we report this absence as a result. However, our criterion is useful since it can be computed from the context-blind agent before any context mechanism is built, and it answers what return comparison doesn't, namely whether a mechanism has predictive information to add at all. Zero headroom rules out that any context pathway supplies missing information to the model, so a return gain from one has to come from elsewhere, while open headroom with matched returns shows control succeeded without resolving the context. In revision, we will make this scope more prominent in the introduction.

Q2 - State identifying context vs. the policy avoiding certain regions

We have tested this hypothesis directly on the identifiable Walker setting. Here, we reconstruct the simulator state from the agent episodes, fix the wind to each grid value, replay the recorded action, and measure the spread of the next states:

VisitationTransitionsMedian spread / p90Steps < 25Steps >= 25
Trained450000.068 / 0.1490.0670.068
Random320000.075 / 0.1280.0760.075

The wind moves the next state at the visited states by ~2.5% of the typical one-step state change, and under a random policy the spread is within 10% of the trained. So the trained policy didn't move the agent toward states where the wind matters less, ruling out avoidance. With the wind decodable from the current (s,a)(s, a) at R2≥0.93R^2 \ge 0.93, the remaining account is the state carrying the wind, as P1 predicts. The revision will report those results in Sec 6.

Q3 - Multi-step and imagined rollouts

We focus on one-step prediction as the quantity world models are trained on and what the context pathway changes, as our probes estimate. Under the conditional Markov, the information that context adds about a kk-step rollout is a sum of 1-step terms conditioned on the rollout prefix, each bounded by the one-step headroom at the corresponding (s,a)(s, a) under the rollout's own context weights. Hence, on the measured visitation it vanishes if the one-step quantity is zero at every step. One-step headroom doesn't capture the compounding of finite-model error along imagined rollouts, which is a property of the learned model, and imagined rollouts also leave the measured support. We focus on on-policy because the agent trains on its own visitation and no reference policy is neutral. We will add this clarification to Sec 3.

Q4 - Value equivalence and objective mismatch

We appreciate those references. Compared to [1], Table 4 in our paper states mismatch estimates at the Bayes level, with finite-model error removed from the predictive side. Compared to [2], exact predictive sufficiency is value equivalence w.r.t. all functions on the visited support, and the approximate versions relax different axes. [2] also explains the hard settings, but not the identifiable settings, where the two predictors are equivalent on the support, so that gain has to come either from OOD states, the finite model, or the policy's use of the context. Furthermore, our study analyzes which conditioning set is sufficient to match the true model, which is not addressed in prior work.

W1 - Prediction accuracy and the policy quality

We agree with the observation that headroom and return do not generally track each other. As noted in Sec 3, whether information improves control is outside our focus, and the question we aim to answer is attribution: e.g., the gain of cRSSM-D can't come from information missing to the transition model, since there is none. We will make those connections more explicit, and further clarify in what sense a contextual mechanism is "needed" within the scope.

W2 - On novelty of Prop 1 & 2

Prop 1 & 2 are not intended as central theoretical contributions of our work. Prop 1 is a necessary step needed to order conditioning sets and define the residual gain, whereas appendix's Prop 2 is a standard simulation lemma that motivates evaluating return separately from the headroom. We also agree that its supremum bound doesn't connect a visitation-averaged quantity to return, so we will clarify its reference accordingly.

W3 - Usefulness of the criterion

Indeed, the criterion needs a trained agent, but only the context-blind baseline that practitioners build anyway, after which the probes apply before any context mechanism is trained. Comparing returns requires training every candidate and doesn't say whether a gain came from information or optimization, as discussed in our Q1 answer.

We thank the reviewer for the detailed comments and will gladly address any further questions.

Thanks for the clarifications, which address my concern in Walker. I understand headroom as a predictive attribution diagnostic, not a predictor of return gains. Since my concern about its control-oriented practicality remains, I will keep my original score.

We thank the reviewer for the follow-up and for confirming that their Walker concern is addressed. We also agree with the interpretation of headroom as a predictive diagnostic, rather than a control mechanism, that's practical in a sense that it informs whether a contextual pathway has information to add at all. As mentioned in our Q1 answer, the revision will elaborate this scope upfront, so our contribution is not read as a claim about return gains, as per Sec 6.

Reviewer_9fBvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper provides a framework for assessing the predictive information available to context mechanisms in world models. For latent contexts fixed within an episode, it defines “predictive headroom” as the reduction in Bayes-optimal one-step log loss from revealing the true context to a predictor conditioned on the current state and action, under the agent's visitation distribution. It separates this headroom into information recoverable from history and a residual requiring the true context, while distinguishing both from finite-model error. “Predictive sufficiency” concerns preserving context information relevant to transitions without requiring recovery of the environment parameters themselves.

Controlled experiments with known transition kernels show how substantial structural variation can coexist with little predictive headroom when the current state already carries context information. Experiments on a common DreamerV3 backbone examine mechanisms using within-episode and cross-episode information, reporting predictive metrics and return. The resulting diagnostics complement an environment's classification by assessing the information available under the trained agent's visitation.

Critical strengths
  • S1. The paper usefully distinguishes variation in environment parameters from additional predictive information beyond the current state and action (Eqs. 3–4). This motivates the recommendation to report estimated predictive headroom alongside an environment's classification (Section 7, p. 9). Accounting for the information available under the agent's policy should help guide experiment design.

  • S2. The controlled diagnostics help explain how current-state information reduces predictive headroom. Structural variation and headroom under visitation differ substantially in some settings (Table 1). In Point-Mass-Attractor, resetting both position and velocity raises estimated headroom, while preserving velocity keeps it near baseline (Figure 5(d), p. 16). This control supports the explanation that velocity already carries context information relevant to prediction.

  • S3. The analysis distinguishes information recoverable from history from estimation, approximation, and optimization errors in finite models (Proposition 1 and Eq. 8), and extends the information decomposition to correlated episodes (Eq. 9).

    It also treats the effect on control as a separate empirical question (discussion following Definition 1, p. 5). Reporting both predictive metrics and return helps distinguish improved transition prediction from improved control (Table 4).

Critical weaknesses
  • W1. The paper characterizes the predictive information available to each mechanism, but the empirical link between that information and the mechanisms' gains remains limited. For example, it relates the LILAC-style agent's gains in Drift to its prediction about cross-episode information (Section 6, p. 9). Although the agents share a DreamerV3 backbone, the comparisons also reflect differences in learning and visitation (Appendix C, p. 16; Section 5, p. 8), leaving the contribution of additional information unclear.

    State interventions and controls with independent context draws support the information analysis (Appendix B, pp. 15–16; Appendix D, p. 18), but do not directly establish what drives the Walker gains. Evaluating each method under its own visitation is deliberate (Section 7, p. 9); the concern is how far these results explain the mechanisms' gains.

Questions for authors
  • Q1 (related to W1). To what extent do the existing comparisons and controls support interpreting the LILAC-style agent's lower mean startup losses at the high drift rate as a benefit of using information from previous episodes (Table 5; Section 6, p. 9)?
2 comments

We are grateful for the reviewer's positive feedback and thoughtful comments regarding our manuscript. We particularly appreciate the recognition of our analysis and its applicability in classifying environments for guiding downstream usage. Below, we address the remaining questions:

W1 - Regarding the empirical link between the information and agent's gain

We thank the reviewer for the opportunity to clarify this point. The Table 5 comparison is made under each method's own visitation, so a difference between methods indeed reflects information / learning / visitation together, as noted in Sec 7. We use within-method probes as the tool for locating information and reporting it per method. Whereas cross-method comparison relies on return and the class-level prediction, as Sec 5 states. The drift result can be read in that sense. In revision, we will make this distinction explicit where Sec 6 and the Discussion interpret the results, and add method differences with their standard errors to Table 5.

Q1 - Regarding LILAC results interpretation

We appreciate the question which helps us state the extent precisely. As reported in Table 5, the chain-latent agent has the lowest start transient and start prior loss at the high rate, and P3 predicts a cross-episode mechanism to have an advantage at episode start, so the reported ordering is consistent with P3. Since the comparison is performed under each method's own visitation, Table 5 doesn't directly separate cross-episode information from that agent's within-episode prediction quality at the corresponding rate. In revision, we will clarify this point in Sec 6, where it refers to P3.

We thank the reviewer again for the opportunity to improve our paper and remain available to answer any follow-up questions.

Thank you for clarifying that the results in Table 5 are consistent with P3, rather than identifying the contribution of additional information. Although the causes of the gains remain uncertain, the state interventions in the other controlled experiments provide concrete evidence for the role of context information. I therefore continue to value the paper’s contribution to understanding the information available to context mechanisms.

I also share the other reviewer’s interest in a clearer explanation of how the practical criterion (stated in absrtact) connects the measurements to decisions. (that's is the reason why I could not raise my overall score 3 to 4 and becasue confidence 2, i.e. not specialist of practice.) I will follow the discussion in that review thread!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.