acceptodds
Under review as a conference paper at ICLR 2027

Condition-Dependent Probe Calibration in a World Model’s Own Rollouts

Abstract

Interpretability work on world models reads latents with probes calibrated on encoder outputs, then reuses them on latents the model generates with its own dynamics; read that way, the intervention effects a model imagines have been found not to match the environment’s. We ask whether that is lost information or a change in the measurement’s own calibration, and whether that calibration is stable across rollout conditions, holding a TD-MPC2 model fixed and varying only how the probe is calibrated. The measured physical variables stay linearly recoverable: on perturbed cheetah states, a map from rolled to encoded latents fit without the targets restores an encoder-calibrated readout from to . The map is specific to the condition it was fit in: it keeps only part of its benefit at other horizons, can do worse than no map on another state distribution, and is poorly captured by an orthogonal map, while identity plus a rank-16 correction recovers most of its gain. Recalibration also changes intervention-consistency effect sizes, by up to across four comparisons, without reversing any feature-versus-control separation; in the comparison a post-hoc check finds both readouts above on the corresponding states. In a second architecture, LeWorldModel on Reacher, the encoder-calibrated readout stays almost perfectly accurate on the model’s training distribution, so the large TD-MPC2 mismatch is not universal across the settings tested.

Primary area:
interpretability and explainable AI
Keywords:
world models, mechanistic interpretability, representation probing, latent representations, probe calibration, model-based reinforcement learning, causal interventions

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_tcUROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks why linear probes calibrated on a world model's encoder outputs fail when reused on latents the model produces with its own dynamics: is the physical information lost, or has the probe's calibration changed? Holding a 5M TD-MPC2 model, its states, actions and edits fixed, the authors compare an encoder-calibrated ridge readout with a readout refit on rolled latents, a target-agnostic linear realignment map from rolled to encoded latents, and no-dynamics baselines, on three DeepMind Control tasks, three state distributions and horizons up to ten model steps. On perturbed cheetah states at k = 5 the encoder-calibrated readout falls from R² = 0.910 on encoded latents to 0.038 on rolled latents, while the refit readout and the realignment reach 0.839 and 0.823, which the authors read as a calibration change rather than lost information. They further report that the realignment is condition-specific and better described as identity plus a low-rank correction than as a rotation, that latent cosine cannot certify a readout (supported by a whitened error bound), and that swapping the readout's calibration source changes intervention-consistency (IC) effect sizes on cartpole by 1.35× in a pre-registered comparison (up to 2.6× across four comparisons) without reversing any feature-versus-control separation. A second architecture, LeWorldModel on Reacher, shows no mismatch on its training distribution.

Critical strengths
  1. The experimental practice is careful and unusually transparent. The IC comparison is paired and pre-registered, failed predictions and deviations are reported openly, and a separately pre-registered within-calibration null on disjoint training halves shows that the IC changes exceed refitting noise. All numbers are generated from released artifacts, and values spot-checked against the supplementary result files match the paper. This makes the individual measurements credible, even where I disagree with their interpretation.

  2. The diagnostic decomposition is simple and practically useful. Comparing the encoder-calibrated readout against a refit readout, a target-agnostic realignment and no-dynamics baselines cleanly separates "information lost" from "calibration lost". The paper also shows that a latent cosine of 0.947, above the common 0.9 gate, can coexist with a readout R² of 0.038, while cosine still ranks conditions by transfer gap (Spearman −0.808), which gives practitioners a cheap triage tool.

Critical weaknesses
  1. Limited novelty, by the paper's own account. Probe failure on rolled latents is prior work (Vakalis, 2026; Guei et al., 2024), and Appendix J claims no novelty for repairing a drifted probe with a label-free low-rank map (cf. Dingeto, 2026; Duan, 2026). Proposition 1 is a loose Cauchy–Schwarz bound (median realised-to-bound ratio 3.2×10⁻⁴ to 0.016), and the whitened-versus-raw point was already made for language-model probes (Ying et al., 2026). The remaining novelty, condition dependence and the IC effect, is weakly supported (W2, W3).

  2. The IC evidence is thin. It comes from cartpole alone, the task with the smallest transfer gap, and only one of three pre-registered contrasts has a ratio interval excluding 1 (1.35 [1.02, 1.89]), a bound the authors note would not survive multiplicity correction. The 2.6× headline is a non-pre-registered replication scored on 3 of 5 variables. IC is inherited from an uncited analysis, its magnitude metric is at floor, and nothing shows which calibration is correct on edited latents.

  3. The condition-dependence and "not a rotation" claims are weaker than presented. The paper's own C_roll, fit once on rolled latents pooled over k = 1–10, reaches R² 0.86/0.85/0.85/0.79 at k = 1/3/5/10 on perturbed cheetah (seed 1), close to per-horizon refits (0.91/0.87/0.85/0.81), so one calibration appears to serve all horizons; pooling across state distributions is never tried. The symmetric share is computed on W rather than W − I, and for an orthogonal Q it equals (1 + tr(Q²)/d)/2, which tends to 100% near the identity, so 61–70% does not argue against a rotation. SimNorm's 64 null directions force zero singular values in any ridge map, so only the large singular values (up to 17.4) remain as structural evidence.

  4. Narrow scope; the large effect sits in a constructed off-distribution setting. All main results use one 5M TD-MPC2 on three proprioceptive DMC tasks with 2–3 checkpoints each. Under the planner the gap is ≤ 0.021 up to k = 5 (0.03–0.14 at k = 10), versus 0.65–0.90 on perturbed cheetah, where the torso is drawn over the full circle under noisy or random actions. The LeWorldModel check is a single checkpoint tested in-distribution only.

  5. The presentation obscures the contribution. The text is dense and heavily hedged, and key methodology (IC, gates, map families) sits in appendices or an uncited pipeline. The OpenReview abstract reports only the 2.6× figure, not the pre-registered 1.35×.

Questions for authors

See weakness part.

Reviewer_o93EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper examines whether probes which have been trained on the encoder representations of a learned world model remain reliable when they are used on latents generated by the model's own dynamics. Because previous studies have shown that the effects of an intervention differ when it is imagined as opposed to when it is actually observed, the authors consider whether the poor performance of the probes is caused by a loss of physical information or by a mismatch in probe calibration.

The authors assess TD-MPC2 with respect to the cartpole, walker, and cheetah tasks, comparing encoder-calibrated probes, probes that have been refitted on rollout latents, and linear transformations which map rollout latents back to encoder representations. For perturbed cheetah states, an encoder-calibrated probe attains an R2 of 0.910 when applied to the encoded latents but only 0.038 when applied to the five-step rollout latents; refitting the probe brings the performance up to 0.839, while a target-agnostic linear alignment reaches 0.823. The authors also demonstrate that these transformations are condition-dependent, do not transfer well across different rollout horizons and state distributions, and cannot be properly described as simple rotations.

The study also looks into the impact of recalibration on intervention-consistency measurements when using sparse-autoencoder features on cartpole; altering the probe changes the measured effect sizes and some of the per-direction statistical conclusions, although the separation between feature and control stays positive. A theoretical bound shows that a high latent cosine similarity does not ensure accurate probe transfer. Further experiments with LeWorldModel on Reacher indicate that the calibration failure observed is not universal among the tested settings.

Critical strengths

1. Conducting careful experiments in order to distinguish between information loss and probe calibration failure.

A key strength of the paper is the way in which it examines the reasons why probes fail when following hypothetical trajectories. Since the world model and the basic trajectories are kept constant and only the readout calibration is varied, the authors are able to tell the difference between information that can no longer be recovered and information that is still there but not accessible to the original probe. The results with the cheetah are especially noteworthy, the performance rising from R2 = 0.038 to 0.839 as a result of recalibration. The target-agnostic alignment experiments also offer further evidence that a large amount of the physical information remains linearly accessible. I also welcome the comparison with the baseline cases that use the initial latent states and actions, since this helps avoid the mistake of taking the recovered probe accuracy as proof that the learned dynamics are accurate.

2. The inclusion of careful controls and clear reporting of the dependence of results on conditions and of negative findings.

The study does more than just show that recalibration leads to higher accuracy; it looks into whether the correction carries over to different rollout horizons and state distributions, studies its geometric properties, and assesses its impact on intervention-consistency measures. The finding that a correction developed under one set of conditions can greatly degrade performance under another is valuable for interpretability research. The authors have also incorporated latent-similarity baselines, used multiple checkpoints, introduced a second world-model architecture, and carried out a preregistered intervention analysis. I really commend them for reporting instances when their initial predictions were wrong and for clearly admitting that the most severe calibration failure and the clearest intervention experiment take place in different settings. This level of transparency enhances my confidence in the empirical observations.

Critical weaknesses

1. No compelling evidence exists to show that a severe calibration failure is associated with changes in the mechanistic interpretation when different tasks are involved.

A key reason is that applying encoder-calibrated probes to imagined trajectories may result in incorrect conclusions regarding internal interventions. Yet the most severe calibration failure is observed in the case of perturbed cheetah states, while the primary intervention-consistency analysis is limited to the cartpole scenario, in which the probe-transfer gap is relatively small. The authors admit that the sparse autoencoder for the cheetah fails the pipeline's design checks and thus cannot support a reliable intervention analysis. As a result, there is no experimental situation that at the same time shows a significant calibration collapse and a corresponding change in a well-validated mechanistic interpretation.

Moreover, even though recalibrating affects Cohen’s d and leads to some changes in the significance of the directional claims, none of the comparisons between features and control do so in terms of direction. The biggest reported effect of 2.6x arises from a further comparison that involves a different subset of physical variables, whereas the preregistered comparison shows only a modest 1.35x change. The fact that the measurement is sensitive to probe calibration has thus been established, but the evidence that this makes a material difference to the main scientific conclusion of an interpretability analysis is still limited. It is important to make this distinction since the implications for mechanistic interpretation are a key reason for writing the paper.

2. The degree of incremental novelty and the ability to generalise beyond the specific experimental conditions is limited.

The paper makes extensive use of Vakalis (2026), a study which had already identified discrepancies between the effects of intervention as imagined and as they actually occur in latent world models. The separation of information loss from probe miscalibration, together with the approach of repairing this through target-agnostic linear alignment, is also closely linked to previous work on representation drift and probe staleness, for example that of Dingeto (2026) and Duan (2026). The more original contribution of the present paper is the dependence of calibration on the rollout conditions and its impact on the intervention measurements. Yet these results are mainly shown using a single TD-MPC2 model size across three control tasks and a relatively small number of checkpoints.

The most serious failures occur when the cheetah states are strongly perturbed, while the model’s performance on its normal planner distribution is much better. The extra experiment with Leworldmodel consists solely of testing Reacher on its training distribution and does not exhibit a similar kind of failure. It is therefore still not clear how often serious calibration mismatches occur during actual use of the model or whether the phenomenon extends to other architectures. Furthermore, the linear realignment depends on having paired rolled and encoded latents from observed trajectories and does not offer a correction that can be applied to arbitrary imagined interventions. Because of these limitations, the general importance of the proposed diagnostic framework is reduced.

Questions for authors

Question 1. Connecting severe calibration failures with a mechanistic interpretation (Sections 5.2 and 5.5).

The largest number of failures occur when the cheetah's states are disturbed, but the intervention-consistency analysis is only carried out in the case of the cartpole because the transfer gap in that case is relatively small. Could the authors examine the effects of recalibration on a task or environment which has a large transfer gap and at the same time permits a reliable measurement of the intervention? And if the sparse-autoencoder pipeline cannot be applied to the cheetah or the walker, would it be possible to use a simpler intervention direction that has already been validated or some other intervention metric to establish the connection? Evidence of this sort would greatly strengthen the paper's main argument about the implications for mechanistic interpretation.

Question 2. The practical significance in the case of ordinary model rollouts (Sections 4, 5.1, and 5.6).

How often do serious calibration failures occur when operating under the model's normal planning distribution as compared to in strongly perturbed states? Although the large cheetah mismatch is concentrated in the perturbed case, the planner results indicate much better transfer. Might the authors make clear the implications for the interpretation of normal planning and also offer further evidence based on different architectures or tasks? This would help to determine whether the phenomenon is a common limitation of world-model probing or whether it is mainly a failure that takes place under specific state distributions?

Question 3. Tell the difference between calibration recovery and accurate dynamics (Sections 3.2 and 5.2).

In cheetah states that are disturbed, a linear baseline based on the initial latent state and the action sequence achieves an R2 value of 0.846, which is only slightly higher than the recalibrated rollout readout. To what extent does the successful recovery from the rollout latents show that the world model has preserved a meaningful representation of the predicted future state, rather than one that could likewise be recovered from the initial state and the actions? Might the authors provide a clearer explanation of what their recovery experiments tell us about the learned dynamics as opposed to the accessibility of the physical variables? And what kind of data would be necessary to calibrate a probe for genuinely unobserved counterfactual rollouts?

Reviewer_WD5NOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the reliability of physical probes when transferred from encoded representations to imagined rollouts in latent world models. It examines whether degraded probing performance can be recovered through recalibration, analyzes the transferability and geometry of latent alignment, and evaluates how calibration affects intervention-consistency measurements. The findings suggest that probe failures can partly reflect calibration mismatch, with recovery varying across rollout conditions and influencing intervention-based interpretations.

Critical strengths
  1. The paper identifies a meaningful reliability issue in interpreting world models, highlighting how representation shifts can compromise physical measurements obtained from fixed probes. This concern is particularly relevant when degraded probe performance is interpreted as evidence of lost physical information or inaccurate model predictions.
  2. The combination of probe refitting, target-agnostic alignment, and controlled comparisons provides complementary evidence for recoverable information in predicted latent representations. In particular, restoring the original probe through alignment fitted without physical target labels strengthens the evidence that probe failure does not necessarily imply information loss.
  3. The evaluation covers different tasks, rollout horizons, and state distributions, with informative ablations, statistical analyses, and negative results that help characterize the scope of the findings.
Critical weaknesses
  1. Limited Evidence of Practical Relevance and Generalizability. The most pronounced probe failures are concentrated in heavily perturbed rollout conditions, whereas substantially smaller discrepancies are observed under the model's planning distribution and in the evaluated second model. Although these results establish a meaningful failure mode, its prevalence and impact under representative world-model usage conditions remain insufficiently demonstrated. The limited evidence across models and operating conditions also makes it difficult to assess the generalizability of the central finding.
  2. The geometric interpretation of the affine alignment is insufficiently justified. Although orthogonal alignment recovers less of the gain on average, the fitted ridge map’s singular values and symmetry in a constrained, low-effective-rank latent space may reflect regularization and poorly identified directions rather than intrinsic non-isometric representation changes. Since the alignment is primarily constrained by the observed latent subspace, its full-space geometric properties may not reliably characterize the underlying representation relationship.
  3. Limited Robustness of the IC Evidence. The evidence that recalibration affects IC relies on a limited set of heterogeneous comparisons. In the preregistered setting, only one of the three feature-versus-control effect-size ratios has a reported 95% confidence interval excluding one, while larger changes arise primarily from non-preregistered comparisons involving fewer physical variables. Moreover, the effect-size intervals resample intervention directions while holding evaluation contexts fixed, leaving uncertainty from context sampling unaccounted for. The differing physical-variable sets further limit comparability across runs.
Questions for authors
  1. Under which planning and interpretability conditions does probe miscalibration become practically consequential? Are these patterns consistent across model architectures when tasks and state distributions are matched?
  2. Which geometric properties of the alignment are identifiable within the data-supported latent subspace, and what evidence distinguishes non-isometric representation differences from fitting artifacts? Is the reported symmetry assessed for the alignment matrix itself or its deviation from the identity, and how does this distinction affect the non-rotation conclusion?
  3. Can the observed IC changes be replicated across independently sampled evaluation episodes using a consistent set of physical variables that satisfy the same probe-accuracy criterion? Do the reported effect-size differences remain statistically supported when paired uncertainty estimates account for variability across both intervention directions and evaluation episodes?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.