acceptodds
Under review as a conference paper at ICLR 2027

Locating Memory In Predictive World Models: Interventional Diagnosis Beyond Probing

Abstract

Linear probing is the default instrument for asking what a learned representation encodes, but it cannot say how a model retains that information. We exhibit three world models — a stateless Joint Embedding Predictive Architecture (JEPA), a GRU and an LSTM — that reach identical probe accuracy (1.000) on the same delayed working-memory task while implementing mechanistically incompatible forms of memory. To separate them we introduce nested interventional diagnostics: stable retention (SR) asks whether information about a past causal event survives a latent rollout; an interventional causal separation test, with a 2x2 extension that toggles recurrent state, asks where the memory resides; and the causal-subspace Lyapunov factor lambda_v(t) asks how the predictor sustains it. Three findings follow. Memory locus is a property of the trained model rather than of the architecture class: GRU and LSTM seeds trained identically span from near-pure state-based to predominantly dynamical solutions, a spread invisible to probe accuracy. A stateless predictor sustains memory by selective subspace preservation: at a prediction horizon of K = 5 it contracts sampled latent distances on average (kappa   0.36) while amplifying the causal direction 6.48x relative to a typical direction, a condition strictly weaker than the global norm preservation that unitary and orthogonal recurrent parameterisations enforce. And a controlled ablation that redistributes a trained encoder's causal weight across an orthonormal basis, holding predictor, task and training protocol fixed, drives stable retention from 0.458 to 0.107 and causal-subspace amplification from 14.2 to 4.3: encoder geometry causally gates whether dynamical memory is available at all.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, probing, world models, latent dynamics, causal intervention, JEPA, memory, recurrent neural networks

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_EnaVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

World models encode the previous information in latent representation. The authors exploit this fact to investigate where the information is memorized and saved in the model. Three diagnostics of stable retention, interventional causal separation, and causal-subspace Lyapunov factor are proposed to examine the memorization.

Critical strengths

World models are attracting interest in the research community, and the research direction is a timely topic. The research question in terms of memory is interesting, and several experiments, including JEPA, were performed.

Critical weaknesses
  • The causal separation test may be trivial; the direction of the counterfactual difference is removed to yield the same trajectories. As the authors noted, this observation can arise by construction, irrelevant to the memory mechanism. Other experimental designs could be considered.
  • The definition of memory in this study measures the distance of two trajectories. Strictly speaking, this definition is different from keeping recovery for information of previous events. For example, different trajectories can yield similar latent distributions. I encourage the authors to validate the memory definition works in experiments.
  • The authors used the Lyapunov norm in Eq. 5 with the intention to measure direction and magnitude. However, the Jacobian can rotate v in the other direction while keeping the norm. Other designs could be considered.
  • Stable retention is defined in Eq. 2. This definition regards exponential expansion, which is literally unstable, as stable. In other words, Eq. 2 does not reflect exponential expansion, such as g(z)=1.1z.
  • The training objective in Eq. 1 has a trivial minimizer with L=0 if f=0 and g(0)=0. In other words, there exists a solution of a constant encoder with no memory to minimize the training objective. Auxillary design to prevent this solution could be considered.
  • Writing should be improved:
    • "holding predictor” → “holding the predictor architecture” would be correct as it is trained.
    • “the same model” → “the same encoder configuration” would be correct as it is trained.
    • “Per-seed values for every aggregate are in Appendix F.” → this was not fully reported.
    • “dynamical memory is parameter-free” → dynamic memory still uses a learned predictor parameter. The intention would be no need for recurrent-state mechanics.
    • “all metrics use l_2” → others were actually used, such as cosine similarity.
    • Cho et al. (2014) and Hochreiter & Schmidhuber (1997) were listed in references but not cited in the manuscript.
    • “tasks with recommended disclosure” → should be explained cleanly.
    • Figure 1 → Possibly AI-generated figure. They are acceptable but could be further enhanced.
Questions for authors

See the weakness above.

Reviewer_Lz18OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This is a diagnostic paper that argues that linear probing can show whether or not a model's representation encodes a variable, but it cannot show if the model uses that encoding or where in the model the information is kept.

The authors train three latent world models, namely: a JEPA, a GRU, and an LSTM. All three reach perfect probe accuracy, but the authors argue that each model stores in the information in different ways. They propose three diagnostics to tell these apart:

  • Stable retention measures the separation between two latent trajectories, and whether their separation survives an autonomous rollout of the predictor
  • A causal separation test projects the estimated causal direction before rollout.
  • A causal subspace Lyapunov factor characterizes how the dynamics sustain the signal

The authors report three main findings. The first is that where the memory is kept depends on the seed and not just the architecture, so some GRU and LSTM seeds are mostly state-based and others are mostly dynamical, and you can't see this from the probe accuracy. The second is that the JEPA predictor contracts distances on average but keeps the causal direction, which they compare to a line attractor, and they use the shape of the Lyapunov factor to explain why retention doesn't improve steadily with the prediction horizon K. The third is that in MiniHack, rotating the encoder's has-key weight away from its trained direction drops stable retention from 0.47 to 0.07, and a control suggests it is the angle that matters and not how the weight is spread out.

Critical strengths

The most interesting part of the paper is the encoder rotation experiment in MiniHack. The authors rotate the has-key weight away from its trained direction while keeping the norm the same, and retrain the predictor each time, and the retention drops from 0.47 to 0.07. They also include an orientation-matched control which rotates the weight by the same angle without spreading it out, and it gives the same result, so it looks like the angle is what matters. I think this is a well designed control and as far as I can tell this result is new. It raises a useful question for JEPA style world models, namely that the encoder may decide whether the predictor is able to keep a memory at all.

Critical weaknesses

The main weakness is that two of the four contributions are not really new. The general finding that identically trained networks find different solutions is well established, for example Turner, Dabholkar and Barak (2021) trained hundreds of networks on tasks including a delay task, and found different solution types from identical hyperparameters and random initialization. This work is not cited. Similarly, the selective preservation finding is basically a line attractor, where one direction is kept while the others contract. The authors cite this themselves (Seung, 1996), Maheswaranathan et al. (2019) found approximate line attractors in trained LSTMs and GRUs, and the authors even say the result is "not surprising". They argue the point is that it emerges from predictive training without any constraint, but that is a smaller contribution than the abstract suggests. The diagnostic suite itself is mostly a combination of existing tools. This matters because it leaves the encoder rotation experiment as the main new contribution, and that is also the least developed part of the paper.

The second weakness is the soundness of the central claims, especially given how small the experiments are. The gap under the zeroed hidden state comes out at about 1 for every GRU and LSTM seed, and the authors read this as the models being able to fall back on their dynamics. But if the difference between the two trajectories at t_c lies almost entirely along v, then removing v with the hidden state zeroed makes the two starting states nearly identical, so the collapse would be about 1 by construction regardless of the dynamics. The authors make this same argument themselves about the encoder ablation, and the JEPA collapse on the binary task is exactly 1, which suggests it applies here too. SR under the zeroed condition is not reported, which would help settle this. Relatedly, it is not explained how the Lyapunov factor is computed for the recurrent models, and in Figure 3 it drops to about zero even though the recurrent models clearly retain separation, so it is not clear what it is measuring there. There is also no check of how much the gap changes for the same seed with a different set of counterfactual pairs, so the spread across seeds could partly be noise in the estimator. For the encoder experiment, the result is from only a few seeds in one environment, the claim that the angle matters and not the spread rests on a null result, and the paper itself says the alignment mechanism is an interpretation and not a tested claim. The models are small and quick to train, so more seeds should have been cheap, and papers asking similar questions about variation between seeds use far more networks.

Questions for authors

Questions that could change my assessment:

Can the authors show that the gap under the zeroed hidden state is not 1 by construction? If the difference between the two trajectories at t_c lies almost entirely along v, then removing v with the hidden state zeroed should make the two starting states nearly identical. Reporting SR under the zeroed condition, and how far apart the two states are after projection, would help settle this. If it is by construction then I don't think the 2x2 test supports the claim that the models fall back on their dynamics.

How much does gap(natural) change for the same seed if it is recomputed with a different set of counterfactual pairs? Without this it is hard to tell whether the spread across seeds reflects different solutions or noise in the estimator. Does the spread hold up with more seeds per architecture? Given how cheap the models are to train, this seems feasible.

For the encoder experiment, what happens if the has-key weight is rotated into another direction that the latent distribution varies along, rather than a random one? If retention survives, that would suggest the effect is about whether the signal lands in a direction the data actually varies along, rather than alignment with the trained direction specifically. This seems like the most direct way to test the alignment explanation the abstract states.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.