acceptodds
Under review as a conference paper at ICLR 2027

Compact but Moving: Intervention-Relevant Geometry in Recurrent World Models

Abstract

Learned world models may have compact interventions even when their recurrent state is high-dimensional, but it is unclear what happens to such a correction after it enters the model. We study this question in a controlled recurrent world model where prior work identified a checkpoint-specific rank-4 interface for one-shot counterfactual velocity interventions. The correction rapidly leaves this fixed entry subspace during autonomous rollout. Nevertheless, a low-rank image obtained by transporting the entry directions through the factual recurrent Jacobian chain continues to capture most of the nonlinear correction. Restarts using the tangent-predicted correction preserve substantial counterfactual future function. This transport/function pattern recurs across independently trained structured-GRU models and a parameter-matched LSTM initialized with a privileged compact correction. We further characterize a finite-horizon future-response operator over the full recurrent carrier. Patching shifts its leading future-sensitive directions toward the matched native-counterfactual organization, and the local operator accurately ranks finite perturbation effects over the registered direction panels at the patched and native-counterfactual basepoints. A separate full-amplitude assay finds substantial factual-endpoint tangent residuals and supports response reconfiguration in two of three checkpoints. Together, these results show that compact intervention structure can persist as a moving, state-dependent local geometry embedded in high-dimensional recurrent dynamics, without implying a fixed or dynamically closed low-dimensional state.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
World models, latent representations, recurrent dynamics, counterfactual intervention, tangent dynamics, representation geometry, mechanistic interpretability

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_QdhPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper examines how a low-dimensional hidden-state intervention affects subsequent predictions in a recurrent world model. Although the resulting state difference quickly leaves the original intervention subspace, it remains largely captured by that subspace after propagation through the model’s linearized dynamics. Restarting from the linearly predicted state also retains much of the intervention’s improvement toward a reference rollout produced by processing an edited observation. These findings recur across trained GRU models and an LSTM comparison. The paper also analyzes local output sensitivity and shows that a single linear approximation becomes less accurate at the full intervention amplitude. Its main contribution is therefore an empirical account of how low-dimensional interventions propagate within high-dimensional recurrent dynamics.

Critical strengths
  1. The geometric analysis is strengthened by testing whether the approximation preserves subsequent predictions. In Section 3.23.2, Eq. (2)–(4)(2)–(4), the authors propagate the intervention subspace through the model’s linearized dynamics and compare it with the actual state difference. However, subspace overlap alone would not establish that this approximation preserves information relevant to prediction. The restart experiment addresses this concern by initializing the unchanged model with the linearly predicted state perturbation and allowing it to continue autonomously. Section 4.24.2 and Figure 22 show that these restarts retain much of the original intervention’s improvement across independently trained models. The result therefore connects low-dimensional structure in the recurrent state to its effect on future predictions.

  2. Replication across trained models strengthens the main empirical conclusion. Section 3.73.7 aggregates results at the checkpoint level, while Section 4.24.2 distinguishes the propagation and prediction results from the stricter overall evaluation, which passes in 77 of 1010 trained GRU models. Section 4.34.3 also reproduces the propagation result in an LSTM, although its initial intervention uses privileged counterfactual information, as explained in Sections 3.23.2 and 5.25.2. This comparison provides some evidence of generality across architectures, within the stated initialization protocol. The distinction in Section 5.35.3 between a low-dimensional intervention and a self-contained reduced state is also appropriate, because the experiments continue to use the full recurrent dynamics.

Critical weaknesses
  1. The relative recovery metric leaves the absolute size of the prediction improvement unclear. Section 3.23.2 evaluates the fraction of the original intervention’s improvement retained by the approximate restart, restricting this calculation to cases where the original improvement exceeds 10−510^{-5}. Section 4.24.2 and Figure 22 principally report this relative measure. The retained fraction measures how much of the original intervention’s benefit survives the restart. However, a high value can reflect either a substantial prediction change or a small difference between otherwise similar rollouts. Therefore, paired prediction errors for the unmodified, intervened, and approximate-restart rollouts are needed to assess the magnitude of the reported effect. The coverage and specificity checks provide useful safeguards, but they do not directly resolve this reporting gap.

  2. I am less convinced by the analysis of how the intervention changes local output sensitivity. Sections 3.3–3.43.3–3.4 compare the leading right-singular subspaces of the output-sensitivity Jacobian—that is, the derivative of future predicted outputs with respect to the current recurrent state. Increased overlap establishes greater similarity between the state directions that most strongly affect those predictions. However, it does not establish that identical perturbations produce similar output changes. The validation in Section 3.53.5 does not close this gap because most perturbation directions differ between the intervened and reference states, and Eq. (7) measures rankings rather than prediction accuracy for effect magnitudes. Moreover, Section 4.4 reports considerably smaller overlap gains for one checkpoint, 0.00430.0043 and 0.00630.0063, alongside less consistent agreement between the complete Jacobians. The authors acknowledge these limitations. Nevertheless, the behavioral significance and robustness of this second finding remain less well established than those of the propagation result.

Questions for authors
  1. How large are the prediction improvements underlying Figure 22? Could the authors report paired errors for the unmodified rollout, the original hidden-state intervention, and the approximate restart, together with the distribution of the original intervention’s improvement? Please also summarize cases excluded by the 10−510^{-5} threshold in Section 3.2. These quantities would distinguish substantial retained changes in prediction from high relative scores obtained when the original improvement is small, which would materially affect my interpretation of the main behavioral result.

  2. Could the authors report the relative error ∥δk−δ^k∥2∥δk∥2\frac{\|\delta_k-\hat{\delta}_k\|_2}{\|\delta_k\|_2} over rollout time? Eq. (4)(4) measures how much of the actual state difference lies in the propagated subspace, but it does not measure whether Eq. (3)(3) predicts the correct direction and magnitude within that subspace.

  3. How sensitive are the subspace comparisons to the chosen truncation rank?

Reviewer_k9YrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I can tell, the paper studies how a low-rank hidden-state intervention in a small recurrent world model propagates over time, arguing it leaves its original subspace but stays captured by a Jacobian-transported version of it. It also analyzes how the model’s future-response sensitivity shifts after patching.

However the presentation of the setup, experiment and analysis is terrible and most text seems AI generated.

Critical strengths

None

Critical weaknesses

CW1: The paper is clearly AI generated, the presentation of the paper makes it impossible to read through. CW2: The paper uses introduces too many concepts without defining them. CW3: The papers figures uses hard to understand labelling such as " checkpoints 291402–291404"

Questions for authors

Could you please improve the presentation of the paper? Unexplained new concepts, and poor presentation such as figures with checkpoint 291411 as x axes make the paper impossible to follow.

Reviewer_FxqTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies what happens after a small, targeted change is made to a world model's internal memory. Previous work showed that changing four combinations of hidden-state coordinate could make a recurrent model predict a different object velocity. This paper asks whether that change stays in the same part of the hidden state as the model continues predicting. In a controlled two-object environment, the authors find that the correction quickly leaves its original directions but can still be tracked by a small set of directions that changes over time. They also restart the model using the predicted correction and show that much of the intended change in the future is preserved. Additional experiments examine which hidden-state changes affect future predictions and where a local linear approximation stops being accurate. The main conclusion is that an intervention can remain compact without the entire world model behaving like a small, self-contained system.

Critical strengths
  1. The paper asks a meaningful (a follow-up) question rather than simply assuming that a small intervention means the model has a small internal state. I find this distinction useful. The experiments directly separate where a correction enters the model from how it is carried through later predictions.

  2. The functional restart experiment is very informative and a good complementary of a geometric comparison alone. The authors check whether the predicted correction still produces the intended future, rather than only showing that hidden-state vectors remain close. The sign-flipped, wrong-object, and matched-size control corrections in Appendix B.5 are helpful for checking that the effect is specific to the intended change.

  3. The reporting is careful about both replication and failure cases. The main transport and recovery results are tested across nine eligible GRU models, and the paper distinguishes these results from the stricter checks, which pass in seven of the ten initially included models. The authors also report the model that failed the initial quality checks rather than replacing it. I appreciate the similarly cautious treatment of the LSTM experiment and the explicit statement that the study does not establish a closed low-dimensional model.

Critical weaknesses
  1. I have concerns on how broadly the finding applies to. The environment contains two objects, the observations directly provide position and velocity when visible, all evaluated actions are zero, and the main rollout lasts 12 transitions. The models are also trained on paired factual and counterfactual trajectories. These choices are reasonable for a controlled study, and the contact-changing cases are useful, but they leave open whether the same behavior holds in a model trained on ordinary action-conditioned data or over longer predictions with repeated interactions. The LSTM result adds evidence across architectures, but its initial correction uses the matched counterfactual hidden state, unlike the operational GRU correction. I would like to see either a comparable learned correction for the LSTM or a further experiment with nonzero actions and a longer prediction horizon.

  2. The almost-perfect prediction of perturbation rankings needs a more demanding test. Section 4.5 reports correlations above 0.999, but Appendix E shows that many tested directions are chosen using the same local operators being evaluated. Even the random directions are adjusted using the response operator, and the perturbation size is only one tenth of the original patch size. It would be beneficial to add independently sampled directions, report prediction errors in effect size rather than only rankings, and show how accuracy changes as the perturbation grows.

  3. The evidence that patching moves future-sensitive directions toward the counterfactual model state is less uniform than the main transport result. In Table H2, one model has median gains of only 0.0043 and 0.0063, much smaller than the other two models. If the author can explain or provide any insights on this? Meanwhile, appendix H.3 also reports that only 53.1% of cases have a positive gain for one of its conditions. Since the support rule mainly checks whether the median is positive, I am not yet sure how meaningful this small improvement is. Could the authors provide uncertainty estimates for these gains and test whether the conclusion is stable when changing the eight-step prediction horizon, the output-coordinate scaling, or the 90% threshold used to select directions?

Questions for authors

see weakness above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.