acceptodds
Under review as a conference paper at ICLR 2027

WorldModelLens: From Attention to Rollouts - A framework for World Model Interpretability

Abstract

Interpreting world models requires action-, step-, and rollout-level abstractions that existing language-model hook-and-cache tools don't provide, forcing researchers to rebuild bespoke analysis code for every new architecture. We present WorldModelLens, an open-source toolkit that unifies recurrent state-space models (e.g., Dreamer), token-based models (e.g., IRIS), and joint-embedding architectures (e.g., I-JEPA) behind a single typed adapter API and a shared hook-and-cache layer, so researchers can write one analysis and run it across model families. Applying it to the I-JEPA ViT-H/14 checkpoint, we find that Integrated Gradients achieves a small but statistically robust localization signal in insertion-based patch restoration (-0.19 standardized effect, p < 10⁻⁴), a much weaker version of the effect found in pixel-reconstruction masked autoencoders (-1.15), suggesting I-JEPA's predictor relies more heavily on distributed context than a pixel-reconstruction objective does, though not to the point of being fully non-localizing. Our latent-attribution method recovers this same ranking while requiring  180× fewer model evaluations than exhaustive patching, and correlates with costlier head and block-level interventions (r=0.70-0.93). By collapsing architecture-specific plumbing into a common adapter surface, WorldModelLens lets the same hooks trace a prediction from an individual attention head up through block-level dynamics to full rollout behavior, turning interpretability of world models from a per-architecture engineering problem into a matter of choosing where, on that continuum, to look.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World models, mechanistic interpretability, capability-typed interface, hook-and-cache tooling, activation patching, sparse autoencoders, probing, intervention replay, imagination rollouts, joint-embedding predictive architectures

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Ei4yOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents WorldModelLens, a framework for mechanistic interpretability of heterogeneous world models. Its main design ideas are a capability-typed interface, time-indexed hook sites, and an intervention-replay mechanism intended to support interventions over recurrent states, actions, and imagined trajectories. The framework also includes several analysis modules, such as probing, replacement ablations, attribution, latent lenses, attribution patching, and statistical testing. The empirical study focuses primarily on I-JEPA ViT-H/14, examining context attribution, information routing through target tokens, the evolution of target representations across predictor depth, and the accuracy-efficiency trade-off of attribution patching.

Critical strengths
  1. The paper addresses an important gap and proposes a world-model-specific abstraction. WorldModelLens makes world-model-specific concepts, such as recurrent state, latent state, actions, environment or imagination steps on top of existing existing interpretability frameworks which already support activation caching and internal interventions.

  2. The I-JEPA case study is carefully controlled and methodologically stronger than many interpretability studies. The paper consistently includes controls that help rule out common interpretability artifacts. For example, the replacement experiments compare zero, mean, and resampled activations rather than relying only on zero ablation. All three intervention types strongly affect target-token positions while having little effect on distant context positions, which makes the result more convincing than a single ablation condition would.

Critical weaknesses
  1. The central methodological contribution is not directly validated in the setting where it matters most. The strongest novelty of the paper lies in its treatment of time, actions, recurrent or stochastic state, and multi-step intervention replay. However, the main case study uses I-JEPA, which does not contain action-conditioned recurrent dynamics or imagined trajectories. In the experiments, the replay horizon is zero, so the proposed replay mechanism becomes standard activation patching. This creates a significant method–experiment mismatch. The paper introduces machinery specifically intended for questions such as how an intervention at one time step affects future predicted states, rewards, values, or policies, but none of these capabilities are tested at scale. A much stronger validation would require experiments on models such as DreamerV3 in which internal states or actions are intervened on and the effects are traced over multiple future steps.

  2. Cross-model generality is demonstrated mainly through backend implementation rather than scientific transfer. The paper implements multiple adapters, but implementing a backend is not equivalent to demonstrating that the same interpretability analysis works reliably across model families. The central claim would be substantially stronger if the paper showed the same analysis code applied, without model-specific logic, to at least two substantially different dynamic world models, with comparable and interpretable results. At present, the paper’s claim of transfer across families rests mostly on interface design and backend coverage rather than on empirical evidence. The authors explicitly acknowledge this limitation.

  3. The practical advantage over existing generic intervention frameworks is not sufficiently demonstrated. Many of the individual analysis techniques used in the paper—probing, Integrated Gradients, activation patching, attribution patching, latent-lens-style readouts, and representational similarity—are established methods. The main added value of WorldModelLens therefore needs to come from its world-model-specific abstraction. However, because the main I-JEPA experiments do not require recurrent state, actions, or multi-step rollouts, the paper does not clearly demonstrate when WorldModelLens enables an analysis that would otherwise be difficult to implement with more general intervention tooling. A concrete comparison showing the amount of model-specific code required to implement the same multi-step analysis in a generic framework versus WorldModelLens would make the infrastructure contribution more compelling.

Questions for authors
  1. Please validate intervention replay on a genuinely dynamic world model. The most important missing experiment is a multi-step intervention study on a model such as DreamerV3. For example, the authors could intervene on a latent state, recurrent state, or action at one time step and trace how the effect propagates through future predictions, rewards, values, or policy outputs.
  2. Please include a strict cross-backend transfer experiment. It would be valuable to implement one analysis once and run exactly the same analysis code on at least two substantially different world-model architectures. This would directly test the paper’s central “write once, run across backends” motivation.
  3. Please distinguish more clearly between implemented, tested, and scientifically evaluated backends. The current backend table demonstrates broad engineering coverage, but only I-JEPA receives a large-scale scientific evaluation. Explicitly distinguishing these levels of validation would make the empirical scope of the paper clearer.
  4. Please clarify the practical advantage over generic intervention frameworks. A concrete case study comparing the implementation of the same multi-step intervention analysis in WorldModelLens and in a generic intervention framework would make the contribution easier to assess.
1 comment
AuthorsOpenReview ↗

DreamerV3 replay and the scope of WorldModelLens

Thank you for your feedback.

  1. We tested the replay procedure in Section 3.2 on 1,000 DreamerV3 Atari Boxing trajectories, each with a 15-step imagined rollout. We made a single edit to recurrent memory, the latent state, or an action and matched the sampling inputs across edited and unedited rollouts. Five steps after swapping in memory from another trajectory, mean relative memory drift was 81.0 percentage points above the latent-noise control with recorded actions and 85.3 points above it with policy-selected actions. Drift remained above the control ten steps after the edit.
  2. We’re testing whether the same replay analysis runs unchanged on a second dynamic architecture. We couldn’t complete that run within the rebuttal window, so the evidence for temporal replay in this revision comes from DreamerV3.
  3. Appendix A lists eight implemented adapters: PlaNet, DreamerV1, DreamerV2, DreamerV3, IRIS, TD-MPC2, I-JEPA, and World Models (Ha & Schmidhuber, 2018). The paper currently evaluates two of them: DreamerV3 and I-JEPA. We intentionally focused on two models so we could go more in depth and run the experiments with more rigor.
  4. With general-purpose intervention hooks, the researcher still has to decide how to replay the model after an edit, how to choose later actions, and how to match the random draws. WorldModelLens makes those choices part of the replay interface and gives researchers a shared way to use tools such as caching, probing, attribution, and patching wherever the model supports them. The differentiation occurs in the unified abstraction offered across multiple world models, and we aim to further add techniques like circuit tracing, along with more world model support. We plan to follow up in a future version by implementing the same multi-step analysis in a general-purpose framework and comparing the two workflows side by side.
Reviewer_g3n4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces WorldModelLens, a toolkit for world-model interpretability with a common interface, time-indexed hooks, and intervention replay across nine backends. Its main validation is a study of I-JEPA, combining attribution, controlled ablations, a latent lens, and patching experiments. The study identifies a checkpoint-loading issue, shows how predictions are carried by target-token positions, and distinguishes early content identification from later representational alignment.

Critical strengths

The toolkit addresses a practical gap: recurrent state, actions, and rollouts do not fit naturally into standard transformer interpretability workflows. A shared interface could substantially reduce the effort needed to study these models.

Critical weaknesses
  1. The main evaluation uses I-JEPA, which has neither actions nor multi-step rollouts. This leaves limited empirical evidence for the framework’s action-conditioned and rollout-level analyses, despite these being central to its motivation.
  2. Attribution-patching estimates are validated against exact patching on only four image pairs. This limits confidence in how robust the estimator recommendations are across inputs, even with the suggested exact-patching checks.
Questions for authors
  1. Beyond the I-JEPA study, what validation is available for intervention replay on recurrent or stochastic backends, particularly for effects across multiple imagined steps?
  2. How were the four exact-patching validation pairs selected, and are the estimator comparisons consistent across individual pairs?
1 comment
AuthorsOpenReview ↗

DreamerV3 replay validation and 50-pair I-JEPA results

Thank you for the feedback. We tested replay on DreamerV3 and expanded the I-JEPA exact-patching comparison to see how the results varied across image pairs.

  1. On testing one-time edits to memory, latent state, and actions in 15-step imagined rollouts from 1,000 DreamerV3 Atari Boxing trajectories, we found 5 steps after a memory replacement, mean relative memory drift went beyond a matched noise control by 81.0 percentage points with recorded actions and 85.3 points with policy-selected actions. The difference continued ten steps after the edit. In both settings, unedited replay matched the saved rollouts exactly on the collection GPU, and all pre-edit and identity-replacement checks passed.
  2. We also scaled exact-patching validation to 50 disjoint pairs across 100 Image Net validation images. For plain attribution patching, we observed higher correlation with exact effects across sampled encoder heads in 48/50 of the pairs4. For path-integrated estimator, higher correlation on block outputs in 37/50. These were the first four neighboring pairs after a seed-0 shuffling of the local images, so neighboring pairs had an image in common. This held for sampled heads in three of those four pairs, and all four pairs for block outputs.
Reviewer_dnNaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces WorldModelLens, a unified interpretability framework for different world model architectures. It provides a common interface for activation caching, probing, patching, and time-indexed interventions, aiming to support the analysis of actions and multi-step rollouts. The authors mainly validate the framework through an I-JEPA case study. While the framework supports multiple backends, the experimental analysis focuses on I-JEPA rather than action-conditioned world models with temporal rollouts.

Critical strengths
  1. The paper tries to propose an interporetability tool for world model. The motivation is interesting to propose unified interface to interpret different world models. The findings on I-JEPA shows that how the tool can be used to study visual understanding model, although their relevance to world model interpretability is less clear to me.
Critical weaknesses
  1. My main concern is that the paper does not clearly justify what constitutes a world model in the context of the proposed framework. Although Section 3.1 provides a unified interface, it seems to cover very different types of predictive models, from action-conditioned recurrent world models to static masked-image predictors such as I-JEPA. I am not sure what common world-modeling properties this framework is actually designed to interpret, or where its applicability ends. I think a clearer definition and motivation for including these different model families is important, especially given the broad claims of the paper.

  2. There is strong mismatch between the motivation and experimental validation. The paper emphasizes action conditioning, recurrent states, and multi-step intervention replay as key differences between world-model interpretability and existing LLM interpretability tools. However, the main experiments are conducted on I-JEPA, which neither conditions on actions nor performs temporal rollouts. This makes me confused whether the paper actually validates its most important contributions. While some of the I-JEPA analyses are interesting, I find their connection to understanding world-model dynamics relatively weak. I would expect at least one substantial experiment on an action-conditioned world model such as DreamerV3 or IRIS, demonstrating how interventions affect future rollouts to support the key claim that the tool is designed for world model.

Questions for authors

same as weakness.

1 comment
AuthorsOpenReview ↗

WorldModelLens scope and the new DreamerV3 experiment

Thank you for the thoughtful review. Your questions helped us clarify why we included I-JEPA and what evidence we needed to provide for temporal replay.

  1. WorldModelLens gives researchers a shared way to use tools such as activation caching, probing, attribution, and patching. Which tools apply depends on what each model supports. We use “action-conditioned world model” for a model that predicts how an environment state evolves under actions. I-JEPA is a static masked-image predictor, and we used it as a first test of the representation analysis tools. The shared interface lets us add methods as different models call for them.
  2. We agree that the original I-JEPA experiments left temporal replay untested. We added a DreamerV3 study using 1,000 Atari Boxing trajectories with 15 imagined steps each. Five steps after a single memory replacement, mean relative memory drift exceeded a matched noise control by 81.0 percentage points with recorded actions and 85.3 points with policy-selected actions. Replacing a single action also produced excess drift in both settings. We replayed the edited rollouts and measured changes in recurrent memory several steps later on this DreamerV3 checkpoint.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.