acceptodds
Under review as a conference paper at ICLR 2027

Depth-Ordered Feature Memory: From Snapshots to Trajectories

Abstract

Sparse dictionaries describe transformers layer by layer and crosscoders assign cross-layer identities, yet neither maintains state across depth. **We introduce Depth-Ordered Feature Memory (DOFM)**, a sparse recurrent observer trained to predict normalized residual updates across a frozen transformer. DOFM integrates three mechanisms: a shared decoder fixing directional coordinates across depth _**(identity)**_; a per-token recurrent state tracking writes, retention, and attenuation _**(history)**_; and cumulative master gates organizing coordinates into a retention hierarchy _**(address)**_. Across diverse backbones, accumulated state reads word sense as accurately as the residual stream itself while revealing what probes cannot: which coordinate carries the sense, when it was written, and how long it persists. The retention prior hierarchically organizes coordinates, allowing a persistent tier to match full-dictionary accuracy and support addressable causal intervention; lexical competition resolves via write-time asymmetric recruitment, where competing senses receive reduced writes and persist without downstream erasure. Feature lifespan indexes persistence and generality over abstraction, turning cross-layer computation into an addressable state trajectory.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, sparse autoencoders, feature lifecycle, recurrent observer, activation steering

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2FwhOpenReview ↗
Rating
2
Confidence
2
Informativeness
2(−1)
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DOFM, a sparse recurrent observer that tracks feature lifecycles across the depth of a frozen transformer, trained without labels to predict the next normalized residual update from prior states. It unifies three mechanisms: (1) Identity: a single shared decoder D (unit-norm columns) fixes a coordinate's direction at every depth; (2)History: a per-token gated recurrent state, which is unrolled as an attention-discounted sum of writes; (3) Address: cumulative master gates (adapted from ordered neurons) partition the dictionary into slots. DOFM provides clear snapshots alongside the layers across multiple models.

Critical strengths
  1. The proposed framework genuinely solves a real structural gap, and the exact margin decomposition turns it into a usable attribution tool.
  2. Experiments across different models demonstrate the effectiveness of the method.
Critical weaknesses
  1. The scale of backbone models is limited below 2B. I would expect how it scales to larger sizes.
  2. The training of DOFM seems to be the prediction of residual connection along layers. Therefore, I am confused about whether it really reveals true behaviors that could not be found.
  3. All experiments are on a single task family (WSD + puns). It remains unclear about the model behavior in downstream scenarios, like QA, summarization, and reasoning.
  4. DOFM requires a 0.56–0.72B observer per backbone, which can not be ignored compared to the backbone model. It would be expected with a larger reequirement of computation resources on larger backbones.
Questions for authors
  1. Your central claim is that sense is written early and carried, not recomputed. Can you test this directly against the backbone — e.g., by ablating the backbone at the layer where the observer says the write happened vs. at later "carry" layers, and comparing outcome damage? Would the early-write story survive training the observer with a materially different inductive bias?
Reviewer_vArrOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Depth-Ordered Feature Memory (DOFM), a recurrent sparse observer for tracking representations across transformer layers. It extends sparse dictionary methods with a shared decoder, recurrent states, and retention-ordered gates, and evaluates the resulting representations through word-sense disambiguation and steering experiments.

Critical strengths
  1. The paper is clearly written and well organized.
  2. The experiments include relevant baselines and ablations, as well as interventions on the original model. These provide evidence that the selected directions carry behaviorally relevant information.
  3. Tracking how representations change across depth is a worthwhile interpretability question.
Critical weaknesses
  1. The paper’s main contribution appears to be architectural complexity rather than explanatory insight. DOFM extends the sparse-autoencoder framework with several additional mechanisms, but I found it difficult to identify a clear new understanding of how or why transformers work. The method produces richer descriptions of feature trajectories, yet the paper does not sufficiently explain what these descriptions teach us beyond the observation that semantic information emerges and persists across layers.
  2. The explanatory value of the sparse dictionary is unclear. Sparse coordinates are treated as features whose histories can be tracked, but their status as meaningful computational units remains insufficiently justified. Shared directions and retention ordering are imposed by the observer’s architecture. Their presence therefore does not, by itself, demonstrate that the transformer organizes its computation in the same way. Predictive readout and successful steering establish relevance, but do not fully validate the proposed decomposition.
  3. The additional complexity needs a clearer justification. An interpretability model should ideally simplify the system being studied, or at least yield a simpler explanation of its behavior. This need not mean that every analysis tool must have a minimal architecture, but its complexity should buy a concrete explanatory advantage. Here, understanding the transformer requires first understanding a substantial recurrent observer, its gates, its dictionary, and its coordinate-selection procedure. It is unclear whether the resulting account reduces the original problem or introduces another model whose behavior also needs explanation.
  4. The experiments do not establish that this complexity is necessary. Dense residual probes achieve comparable word-sense accuracy, dense contrastive directions generally provide stronger steering, and removing the retention prior preserves similar full-dictionary readout and steering performance. These results suggest that DOFM primarily reorganizes information already accessible through simpler methods. The paper should demonstrate a substantive mechanistic conclusion that those methods cannot provide and validate that conclusion independently of the observer.
Questions for authors
  1. What is the clearest new insight about transformer computation established by DOFM, beyond a more detailed description in learned coordinates?
  2. Which findings reflect properties of the backbone rather than consequences of the observer’s imposed recurrence and retention hierarchy?
  3. Can the central findings be recovered with a substantially simpler model? If not, what essential explanatory capability requires the additional complexity?
Reviewer_BJQZOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper looks at the sparse autoencoder (SAE) area of work, where the idea is to train a SAE on LLM layer embeddings or features so that interpretable features can be obtained. They propose DOFM that (1) [core idea] propagates a memory state across layers (2) shared decoder that keeps the feature directions the same for all layers and (3) retention shelves on the memory states that has a Q1 to Q4 hierarchy, with Q4 holding persistent features and Q1 holding less persistent features as layers progress. The motivation comes from the fact that existing methods such as the layer-wise SAE does not keep identity or maintain a state, and the crosscoder although it keeps a feature identity across layers, it does not maintain a recurrent state that propagates memory. They argue that by integrating (1)+(2)+(3), one can get "traceable, inspectable and queryable" feature trajectories.

Experiments are conducted with Gemma and Qwen models and compared with notably layer-wise SAE and static crosscoder that is "our shared no-memory arm" (so that probably refers to DOFM but without the recurrent memory state?):

  • Experiment 3.1 looks at the "cross-depth consistency and tracking" and it is clear that DOFM with prior's selectivity exceeds that of the static Crosscoder.
  • Experiment 3.2 studies how word semantic representations change across layers and finds that DOFM gives state accuracy higher than write accuracy.
  • Experiment 3.3 studies the retention shelves for localization and steering, and it is shown that (1) accuracy rises from Q1 to Q4 and (2) Q4 steering effects (on polysemous words) are the largest.
  • Experiment 3.4 looks at taking the LLM layer features HlH_l and hitting it in a dot product with the decoder's directions djd_j, and seeing how that number changes across layers for different retention shelves. It's shown that the DOFM labels roughly stick to how the LLM layer features behave.
Critical strengths
  • The idea is very interesting and is clearly is motivated by the drawbacks of the layer-wise SAE and crosscoder. The idea of introducing hierarchical retention shelves allows also for steering directions to be located.
  • Experimental results are strong and covers many properties, from feature lifecycle, word semantic representation changes across layers to retention shelves.
Critical weaknesses
  • Standard crosscoder of Lindsey et al. (2024) not used as a baseline. This perhaps needs to be compared to in order to stay close to existing works.
  • 20M tokens from OpenWebText seems small. Most papers go to billions of training tokens e.g. Gemma-2-2B also used in the Gemma Scope paper uses 4B training tokens, SAEBench with 500M tokens. So perhaps 20M tokens is a little small for a 131K dictionary
Questions for authors
  • What is the reason for not using SAEBench as a benchmark task? It would seem like a nature place to benchmark your method as it is a commonly used benchmark.
1 comment
Reviewer_BJQZOpenReview ↗

Response to Rebuttal

Many thanks to the authors for providing new results and detailed additional explanations.

[3/3] Response to Question 1: Benchmarking DOFM on SAEBench (6 Classification Suites & Cross-Depth Absorption)

There are still only comparisons between State Q4, Write Q4 etc... on table 29. Is it possible to include other external baselines?

[1/3] Response to Weakness 1: Comparison with Lindsey et al. (2024) Untied Crosscoders

1280 (Gemma-2) or 1024 (Qwen3-8B) is significantly lower than dmodel=2304 and dmodel=4096, the residual stream dimension that it is trying to describe. Usually you need much larger dimensions for SAEs? Then the baselines are not really fair? You could either retrain the much larger untied crosscoder model, or retrain DOFM with the same smaller width?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.