acceptodds
Under review as a conference paper at ICLR 2027

Weight-Dictionary Decomposition: Reading a Transformer's Residual Stream with Its Own Writes

Abstract

Sparse autoencoders interpret the residual states of a transformer through an overcomplete dictionary trained on activations. We build the dictionary from the weights instead. Each component's write into the residual stream is confined to a direction or subspace that its weights fix in advance, so collecting those directions from the checkpoint yields an overcomplete, training-free dictionary in which each atom is labeled with the component that wrote it. Weight-dictionary decomposition (WDD) sparse-codes an observed state over this dictionary to infer its largest native writes and their strengths. We score it against exact ground truth from hooked forward passes in eight models from six families, up to 7B parameters, on two corpora. An identified write has the right sign in more than 99.7% of cases in all eight models. Its magnitude is recovered with a median relative error between 0.25 and 0.54 in seven of them, biased upward for small writes because a 64-atom support absorbs the writes it leaves out, and it is not recovered in Pythia-6.9B. The identification rate rises with scale to 0.98 for Llama-7B. The rest of the support is not a list of writers, since only a quarter to seven tenths of the MLP atoms it selects are real writes at that token. Identification is bounded by an observability ceiling that we measure in all eight models. Once the other writes cancel a write along its own direction, OMP no longer finds it at the tested budget, although a probe on the full state shows that the information usually survives the cancellation. Used as measurement axes, the write directions single out a GPT-2 neuron that appears to erase the model's massive activation channel, and interventions under a frozen-normalization control reinterpret it as a participant in maintaining that channel, one that a distributed suppression field keeps below its clean level. In pre-specified comparisons at thirty nominees per arm, WDD beats a marginal outlier screen at nominating live circuits but does not outperform a correlation screen, and we argue that its distinctive contribution is the mechanistic content that each nomination carries.

Primary area:
interpretability and explainable AI
Keywords:
mechanistic interpretability, sparse dictionary learning, residual stream, attribution, sparse autoencoders, massive activations, attention sinks, orthogonal matching pursuit, training-free, circuit discovery

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_NMa5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

So the autoencoder learned The Overcomplete Dictionary from the Medium of the Activation. In this paper, authors are talking about a solution to make this problem being resolved efficiently. by using the residual stream, The model will be able to confine the advance and the direction or subspace, fix it by the weights. The author concluded that geometry nominates and the intervention sign meaning and the WDD distinctive value is the mechanistic content attachd to each nominations.

Critical strengths

This paper provided an honest report of the negative result. It also has a very rigid and well-designed control group that considers about exact ground truth and carefully separates the conceptual idea and the statistical ideas. This paper also included a decent amount of breath.

Critical weaknesses

However, this paper also have limited practical value and it is not talking about a clear contribution. Consider about the top one record is only 0.46 to 0.69 below 1 billion parameters. The channel of the 447 study is narrow and WDD's necessity isn't shown. Also, I think the presentation of this paper is kind of bad. There are way too many tags. Without the bigger explanation and equation, it's hard to read through and understand.

Questions for authors

Do the results hold at early and late layers, for models beyond 7B, or for MoE architectures? What ground truth exists for attention and embedding atoms (20–40% of the support)? Could you build an oracle for heads, e.g., the true per-head output projected into its row space?

Reviewer_MgDZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes replacing sparse autoencoder training with orthogonal matching pursuit (OMP) over the model’s own write vectors.

Critical strengths

None

Critical weaknesses
  • Neurons activate sparsely, so it is not surprising that the residual stream is mostly composed of the writes of the active neurons plus the attention heads. The main result follows almost directly from the architecture and activation sparsity. The paper should compare against the trivial baseline of ranking the known writes by their projection onto the residual. The paper asks a related question, "whether the atoms OMP selects are the writes that produced the state", and it is already suspicious that the overlap is low.

  • It is unclear what follows from this. The point of SAEs is that they yield somewhat interpretable features. The paper does not show whether the recovered writes are interpretable or whether the method leads to similar insights into the model. Without this, the comparison to SAEs is incomplete.

Questions for authors
  • How does OMP compare to simply ranking the known write vectors by their projection onto the residual? What does the greedy pursuit add when the dictionary is the true set of sources?

  • Are the recovered dominant writes interpretable? Can you show examples and compare against SAE features on the same positions?

Reviewer_7MWEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary
  • The paper introduces weight-dictionary decomposition (WDD): using an overcomplete dictionary of the transformer's write directions to sparsely decompose arbitrary residual stream vectors. The method does not require training and provides component provenance "for free".
  • They operationalise their method in a setup where a single residual stream vector and a full model checkpoint are available.
  • The decomposition sparse-codes a representation using top-k (e.g. 64, chosen using OMP) dictionary components writing before and in the residual stream block of the studied vector.
  • The authors show that while their method reconstructs residual stream states reasonably, it cannot be directly understood as evidence for attribution. They obtain this result by showing only a modest increase in the FVU metric when using dictionary directions from components that couldn't have written the residual stream state vector.
  • When scoring WDDs against ground truth, the largest MLP write with the correct sign is included in the decomposition 46%-98% of the time, but the identified magnitudes are uncalibrated.
  • The authors additionally use write directions to classify the roles of writing components and study GPT-2's massive-activation channel.
  • Their main results replicate across eight models (124M-7B parameters), with additional findings presented in GPT-2.
Critical strengths

The strongest side of this paper is its recurring theme of controls and baselines. The authors use them to test plausible interpretations of their results, and when their control results falsify those interpretations, they change their conclusions. A couple of examples:

  • In Section 4.1, in addition to using baseline dictionaries, they also build a control dictionary using write directions from blocks that couldn't possibly contribute to the residual stream vector under decomposition. They showed that such a dictionary still allows a decomposition that results in an FVU of 0.449 vs 0.358 with embedding directions, 0.490 vs 0.375 without embedding directions; random baseline at 0.5. They therefore conclude that reconstruction cannot validate component provenance. I believe the authors should revisit their conclusion, given that the non-embedding numbers suggest the opposite, yet this does not invalidate the strength of their control setup.
  • Still in Section 4.1, to diagnose and distinguish the source of the dictionary's performance between overcompleteness and coordinate alignment, they apply a Gram-preserving rotation. The outcome points to the latter, which the authors reasonably conclude.
  • The experiment with freezing LayerNorm in Section 4.5 is another example of a good control.
Critical weaknesses
  1. The paper, in its current form, is a hard-to-follow, high-density listing of experimental outcomes rather than their well-thought-through analysis.

The results section reads like a collection of findings that do not compose into a coherent support for the main claim or demonstrate the proposed method. Furthermore, I find the results incredibly difficult to follow: the experimental outcomes are not thoroughly analysed and are instead simply listed in the text. The sections are full of comparisons between metric values, with little to no further analysis or integration into claims. I believe the manuscript's quality is limited by the lack of supporting figures, diagrams, and tables (currently only a single figure and a single table in the main body). Many numerical comparisons in the text would be clearer with graphical elements. Similarly, a diagram could illustrate the idea of WDD.

Unfortunately, the main body reads throughout as difficult-to-follow AI slop. The abstract uses a similar style, and the authors acknowledge they used AI tools to draft and refine the entire manuscript. Whatever the cause, I believe the submission is low quality. One example of this is the final two sentences of the conclusions:

[...] We validated it across eight models of up to 7B parameters and resolved the apparent channel eraser of GPT-2 into a maintenance participant. Geometry nominates, and intervention assigns meaning.

  1. The setup for the proposed method is not realistic I am not sure whether to treat the proposed research as an interpretability method or as a basic-science finding about how much of a component’s write remains readable along its direction. If it's the former, it seems very rare to have access only to a full model checkpoint and only a single residual stream vector. Usually, with access to the full model checkpoint, one can run any input through the model and collect ground-truth activations corresponding to the writer directions. The reproducibility statement notes that one OMP pass over a 7B state (2×10112\times10^{11}) costs more multiply-adds than the cost of the forward pass that gives the exact answer (which is on the order of 7×1097\times10^{9}). I can see how sparse decomposition into writer components could help with a handful of marginal examples, such as understanding a synthetically generated steering vector, though even there the useful question is which component directions the vector resembles. This can be achieved without the sparse solver. It would help if the authors could state a concrete setting in which one has the state and the checkpoint but cannot run the hooked forward pass.
Questions for authors
  1. Suggestion regarding the paper presentation.

I suggest content improvements that directly address weakness 1, as described above.

  1. Clarification with regard to the conclusion from the Section 4.1 provenance experiment.

The authors build a dictionary whose directions come from blocks 7–11 only, which cannot have written a layer-6 state, and find it reaches FVU 0.449 against 0.358 for a correctly sourced dictionary of the same size (0.490 vs 0.375 with embedding rows removed). Their reasoning is that if reconstruction reflected provenance, such a dictionary should reconstruct badly, and since it does not, low FVU cannot validate provenance.

I am not sure the numbers support this. The comparison is against the Gram-preserving rotation (0.529), but the rotation actually does worse than the random-direction baseline (0.500). Against random, the embedding-free control is within 0.01, and the random baseline also has about four times as many directions as the embedding-free control, so a size-matched random dictionary would presumably do worse. It seems to me that the non-writer dictionary reconstructs about as well as random, and the writers reconstruct much better, which does not provide enough evidence to support the claim.

I would appreciate it if the authors could clarify this. One way to do it would be to add a size-matched random baseline next to each provenance control, report the control’s margin over it across k, and clarify whether they have additional evidence for the stronger claim. As written, "the most important negative result of the section" looks to me like a positive result for the dictionary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.