- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces MixCoder, a method to replace the MLP layer in a transformer for the purpose of mechanistic interpretability. The main motivation is that a transcoder doesn't act on a multidimensional representation and MOLT doesn't say what the transformations act on. MixCoder combines both of these by introducing blocks to be the representations, having a sparse router select which blocks to activate, and then each selected block has an operator that reads the encoded input and writes to the residual stream. To justify separating the three roles, the authors write the restricted jacobian of a MLP plus ReLU as a product of the activation pattern, transformation, and representation read. They train to match the original MLP's output.
The first main experiment compares MixCoder to transcoders, MOLT, and sparse autoencoders on GPT-2 using multiple metrics. The second tests whether the block that selects an operator is also the representation that the operator mainly reads. The third sweeps operator rank, showing that increased writer rank improves reconstruction.
Critical strengths
(1) I think that MixCoder brings together pieces that the other methods lack. A transcoder feature reads along one direction and writes along one direction only. MoLT has transformations, but doesn't tell us anything in terms of representation. MixCoder gives a small map (eqs 10 and 11) that say which directions inside the block is read, the strength, and where they are written. I think that this gives us the better interpretability of transcoders while giving us the richer transformations of MOLT.
(2) While a small paragraph at the end of section 3, I like the property that the construction's attribution objects are invariant to whatever arbitrary coordinates that training decides on inside each block. This does convince me that representation blocks are natural.
Critical weaknesses
(1) The authors claim that MixCoder improves replacement fidelity over MOLT at "matched active count and operator rank". While I agree, I think it's important to note that this holds when their construction uses about 3x the compute according to Appendix D, and more importantly, the one that wins (linear/all) changes their construction in section 3.2. Section 3.2 uses the block in the routing (eq 7), while the linear/all variant does not use the block in it's scoring. I could be mistaken, but I think this is just the MOLT gate. This means that proofs provided in Appendix A.3 and onwards do not hold for the variant of MixCoder that actually wins.
(2) I don't agree (or understand) with some of the data/figures that is supposed to support with some of the claims.
(a) Section 4.3 concludes that "these results support a role for both effective write space and transformation interference in model performance". When I look at Appendix D.3 at panel a) G1: joint, which is what I believe to be the set-up that is used in the experiments, I do not see the supporting evidence. From my understanding, supporting evidence would mean that the blue is below 0 and the orange is above 0. I see it when the writer is fixed in panel c), but in panel a) it's just bouncing tightly around 0. The authors say this in the paragraph above starting on lines 1248.
(b) I am confused about the claim "In Qwen’s month and number tasks, cross-block reads carry most of the starting-value–offset interaction". In Appendix F.4, it's said that "Removing all active cross-reads changes correct answers from 106 to 105". This seems like conflicting evidence to me. The authors say that the rest of the model makes up for it, which I understand, but I think then the claim is not about how the model actually gets the answer, but only how the fitted proxy splits up layer 34's write (table 7). Since the model doesn't need the cross-reads to answer correctly, it seems wrong to say the cross-reads carry the interaction.
(c) Appendix A.5 brings up the point that different training runs could learn different blocks and ends with saying that "cross-run block matching and attribution consistency must be measured separately". In the supplementary material, there's a file with the cosine similarity for the jacobians for seeds 0 and 42. For norm/all MixCoder, it's 0.46, which is the lowest of all the models. If they compute the same local map, the value should be 1, but norm/all MixCoder is doing the worst (Top-64 MOLT gets 0.87).
As a smaller point, I think explaining MOLT in more depth, either in the main text or appendix, would help with readability as this is one of the main baselines.
Questions for authors
Could the authors clarify any of my misunderstandings in the weaknesses section? Specific questions are
(1) Is my understanding of the linear/all version of MixCoder correct?
(2) Could you explain why we don't see the same evidence in G1 when things are trained jointly vs when frozen?
(3) Could the authors either explain why the cosine similarity of the jacobians they computed is low for their model, or explain why this is not the right metric? If it's not the right metric, could the right metric be identified and a small experiment done?
3 comments
We sincerely appreciate the time you devoted and the thoughtful comments (2/2)
Q4: Why is the cross-seed Jacobian cosine similarity of norm/all only 0.46, lower than the value of 0.87 for Top-64 MoLT?
Most of the error in MixCoder's single-seed JVP is concentrated in the same direction (see Appendix D.2). This is caused by LayerNorm: the actual input to the MLP never varies along this direction, so training cannot constrain the model's response along it. In response to your question, we removed this one direction with the largest error and recomputed the metric. MoLT's single-seed JVP remains essentially unchanged, while the corresponding value for MixCoder increases to 0.804.
Q5: Could MoLT be introduced more fully in the main text or in the appendix?
Thank you for your suggestion. Consider a ReLU MLP: once the activation pattern has been determined, the remaining computation is an affine map; if biases are ignored, it is a linear map. Thus, from this piecewise-affine perspective, the role of the nonlinearity is to select which map to use according to the current input, while the selected map performs the specific read and write transformation. This naturally suggests using a small collection of linear maps, each controlled by its own gate, to approximate the MLP, rather than enumerating exponentially many activation patterns. MoLT is an implementation of this idea, and constructing a surrogate model with it is both simple and efficient.
However, the basic modeling objects in MoLT are transformations selected by activation conditions. MoLT does not explicitly specify which representations these transformations act on or how they perform computations between representations. When we use an SAE to decompose the input into features, expressing one feature may be accompanied by weak responses along the directions of other features in a nonorthogonal superposed representation, giving rise to virtual weights. Therefore, the expansion of each transformation at the feature level may involve many unrelated features, increasing the difficulty of interpreting MoLT.
A detailed introduction to MoLT has been added to Appendix H.
Dear Reviewer x8wX,
Thank you for your detailed questions about the experiments and the comparison with MoLT. As the discussion wraps up, we would like to recap the analyses and revisions prompted by your feedback.
- Joint training and read–write interference. To follow up on Appendix D.3.2, we started from the same rank-16 checkpoint, froze the encoder, and compared continuing with a fixed writer against training the reader and writer together. After adjusting for random rotations, joint adaptation reduced the extra reconstruction benefit of directed separation by 61%, or 49% when the intervention pairs were reselected (our response, Q2).
- Cross-reads and final answers. Appendix F.4 reports the original layer-34 ablation, where subtracting the estimated cross-read contribution reduced correct answers from 106 to 105. In a layer-24 follow-up on the same 144 prompts, the same type of ablation reduced correct answers from 106 to 96. The norm-matched random and same-answer controls yielded 102 and 92 correct answers, so the specificity of the behavioral effect remains open.
- Cross-seed agreement on valid GPT-2 inputs. We added a diagnostic that projects the derivative probes onto the LayerNorm constraint. The cross-seed cosine of norm/all MixCoder rose from 0.4607 to 0.7944, while Top-64 MoLT stayed near 0.8735 under the same evaluation (Appendix D.2).
- MoLT, routing, and compute. We added Appendix H to explain gated low-rank transforms and how post-hoc SAEs can help interpret them. The recovery guarantees cover the zero-bias norm-routing submodel (Appendix A). Linear/all uses 27.53M forward MACs per token, versus 8.65M for Top-64 MoLT and a budget comparable to Top-256 MoLT (Table 1 and Appendix D).
If these results and clarifications address your questions, we would appreciate your taking them into account in your assessment.
Best regards,
The authors
We sincerely appreciate the time you devoted and the thoughtful comments (1/2)
Dear Reviewer x8wX,
Thank you for your careful observations. Your thoughtful analysis of our experiments and your feedback have been immensely helpful to us. We have updated the manuscript, and our responses follow below.
Q1: The linear variant performs better than the block-norm variant. Does this contradict the theoretical claims? How should its computational cost be evaluated relative to MoLT?
The better performance of linear routing does not conflict with our modeling claims. Our modeling assumption is a block-level spike-and-slab prior: the activation of each block is given by
zb=sbub,
where sb determines only whether a concept is present, and the distribution of ub remains invariant under rotations within the block. Consider the "year" manifold as an example. Under this modeling assumption, there is no difference in activation strength between 1987 and 2003. Activation changes only according to whether the concept of a "year" is present or absent, while the state corresponding to the particular year is carried by the representation within the block. Therefore, whether a block should be activated should be determined by its block norm ∥zb∥. Sparsity should be imposed on blocks rather than individual atoms; imposing sparsity at the atom level would introduce a privileged basis within a block, which would contradict rotational invariance.
As for how to select which blocks to activate in order to minimize the loss, this is a block-subset selection problem, and it is generally NP-hard. The different variants of our method provide different proximal operators for this selection step. In particular, the linear variant does not need to compute the norms of all blocks at each selection, so it has a lower computational cost at this selection step. Differences in the performance of these selection rules do not conflict with our modeling claims concerning block-level sparsity and rotational invariance within blocks.
Q2: Do the results for G1 with joint training support the conclusion that effective write space and transformation interference both play a role in model performance?
In Appendix D.3.2, separating directions after joint training in G1 does not yield a clear improvement, whereas the fixed-writer control shows a more pronounced pattern: separation improves reconstruction and alignment worsens it. We hypothesize that joint optimization of the read and write channels may already have alleviated some interference, leaving less additional benefit from manually separating directions after training.
To test this explanation, we started from the same trained rank-16 fixed-writer model and compared two continuation branches with the encoder frozen, using the same data and number of updates. One branch kept the writer fixed and adjusted only the reader; the other adjusted both the reader and the writer. After 256 updates per branch, reconstruction MSE on the same 256 held-out tokens was 1.539 and 0.988, respectively. After subtracting the random-rotation control, the latter branch's benefit from directed separation was approximately 61% smaller; when intervention pairs were reselected at each endpoint, the benefit was still approximately 49% smaller.
Thus, allowing the read and write channels to adapt together reduces the additional reconstruction benefit of manually separating directions, consistent with our explanation. Separation remains beneficial, and the actual perturbation magnitudes differ between branches, so we regard this result as supporting evidence for the explanation.
Q3: If cross-block reads carry most of the interaction between the starting value and the offset, why does removing them change the number of correct answers only from 106 to 105?
During our initial screening of candidate layers, we found that interventions at layer 24 had a greater effect on the final logits. We selected layer 34 because the relevant representations could be read out most clearly at both the input and the output of this layer. This makes it easier to trace the correspondence between the states at the two ends and to illustrate the mapping from input to output. Given the limited effect of interventions at layer 34 on the final answers, we conducted additional intervention experiments at layer 24 and observed a substantially larger effect on the model's final answers.