acceptodds
Under review as a conference paper at ICLR 2027

MixCoder: Decomposing Neural Computation into Low-Rank Transformations over Low-Dimensional Representations

Abstract

Fine-grained circuit analysis seeks to decompose neural computation into interpretable units and describe their interactions with attribution graphs. Existing feature-based approaches typically model semantic representations as isolated directions and computation as interactions among features, binding what is represented and how it is transformed within the same feature-level description. Growing evidence instead suggests that neural representations can occupy structured low-dimensional geometries, while effective computation over them can take the form of low-rank transformations. Motivated by this view, we argue that mechanistic interpretability should explicitly separate activation patterns, representations, and transformations, and decompose model computation into **low-rank transformations of low-dimensional representations**. From a generative model that factorizes these roles, we derive **MixCoder**, which identifies low-dimensional representation modules, lets representation states determine sparse activation patterns, and uses explicit operators to model transformations over them. Applied to language models, MixCoder reveals which representations are jointly read, how they are transformed, and where the results are written back. We further use this framework to study **transformation interference** when multiple low-dimensional transformations share limited computational space and its effects on model performance, providing insights into improving performance through mechanistic interpretability.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Circuit Analysis, Computational Interference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_x8wXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MixCoder, a method to replace the MLP layer in a transformer for the purpose of mechanistic interpretability. The main motivation is that a transcoder doesn't act on a multidimensional representation and MOLT doesn't say what the transformations act on. MixCoder combines both of these by introducing blocks to be the representations, having a sparse router select which blocks to activate, and then each selected block has an operator that reads the encoded input and writes to the residual stream. To justify separating the three roles, the authors write the restricted jacobian of a MLP plus ReLU as a product of the activation pattern, transformation, and representation read. They train to match the original MLP's output.

The first main experiment compares MixCoder to transcoders, MOLT, and sparse autoencoders on GPT-2 using multiple metrics. The second tests whether the block that selects an operator is also the representation that the operator mainly reads. The third sweeps operator rank, showing that increased writer rank improves reconstruction.

Critical strengths

(1) I think that MixCoder brings together pieces that the other methods lack. A transcoder feature reads along one direction and writes along one direction only. MoLT has transformations, but doesn't tell us anything in terms of representation. MixCoder gives a small map (eqs 10 and 11) that say which directions inside the block is read, the strength, and where they are written. I think that this gives us the better interpretability of transcoders while giving us the richer transformations of MOLT.

(2) While a small paragraph at the end of section 3, I like the property that the construction's attribution objects are invariant to whatever arbitrary coordinates that training decides on inside each block. This does convince me that representation blocks are natural.

Critical weaknesses

(1) The authors claim that MixCoder improves replacement fidelity over MOLT at "matched active count and operator rank". While I agree, I think it's important to note that this holds when their construction uses about 3x the compute according to Appendix D, and more importantly, the one that wins (linear/all) changes their construction in section 3.2. Section 3.2 uses the block in the routing (eq 7), while the linear/all variant does not use the block in it's scoring. I could be mistaken, but I think this is just the MOLT gate. This means that proofs provided in Appendix A.3 and onwards do not hold for the variant of MixCoder that actually wins.

(2) I don't agree (or understand) with some of the data/figures that is supposed to support with some of the claims.

(a) Section 4.3 concludes that "these results support a role for both effective write space and transformation interference in model performance". When I look at Appendix D.3 at panel a) G1: joint, which is what I believe to be the set-up that is used in the experiments, I do not see the supporting evidence. From my understanding, supporting evidence would mean that the blue is below 0 and the orange is above 0. I see it when the writer is fixed in panel c), but in panel a) it's just bouncing tightly around 0. The authors say this in the paragraph above starting on lines 1248.

(b) I am confused about the claim "In Qwen’s month and number tasks, cross-block reads carry most of the starting-value–offset interaction". In Appendix F.4, it's said that "Removing all active cross-reads changes correct answers from 106 to 105". This seems like conflicting evidence to me. The authors say that the rest of the model makes up for it, which I understand, but I think then the claim is not about how the model actually gets the answer, but only how the fitted proxy splits up layer 34's write (table 7). Since the model doesn't need the cross-reads to answer correctly, it seems wrong to say the cross-reads carry the interaction.

(c) Appendix A.5 brings up the point that different training runs could learn different blocks and ends with saying that "cross-run block matching and attribution consistency must be measured separately". In the supplementary material, there's a file with the cosine similarity for the jacobians for seeds 0 and 42. For norm/all MixCoder, it's 0.46, which is the lowest of all the models. If they compute the same local map, the value should be 1, but norm/all MixCoder is doing the worst (Top-64 MOLT gets 0.87).

As a smaller point, I think explaining MOLT in more depth, either in the main text or appendix, would help with readability as this is one of the main baselines.

Questions for authors

Could the authors clarify any of my misunderstandings in the weaknesses section? Specific questions are

(1) Is my understanding of the linear/all version of MixCoder correct?

(2) Could you explain why we don't see the same evidence in G1 when things are trained jointly vs when frozen?

(3) Could the authors either explain why the cosine similarity of the jacobians they computed is low for their model, or explain why this is not the right metric? If it's not the right metric, could the right metric be identified and a small experiment done?

3 comments
AuthorsOpenReview ↗

We sincerely appreciate the time you devoted and the thoughtful comments (1/2)

Dear Reviewer x8wX,

Thank you for your careful observations. Your thoughtful analysis of our experiments and your feedback have been immensely helpful to us. We have updated the manuscript, and our responses follow below.


Q1: The linear variant performs better than the block-norm variant. Does this contradict the theoretical claims? How should its computational cost be evaluated relative to MoLT?

The better performance of linear routing does not conflict with our modeling claims. Our modeling assumption is a block-level spike-and-slab prior: the activation of each block is given by

zb=sbub,z_b=s_bu_b,

where sbs_b determines only whether a concept is present, and the distribution of ubu_b remains invariant under rotations within the block. Consider the "year" manifold as an example. Under this modeling assumption, there is no difference in activation strength between 1987 and 2003. Activation changes only according to whether the concept of a "year" is present or absent, while the state corresponding to the particular year is carried by the representation within the block. Therefore, whether a block should be activated should be determined by its block norm ∥zb∥\|z_b\|. Sparsity should be imposed on blocks rather than individual atoms; imposing sparsity at the atom level would introduce a privileged basis within a block, which would contradict rotational invariance.

As for how to select which blocks to activate in order to minimize the loss, this is a block-subset selection problem, and it is generally NP-hard. The different variants of our method provide different proximal operators for this selection step. In particular, the linear variant does not need to compute the norms of all blocks at each selection, so it has a lower computational cost at this selection step. Differences in the performance of these selection rules do not conflict with our modeling claims concerning block-level sparsity and rotational invariance within blocks.


Q2: Do the results for G1 with joint training support the conclusion that effective write space and transformation interference both play a role in model performance?

In Appendix D.3.2, separating directions after joint training in G1 does not yield a clear improvement, whereas the fixed-writer control shows a more pronounced pattern: separation improves reconstruction and alignment worsens it. We hypothesize that joint optimization of the read and write channels may already have alleviated some interference, leaving less additional benefit from manually separating directions after training.

To test this explanation, we started from the same trained rank-16 fixed-writer model and compared two continuation branches with the encoder frozen, using the same data and number of updates. One branch kept the writer fixed and adjusted only the reader; the other adjusted both the reader and the writer. After 256 updates per branch, reconstruction MSE on the same 256 held-out tokens was 1.539 and 0.988, respectively. After subtracting the random-rotation control, the latter branch's benefit from directed separation was approximately 61% smaller; when intervention pairs were reselected at each endpoint, the benefit was still approximately 49% smaller.

Thus, allowing the read and write channels to adapt together reduces the additional reconstruction benefit of manually separating directions, consistent with our explanation. Separation remains beneficial, and the actual perturbation magnitudes differ between branches, so we regard this result as supporting evidence for the explanation.


Q3: If cross-block reads carry most of the interaction between the starting value and the offset, why does removing them change the number of correct answers only from 106 to 105?

During our initial screening of candidate layers, we found that interventions at layer 24 had a greater effect on the final logits. We selected layer 34 because the relevant representations could be read out most clearly at both the input and the output of this layer. This makes it easier to trace the correspondence between the states at the two ends and to illustrate the mapping from input to output. Given the limited effect of interventions at layer 34 on the final answers, we conducted additional intervention experiments at layer 24 and observed a substantially larger effect on the model's final answers.

AuthorsOpenReview ↗

We sincerely appreciate the time you devoted and the thoughtful comments (2/2)

Q4: Why is the cross-seed Jacobian cosine similarity of norm/all only 0.46, lower than the value of 0.87 for Top-64 MoLT?

Most of the error in MixCoder's single-seed JVP is concentrated in the same direction (see Appendix D.2). This is caused by LayerNorm: the actual input to the MLP never varies along this direction, so training cannot constrain the model's response along it. In response to your question, we removed this one direction with the largest error and recomputed the metric. MoLT's single-seed JVP remains essentially unchanged, while the corresponding value for MixCoder increases to 0.804.


Q5: Could MoLT be introduced more fully in the main text or in the appendix?

Thank you for your suggestion. Consider a ReLU MLP: once the activation pattern has been determined, the remaining computation is an affine map; if biases are ignored, it is a linear map. Thus, from this piecewise-affine perspective, the role of the nonlinearity is to select which map to use according to the current input, while the selected map performs the specific read and write transformation. This naturally suggests using a small collection of linear maps, each controlled by its own gate, to approximate the MLP, rather than enumerating exponentially many activation patterns. MoLT is an implementation of this idea, and constructing a surrogate model with it is both simple and efficient.

However, the basic modeling objects in MoLT are transformations selected by activation conditions. MoLT does not explicitly specify which representations these transformations act on or how they perform computations between representations. When we use an SAE to decompose the input into features, expressing one feature may be accompanied by weak responses along the directions of other features in a nonorthogonal superposed representation, giving rise to virtual weights. Therefore, the expansion of each transformation at the feature level may involve many unrelated features, increasing the difficulty of interpreting MoLT.

A detailed introduction to MoLT has been added to Appendix H.

Dear Reviewer x8wX,

Thank you for your detailed questions about the experiments and the comparison with MoLT. As the discussion wraps up, we would like to recap the analyses and revisions prompted by your feedback.

  • Joint training and read–write interference. To follow up on Appendix D.3.2, we started from the same rank-16 checkpoint, froze the encoder, and compared continuing with a fixed writer against training the reader and writer together. After adjusting for random rotations, joint adaptation reduced the extra reconstruction benefit of directed separation by 61%, or 49% when the intervention pairs were reselected (our response, Q2).
  • Cross-reads and final answers. Appendix F.4 reports the original layer-34 ablation, where subtracting the estimated cross-read contribution reduced correct answers from 106 to 105. In a layer-24 follow-up on the same 144 prompts, the same type of ablation reduced correct answers from 106 to 96. The norm-matched random and same-answer controls yielded 102 and 92 correct answers, so the specificity of the behavioral effect remains open.
  • Cross-seed agreement on valid GPT-2 inputs. We added a diagnostic that projects the derivative probes onto the LayerNorm constraint. The cross-seed cosine of norm/all MixCoder rose from 0.4607 to 0.7944, while Top-64 MoLT stayed near 0.8735 under the same evaluation (Appendix D.2).
  • MoLT, routing, and compute. We added Appendix H to explain gated low-rank transforms and how post-hoc SAEs can help interpret them. The recovery guarantees cover the zero-bias norm-routing submodel (Appendix A). Linear/all uses 27.53M forward MACs per token, versus 8.65M for Top-64 MoLT and a budget comparable to Top-256 MoLT (Table 1 and Appendix D).

If these results and clarifications address your questions, we would appreciate your taking them into account in your assessment.

Best regards,

The authors

Reviewer_5AVYOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

The authors contribute MixCoder, a new family of unsupervised learning techniques for understanding transformations performed by transformer layers. MixCoder extends mixture of linear transforms (MoLT), and represents outputs of cross-layer transformations or single MLPs in terms of a sum of linear transformations of the input, with nonlinearly computed k-sparse weights. The model has interacting sparsely-active blocks, where self-interactions and cross-interactions of blocks are argued to decompose computation into structures of interest. They apply MixCoder to understanding a vision transformer (DINOv3) as well as computations such as modular arithmetic on months in language models. They show that MixCoder variants can outperform existing unsupervised learning techniques in terms of MSE at a given sparsity (albeit with increased training and inference compute). They also show some qualitative examples of the structure that MixCoder captures.

Overall, I weakly recommend to reject the paper in its current form. It shows sophisticated theory and a seemingly comprehensive set of controls and experiments. But I find it quite difficult to understand, and see no clear evidence that the new method allows for insights into the structure of transformations in deep learning networks, beyond existing methods. I am quite uncertain given my difficulty understanding the content, and I could be convinced given additional effort to make the paper more accessible, and to show qualitative examples in a way that could be compared against existing methods.

Critical strengths

The ablations are comprehensive; each design choice is studied in combination with others. It seems that enough information is here to thoroughly understand the role of each design choice in their models’ success. The “science of unsupervised learning” experiments (synthetic manifolds, sensitivity analyses) are extensive. The mathematics showing identifiability of ground truth MixCoder parameters from the structure of inputs and outputs in a simplified setting appears interesting, and I (while inexpert) am not aware of similar results having been shown for other recent unsupervised learning methods.

Critical weaknesses

The presentation is quite dense and figure explanations are often difficult to follow. A paper introducing an improved unsupervised learning technique for interpretability should contain evidence to convince the reader that it is worth the complexity of implementing it, in order to gain new insight into the function of deep networks. The qualitative evidence is limited: an image overlay from a vision transformer with no baseline, and a study of single-layer language model computation where the layer of interest contributes only weakly to the full nonlinear transformation. The proposed extensions of MoLT are complex and computationally costly, and the claimed improvement of any given variant over MoLT is inconsistent. Linear/all (using linear routing rather than the routing scheme that primarily motivates the method) outperforms top-64 MoLT with matched number of active blocks and underperforms top-256 MoLT except on ∆CE. Norm/all, which is the main theoretical focus, underperforms top-64 MoLT. They compare to SAE-MoLT-SAE, but not to MoLT with post-hoc application of SAE, as was proposed in the original MoLT work.

Questions for authors

Where do period 26 and 52 come from? What are the X’s in fig. 4a? What are the x and y coordinates in fig. 4b? I believe I understand that fig 4b is showing cross-interaction terms couple month and offset representations (loops matching the teacher appear saliently only when cross-interactions are added). But I do not understand concretely what the coordinates represent. What are n, E, O, value, and xy coordinates of period 52 and period 26 plots in fig. 4c? Is there evidence that month information and offset information are captured in separate blocks? (I would think this is the intended upside of the MixCoder architecture?). What do the 5 cat photos correspond to in figure 13? I understand the largest and two of the smaller photos have high amplitude corresponding to sensitivity. I infer that the four smaller photos correspond to active readouts 45 and 56. If this reading is correct, is there meant to be some interpretative significance to the structure of the sensitivity within the image? In figure 8, is the fact that none of the students (including MixCoder) have unit activation boundaries matching the teacher patterns a negative update on all the methods attempted?

3 comments
AuthorsOpenReview ↗

We sincerely appreciate the time you devoted and the thoughtful comments (1/2)

Dear Reviewer 5AVY,

Thank you for reminding us that interpretability papers should be clear and easy to read. This is particularly important for our paper. We have revised the relevant sections accordingly and would welcome your feedback on the revisions. The manuscript has been updated. Please find our detailed point-by-point responses below.


Q1: Why does the stronger performance of linear routing not contradict the modeling motivation?

Our modeling motivation is block-level sparsity with a rotationally invariant within-block state. Under a block-level spike-and-slab prior,

zb=sbub,z_b=s_bu_b,

sbs_b indicates whether a concept is present, while the distribution of ubu_b is invariant under within-block rotations. For an idealized “year” representation, 1987 and 2003 can have equal activation strength while their within-block coordinates distinguish the two years. This motivates sparsity over whole blocks and norm-based selection: atom-level sparsity instead privileges particular coordinates within a block.

Choosing a block subset to minimize reconstruction loss is generally NP-hard. Our variants use tractable selection rules. The linear router learns an independent scalar score for each operator, avoiding block-norm calculations in the scoring step while retaining multidimensional encoded sources and explicit source-to-operator maps. Its stronger empirical fit therefore does not contradict the motivation for preserving within-block states. The norm-router recovery results apply to the norm-based submodel; they are not a guarantee for the linear router.

We also distinguish matched active count from matched compute. Linear/all outperforms Top-64 MoLT at the same active count and operator rank, but costs 27.53M forward MACs per token versus 8.65M. At comparable forward cost, Top-256 MoLT has better MSE, FVE, and JVP agreement, whereas linear/all has lower replacement Δ\DeltaCE and KL. The results support a tradeoff between fidelity, compute, and the available decomposition, rather than uniform superiority.


Q2: Qualitative comparisons with existing methods

We have conducted controlled qualitative comparisons on the same Qwen inputs, including transcoders and MoLT interpreted with post-hoc SAEs, which differs from the frozen SAE–MoLT–SAE architecture. Given the response length limit, we summarize the findings: transcoder units exhibit output-family structure, MoLT routing depends on output format and prediction position, and MixCoder exposes native four-dimensional source states together with the gates and fixed read–write maps that act on them. Detailed cases are provided in Appendix F.5.

AuthorsOpenReview ↗

We sincerely appreciate the time you devoted and the thoughtful comments (2/2)

Q3: Experimental and figure details

Q3.1: Figure 4 — periods, coordinates, and module roles

Integer addition can be expressed through angle combinations in multiple periodic channels. We scanned candidate periods in teacher activations and selected 52 and 26 using correlations at both the input and output, together with a frequency-separation rule. These are candidate periods used to construct the frozen readouts.

In Figure 4a, the crosses mark the true post-MLP endpoints P+yP+y, using the same frozen output readouts as the proxy.

In Figure 4b, the two readout directions correspond to the cosine and sine coordinates of the answer month. We extract the part of each write component that depends jointly on starting month and offset, after removing their separate mean effects, and add offset kk as a third display dimension. The horizontal page axis projects these three dimensions, while the vertical page axis projects the two readout coordinates.

At a fixed offset, the twelve starting months form a periodic contour. Cross-reads CC capture the main month–offset interaction in the fitted proxy, so their contour resembles the teacher's; the interactions in the shared component GG and self-reads DD are smaller. Matching the teacher after adding CC means recovering its interaction geometry, rather than creating the month circle for the first time.

In Figure 4c, nn is the starting integer. The Value column plots nn horizontally and the standardized answer value predicted from activations vertically. E and O denote even and odd starting integers and occupy two categorical horizontal positions. The vertical coordinate reads out answer parity, with target cos⁡(π(n+k))\cos(\pi(n+k)): +1+1 for even answers and −1-1 for odd answers. The figure fixes an offset of +3+3, so answer parity is opposite to starting-integer parity. Separating E and O makes the pre/post-write parity changes easier to compare between the two groups.

For each period T∈{52,26}T\in\{52,26\}, the horizontal and vertical coordinates are frozen readouts predicting cos⁡(2π(n+k)/T)\cos(2\pi(n+k)/T) and sin⁡(2π(n+k)/T)\sin(2\pi(n+k)/T), respectively. The first row, DD, shows write increments in these coordinates. The next two rows show state movements after adding CC and before/after the true MLP write, with open-to-filled markers indicating paired states.

Regarding module roles, the original analysis identified offset-related activation conditions and month-related source blocks: operator 2865's gate is associated with offset, while source block 2918 is associated with starting month. These provide candidate evidence of role differentiation. MixCoder explicitly connects within-block representations, operator activation conditions, and actual read–write maps, allowing us to examine which state is transformed, under which condition, and how.


Q3.2: Figure 13 — the five cat images and their spatial structure

The large image shows the source block's coordinates at different image locations. The upper-left and upper-right images show this path's write to readouts 45 and 56, respectively: the response in 45 is strong, while that in the still-active readout 56 is near zero. The lower-left image shows an intervention that increases the response in 56 while preserving 45; the lower-right shows an energy-matched control preserving both readouts. The four maps share a magnitude scale. Their spatial structure shows where the source path writes into each readout. This illustrates readout selectivity, analogous to output-null activity in neuroscience: the same source signal can strongly affect one downstream readout while having almost no effect on another.


Q3.3: Figure 8 — what does a mismatch with teacher activation boundaries mean?

A mismatch in support boundaries does not by itself imply failure to approximate the function. MoLT reads the full input, and MixCoder reads all encoded blocks; their operators can transform changing states while retaining relatively stable supports. For transcoders, increasing the active atom count brings more boundaries into alignment with the teacher. To account for multiple units remaining active together, the revised Figure 8 displays the complete support partition formed jointly by all active units along the input path. This makes recovered, additional, and missed boundaries directly visible, rather than relying on individual-unit activation plots.

Dear Reviewer 5AVY,

Thank you for pointing out where the comparisons and figures needed clearer explanations. As the discussion wraps up, we would like to highlight the changes prompted by your questions.

  • Linear routing and comparison at similar compute. We explained how linear routing preserves multidimensional source states and explicit read–write maps (our response, Q1). At similar forward cost, Top-256 MoLT has better reconstruction and local derivative agreement, while linear/all MixCoder has lower replacement ΔCE and KL (Table 1 and Appendix D).
  • Qualitative comparisons on the same Qwen inputs. We added month and number cases for transcoders, MoLT with post-hoc SAEs, and MixCoder. The cases show output-family structure in transcoders, how MoLT routing varies with output format and prediction position, and how MixCoder traces a four-dimensional source state through a gated read–write map (Appendix F.5, Figure 14).
  • How to read the Qwen plots. We rewrote the text and caption for Figure 4 to explain the choice of periods 26 and 52, the axes and endpoint markers, and how cross-reads account for most of the starting-month–offset interaction in the fitted proxy. We also traced the path from starting-month-associated source block 2918 through offset-sensitive operator 2865 (Appendix F.5).
  • Activation boundaries and the DINO images. We replaced Figure 8 with a whole-path view that marks matched, extra, and missed teacher boundaries (Appendix C). In our response, we explained the five panels of Figure 13 and how the same source path produces a strong write to readout 45 and a near-zero response in readout 56 (Appendix E).

If these revisions address your concerns, we would be grateful if you considered them in your assessment.

Best regards,

The authors

Reviewer_d4EcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+1)
Read review
Paper summary

The paper introduces/proposes MixCoder, highlighting a method that help decompose computation into low-dimensional representations, input-dependent routing conditions, and explicit low-rank transformations. MixCoder learns block encoders and conditional operators that can read from an aggregate of representations, providing condition source maps that help understand which representations are read, how their states are transformed and finally written back. The authors evaluate on widely used GPT-2 Small, along with Qwen3-4B-Instruct, and DINOv3. The primary contribution is a new conceptual/computational framework for analyzing multidimensional transformations in addition to only feature-level interactions. The experiments demonstrate some approximation fidelity and illustrate the framework's analytical utility.

Critical strengths
  • The motivation of the paper is quite strong, separating represented states, routing conditions, and transformations, addressing a genuine limitation of feature-centric circuit analysis, a feature that selects a computation need not be the representation that supplies its inputs, laying out a new path for moving beyond collections of feature-to-feature edges toward reusable, multidimensional computational mechanisms.

  • There is some additional analysis that the paper states that might be good for the research community to look at. The Qwen experiments suggest that cross-block reads account for much of the starting-value-offset interaction in month and number transformations, illustrating how the framework can distinguish computational routing from information flow. The GPT-2 experiments also show competitive replacement fidelity relative to operator-based baselines, with improvements on several reported metrics over Top-64 MOLT.

Critical weaknesses
  • There are some gaps in the motivation and the results that the paper shows, the paper motivates/proposes MixCoder as a solution to the separation of representations, routing conditions, and transformations; however, to establish that the learned blocks and operators recover identifiable/causally meaningful structure in the original network is yet left unanswered. The reported results primarily evaluate proxy reconstruction and prediction fidelity; the Qwen analysis relies on a fitted proxy and frozen readouts, while the intervention evidence is limited. Moreover, the main claim that MixCoder reveals how the original model computes remains insufficiently supported, rather than just demonstrating a more expressive parameterization.

  • It is not completely clear if the comparisons are sufficient to disentangle the benefits of low-rank operators, aggregate-state reads, and learned routing. Specifically, the strongest GPT-2 results use linear routing, whereas norm-based routing shows substantially weaker JVP fidelity, making it difficult to understand the improvements to the proposed representation transformation decomposition itself. Overall, its difficult to lay out a clear idea about the fundamental mechanistic advantage over existing operator-based approaches. I think writing them more clearly, with clear motivation and implications, will give a better idea of the primary utility of the proposed mixcoders.

  • As a general note, the proposed MixCoder jointly learns overlapping encoder blocks/routing functions/low-rank operators from an input-output based objective. I think multiple parameterizations can implement essentially the same function, and the within-block rotation invariance acknowledged by the authors is only one source of non-uniqueness. More fundamentally, the paper would be good if it establishes the conditions under which its learned blocks correspond to distinct underlying representations, or its operators to the transformations actually used by the model. The derivation of local affine structure does not completely resolve this, it helps motivate the decomposition, and to imply that the decomposition can be uniquely recovered from observations, would require more controlled experiments. It would be good to have some identifiability results or convincing recovery experiments under controlled ground-truth settings; the claim of mechanistic discovery is substantially stronger than the evidence supports.

Questions for authors
  • Since the operators are trained to approximate the original MLP's outputs, improved reconstruction does not establish that the learned decomposition reflects the original network's computation; it might be an alternative implementation of the same mapping. Some thoughts/clarity on those lines would be better to include in the paper for broader impact.

Minor comments:

The paper does not follow the recommended ICLR template, contains different font styles, and has some section formatting that is different (section names are not capitalized, bold is missing in a few places, etc.). It would be good if the authors could restyle the paper and recheck whether the template matches the one provided by the conference.

5 comments
AuthorsOpenReview ↗

We sincerely appreciate the time you devoted and the thoughtful comments (1/2)

Dear Reviewer d4Ec,

Thank you for the expert advice you have provided as a specialist in this field. We have benefited greatly from your suggestions and have updated the manuscript accordingly. Please find our responses below.


Q1. Correspondence between the learned decomposition and causal structure in the original network.

The original submission already reported interventions in the original model at layer 34 (Appendix F.4). At the natural intervention dose, the selected cross-read components increased the target-answer logit margin by 0.75 relative to norm-matched shuffled directions, although changes in the final answer were limited. We selected layer 34 for the readability of the relevant representations before and after the MLP, which facilitates examining the mapping learned by the proxy. To test counterfactual predictions at a site with stronger behavioral effects, we additionally evaluated layer 24 on 576 month prompts from four new templates. For each recipient, we compared replacing its MLP writeback with a real donor's writeback against adding the proxy-predicted donor–recipient difference to its original writeback; the recipient's residual bypass and the subsequent original network were retained. Among cases where the real MLP intervention changed the answer, the full proxy reproduced the post-intervention answer in 135/195 cases (69.2%) for starting-month changes and 135/193 (69.9%) for offset changes. These results test intervention-response fidelity beyond reconstruction, while leaving the identification of unique native modules open.


Q2. Stronger performance with linear routing is compatible with the modeling motivation.

Our motivation is a block-level spike-and-slab model,

zb=sbub,z_b=s_bu_b,

where sbs_b specifies whether a representation is present and ubu_b carries its state, with a distribution invariant under rotations within the block. For a representation of years, for example, 1987 and 2003 are different states of the same concept. The prior does not privilege either state because of its coordinates. This motivates sparsity over whole blocks and a rotation-invariant criterion such as ∥zb∥\|z_b\|, rather than coordinate-wise sparsity that favors a particular basis.

Finding the loss-minimizing block subset is generally NP-hard. The routing variants provide practical selection rules; the modeling motivation does not imply that norm routing must achieve the best reconstruction or derivative fidelity. Linear routing uses independent linear scores and avoids computing all block norms for selection, reducing the cost of that step. It retains the explicit source-block and read–write decomposition, but the norm-routing identifiability results do not automatically extend to it. Thus, the empirical advantage of linear/all should be distinguished from the theoretical properties of the norm-based submodel.

Low-rank operators let the model reduce interference and loss through read–write channels rather than coordinate-wise activation functions. Aggregated reads allow operators to achieve a given level of performance at lower rank. Norm-based routing makes block identification possible under the stated assumptions. Compared with MoLT, MixCoder adds explicit representation blocks for interpreting what each operator reads, transforms, and writes.


Q3. Within-block representations, transformations, and their activation conditions.

Synthetic manifolds. We compose complex inputs from simpler manifold components and examine the native codes of the fitted MixCoder. A source block retains a continuous trajectory corresponding to a circle component. Holding the circle state fixed while changing the co-occurring components shifts its code, indicating that the representation also contains background information. This supports a state-bearing block interpretation, without establishing a fully disentangled circle representation.

Qwen. At layer 34, blocks 2692 and 834 carry information associated with output families beginning with 4 and 6, respectively. Their inspected self-read paths write predominantly along one effective direction. Both calculation and copying can activate these routes, and activation also depends on format and context; these are therefore output-family-related channels rather than arithmetic-specific modules. See Appendix F.5. At layer 24, source block 2364 contains offset- and numeral-related information, while operator 2092 preferentially activates during forward month shifts from year-end starting months. Copying or merely mentioning the same words usually leaves it inactive, although counterexamples remain. These cases separately examine the source state, activation condition, and actual read–write map; the inspected paths do not explain the complete operators or recover the full teacher algorithm.

AuthorsOpenReview ↗

We sincerely appreciate the time you devoted and the thoughtful comments (2/2)

Q4. Identifiability.

You are right that identifiability is a crucial and difficult problem. Within-block rotations are only one source of nonuniqueness. Appendix A distinguishes two questions: recovering the structure of a fitted proxy from its observed routes and outputs, and establishing that independently fitted proxies recover the same underlying teacher structure.

For the zero-encoder-bias, norm-routing submodel, we obtain conditional projector recovery (see Appendix A.3). When two neighboring Top-KK supports share K−1K-1 blocks and exchange blocks ii and jj, their boundary satisfies

x⊤(Pi−Pj)x=0,Pi=EiEi⊤.x^\top(P_i-P_j)x=0,\qquad P_i=E_iE_i^\top.

Such exchanges constrain the block subspaces. Recovery requires boundary coverage, a connected exchange graph, and joint rigidity of the projector constraints; sufficient data or connectivity alone does not guarantee uniqueness. Recovering the paired effective operators additionally requires response excitation (Appendix A.4). The identifiable objects are projectors and effective operators, up to block permutation and within-block orthogonal changes of coordinates.

A further ambiguity remains across fits: different block decompositions may approximate the same total function. The schematic redistribution a+b=(a+c)+(b−c)a+b=(a+c)+(b-c) illustrates why total output alone need not determine individual contributions. Classical independent-subspace results rely on additional source assumptions that we do not impose on overlapping LLM representations. Our conditional result therefore does not establish unique recovery of the teacher's representations or transformations, and cross-run agreement remains a separate empirical question.


Q5. The role of within-block rotation invariance.

Rotation invariance makes the interpretation independent of an arbitrary coordinate basis. With Eb⊤Eb=IE_b^\top E_b=I, replacing EbE_b by EbRbE_bR_b for an orthogonal RbR_b, and transforming the associated coordinates and biases consistently, preserves the block projector, its norm-based routing score, and its actual residual-stream writeback. The model can therefore retain a representation's multidimensional state without selecting privileged axes within its subspace. This removes the basis dependence introduced by coordinate-wise sparsity; it does not itself guarantee that a learned block corresponds to a unique semantic concept.


Thank you also for pointing out the formatting issues. We have corrected the manuscript to follow the official ICLR template.

Dear Reviewer d4Ec,

Thank you for your careful reading and for asking what the learned decomposition reveals about the original network. As the discussion wraps up, we would like to recap the main updates addressing your questions.

  • Interventions in the original model. We tested the full proxy's predictions on 576 month prompts at Qwen layer 24. Among cases where replacing the original MLP write with a donor write changed the answer, the proxy's predicted write change reproduced the new answer in 135 of 195 cases for a starting-month change and 135 of 193 for an offset change (our response, Q1). The original layer-34 intervention is reported in Appendix F.4.
  • What the blocks encode and when operators act. We examined a synthetic block that tracks a continuous circle state and also varies with the other input components (our response, Q3). We added Qwen cases that trace four-dimensional source states through actual read–write maps and show how gates vary with the requested output and prompt context (Appendix F.5, Figure 14).
  • Routing and identifiability. We explained how learned linear scores can improve fit while preserving multidimensional source states (our response, Q2). Appendix A.2–A.5 shows why within-block rotations preserve attribution and gives conditions for recovering subspaces and effective operators of a fitted zero-bias, norm-routed proxy. Whether independent fits recover the same native structure remains an empirical question.

We also corrected the formatting to follow the official ICLR template. If these updates address your concerns, we would appreciate your taking them into account in your assessment.

Best regards,

The authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

We sincerely appreciate the time, effort, and thoughtful feedback

Comment

Dear Reviewers, AC, and SAC,

Thank you for your careful reading and constructive feedback on MixCoder. We summarize the main revisions and findings from the discussion below.

Reviewer d4Ec. We clarified the scope of our identifiability result. Under the conditions in Appendix A, the block projectors and effective operators of a fitted zero-bias, norm-routed proxy can be recovered from its routes and outputs. To test how well the fitted proxy predicts the original network’s response to an intervention, we ran a new donor-write experiment at Qwen layer 24. Among cases where replacing the original MLP writeback with a donor’s changed the answer, the full proxy reproduced the new answer in 135/195 starting-month interventions and 135/193 offset interventions. We appreciate the reviewer’s positive follow-up on the clearer identifiability framing and their increase in soundness from 2 to 3.

Reviewer 5AVY. We added side-by-side comparisons of transcoders, MoLT with post-hoc SAEs, and MixCoder on the same Qwen inputs (Appendix F.5, Figure 14). The cases show output-family structure in transcoder features and format- and position-sensitive routing in MoLT. With MixCoder, we trace a four-dimensional source state through an explicit gated read–write map. We also rewrote the explanation of Figure 4, replaced Figure 8 with a whole-path view of activation boundaries, and clarified the DINOv3 panels in our response. At comparable forward compute, Top-256 MoLT achieves better reconstruction and local derivative agreement, while linear/all MixCoder yields lower replacement ΔCE and KL.

Reviewer x8wX. In a controlled continuation from the same checkpoint, joint reader–writer training reduced the additional reconstruction benefit of manually separating read–write directions. We also clarified that cross-block reads account for most of the starting-value–offset interaction in the fitted layer-34 writeback. Subtracting the estimated cross-read contribution changed the number of correct answers from 106 to 105, and the larger layer-24 ablation effect fell within the range of its controls. In GPT-2, restricting derivative probes to directions allowed by LayerNorm raised the norm/all cross-seed cosine from 0.4607 to 0.7944, with Top-64 MoLT at 0.8735 under the same protocol (Appendix D.2). We also added Appendix H to explain MoLT and its interpretation with post-hoc SAEs.

Main contribution. MixCoder separates the state a model represents, the condition that selects a computation, and the low-rank map that reads and transforms that state. A selected operator can read from a different block than the one associated with its gate, making it possible to trace where the information comes from and where the result is written. Across controlled synthetic manifolds, GPT-2, Qwen, and DINOv3, we use this decomposition to study multidimensional representations, cross-block information flow, and interference between transformations.

Thank you again for considering our work. We would be grateful if you took the new results and revisions into account in your final assessment.

Best regards,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.