- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes sgatlin, a feedforward layer that replaces each transformer FFN with a large pool of single-neuron linear experts, of which only a small subset is selected per token by a product-key top-k gating network [1]. Compared to PEER [2], sgatlin removes the nonlinear activation of the experts; whereas PEER lets multiple heads share a single expert pool, sgatlin gives each of its channels a separate pool of neurons and shares only the query projection across channels to reduce the routing cost. Given the gating weights, each token is processed by an exact low-rank linear map, which the authors call a sparse, linear "feedforward circuit". In isoflop comparisons on SlimPajama across compute budgets from 1e17 to 6e18 FLOPs, sgatlin achieves lower test perplexity than MLP, SwiGLU, GPT-OSS-style MoE, and PEER. In an additional experiment trained on a different dataset (ClimbMix) with up to 1e19 FLOPs, sgatlin also achieves higher downstream scores than MLP, SwiGLU, and MoE. In a case study on a 4-layer TinyStories model, the authors show that the gating vectors form a semantically organized metric space (UMAP; nearest-neighbour-based automated interpretation accuracy of 0.98--0.99), and that patching the gating weights has a measurable causal effect on factual recall.
References
[1] Lample, Guillaume, et al. "Large memory layers with product keys." Advances in Neural Information Processing Systems 32 (2019).
[2] He, Xu Owen. "Mixture of a million experts." arXiv preprint arXiv:2407.04153 (2024).
Critical strengths
S1. Systematic compute-matched comparison. Using an isoflop design that jointly varies model size and the number of training tokens across four compute budgets (1e17--6e18 FLOPs), the paper compares two dense variants (MLP, SwiGLU) and two MoE variants (GPT-OSS-style MoE, PEER) on the same scaling ladder. Because the comparison is made between the compute-optimal points of each budget rather than at a single model size, the results are more reliable.
S2. A counterintuitive ablation result. In an architecture where single-neuron experts are selected via top-k, adding back an element-wise activation function substantially worsens perplexity (18.10 -> 20.6--21.1). This is an interesting result that supports the authors' interpretation that top-k selection alone provides sufficient nonlinearity for the layer.
S3. A concise modification of PEER with measured efficiency gains. Without making the architecture substantially more complex than PEER, sgatlin reduces the routing cost by sharing the query projection, and the H100 prefill/decode throughput measurements show that this advantage translates into actual speed. The stated release of all experimental code also benefits reproducibility in this line of work.
S4. Treating the gating vector as an identifier of the circuit. Since the FFN computation is fully determined once the gating weights are fixed, the gating vector itself can serve as a label for the circuit. This makes it possible, without a replacement model, to define distances between circuits and to intervene by swapping only the circuit applied to an otherwise unchanged input. The observation in the Discussion that fixing the gating weights, attention patterns, and normalization statistics turns the entire transformer into an exact affine transformation is also theoretically interesting.
S5. An attempt at quantitative evaluation of automated interpretation. The circuit discrimination quiz, which generates an explanation from the contexts of the nearest-neighbour circuits and then uses that explanation to identify the target circuit's actual context, is a meaningful attempt to evaluate automated interpretation quantitatively.
Critical weaknesses
W1. It is not disentangled whether the performance gain comes from the architecture or from the number of parameters. In the scaling ladder (Appendix B.2), sgatlin uses a separate pool of neurons for each of its 16 channels, whereas PEER uses neurons in a single shared pool. Hence, at the same depth, sgatlin has 4× as many total neurons as PEER, while the number of active neurons per token (128) is identical. Since PEER [1] has already shown that perplexity improves when only the total number of experts is increased with the number of active experts fixed, it is unclear whether the gain over PEER comes from the architectural changes or simply from having more neurons. There is also no ablation of "eliminating expert parameter sharing across channels", which the introduction presents as a key change. A comparison with PEER matched in total number of neurons and sparsity is needed. In addition, PEER is missing from the Core Evals comparison (Figure 4).
W2. The parameter memory cost is not discussed, and relevant prior work is missing. In Table 2, the neuron storage of sgatlin is , i.e., times that of PEER, yet Appendix A.2 attributes the difference between the two methods only to the shared query projection, and training memory is not reported. Monet [2] pointed out this memory bottleneck of PEER and reduced it to via product key composition. By integrating sparse dictionary learning into end-to-end MoE pretraining, Monet also made the same argument earlier as the positioning in Section 5.2 ("conceptually very similar to building a sparse autoencoder into the architecture from the outset"). The trade-off that linear experts require storing all neurons independently per channel should be discussed, citing Monet [2], MoE-X [3], Memory Layers at Scale [4], and Bilinear MLPs [5].
W3. The interpretability evidence does not rely on properties unique to sgatlin, and there are no baselines. Although the paper puts linearity at its center, all three analyses in Section 4 (UMAP, the discrimination quiz, and gating patching) use only the gating vectors and never analyse the linear circuit matrix . The separation of circuit formation and application is also shared by sparse routing in general, such as PEER, so the same analyses could be applied to PEER directly. Moreover, the quiz accuracy (0.98--0.99) and the NIE (about 0.4) are reported without any point of comparison, such as PEER routing, dense MLPs, or SAEs, which makes them hard to interpret. As a result, the claim of interpretability without replacement models currently reads as a property of sparse routing in general.
W4. The training scale is small, making it difficult to assess the gain at practical scales. The maximum compute is 6e18 FLOPs (isoflop) and 1e19 FLOPs (ClimbMix), the number of training tokens is not reported, and the interpretability analysis is limited to a single 4-layer TinyStories model. Read roughly from Figure 1, the gain over coarse-grained MoE tends to shrink from about 11% at 1e17 FLOPs to about 6% at 6e18 FLOPs, and the isoflop design alone cannot show what happens to this gain at larger budgets or with longer training. Compared with Monet [2], which trained 850M--4.1B models on 100B tokens, the experimental scale is small, and the practical implications of the current results are somewhat limited.
References
[1] He, Xu Owen. "Mixture of a million experts." arXiv preprint arXiv:2407.04153 (2024).
[2] Park, Jungwoo, et al. "Monet: Mixture of monosemantic experts for transformers." International Conference on Learning Representations. Vol. 2025. 2025.
[3] Yang, Xingyi, et al. "Mixture of experts made intrinsically interpretable." ICML 2025.
[4] Berges, Vincent-Pierre, et al. "Memory layers at scale." ICML 2025.
[5] Pearce, Michael T., et al. "Bilinear MLPs enable weight-based mechanistic interpretability." ICLR 2025.
Questions for authors
Q1. Design of the circuit discrimination quiz. How were the target circuits and the distractor options selected? In the example in Figure 7, the highlighted tokens of the distractors (They, they, an, and) all differ from the token of the correct option ('t), so the quiz appears solvable from the identity of the input token alone rather than from the circuit. What is the accuracy when the distractors are occurrences of the same token in different contexts? How does the pipeline compare with a baseline that retrieves, as neighbours, positions with the same input token instead of using the gating distance?
Q2. Interpretation at the level of individual neurons. The unit of the current analysis is the entire gating vector, composed of neurons per layer. Is a meaningful interpretation also possible at the level of individual neurons (rank-1 components)? Does the co-activation of neurons caused by sub-key sharing (Figure 9) make this difficult?
Q3. Discrepancy between Table 1 and Figure 3. According to Appendix B.1.1, the isoflop comparison in Figure 3 uses identity gating, so it should share the setting of the identity gating result in Table 1 (1.2B model, 6e18 FLOPs, 18.10). However, in the 6e18 FLOPs panel of Figure 3, the sgatlin model of apparently the same size lies clearly below a perplexity of 18, and appears to be lower even than the best softmax gating result in Table 1 (17.82). Where does this difference come from?
Q4. PEER baseline results. In Figures 1 and 3, PEER has higher perplexity than coarse-grained MoE at all four budgets, which is the opposite of the trend reported in the original PEER paper [1]. Unlike the original paper, PEER is applied to all layers here, the number of experts is much smaller, and the use of query BatchNorm is not stated. Are these differences in setup the cause?
Q5. Clarification of the experimental setup. (1) Which gating function was used in each experiment, and how is the statement in Section 5.1 that sgatlin removes "the softmax nonlinearity applied to the gating scores" consistent with the finally recommended setting (softmax)? Do the results in Section 4 hold for a softmax-gated model? (2) Is the query projection shared across channels or across layers? Section 2 and Appendix A.1 differ on this point. (3) What are the number of channels, , and the gating function of the TinyStories model?
References
[1] He, Xu Owen. "Mixture of a million experts." arXiv preprint arXiv:2407.04153 (2024).
5 comments
W3: Contextualizing and refining the interpretability analysis
Thank you for this suggestion. In our general response, we extend our feedforward circuit analysis to other architectures to better contextualize our findings. Our results here demonstrate that the semantic regularity we find for sgatlin circuits is not found for individual neurons (e.g. in an MLP, sgatlin or PEER). For the coarse-grained MoE, we find that both the routing patterns and individual experts are semantically also noticeably less structured. As you correctly point out however, the sparse routing property is shared by both sgatlin and PEER and we find that both have high semantic regularity in their circuit space as a result. The linearity of sgatlin’s circuits is an additional theoretical property that opens interesting future research directions such as tracing an entire transformer forward pass as a single linear map. We will make this distinction of how sparsity and linearity affect interpretability more clearly in the paper.
W4: Training scale
As you point out in the strengths “S1. Systematic compute-matched comparison”, we follow prior work [1,2,3] and invest our compute budget into a systematic isoflop analysis, to ensure a fair comparison across architectures. For each compute budget we consider, this requires sweeping each architecture across multiple model sizes / number of training tokens. The total wall-clock time for running the isoflop comparison alone amounts to thousands of H100 GPU hours. In this compute-optimal setting, we find that sgatlin consistently improves language modelling performance.
It is true that the performance gain of the coarse-grained MoE over the MLP improves over the scales we consider but relative gain of sgatlin to the MLP remains consistently high. To make this point possibly more convincing for you, we trained the compute-optimal model sizes, identified in the isoflop comparison on SlimPajama, on the ClimbMix dataset. To mirror your sample calculation, we report the relative improvement over the dense MLP baseline in % in the following:
| Feedforward architecture | 1e+17 FLOPs | 1e+18 FLOPs | 1e+19 FLOPs |
|---|---|---|---|
| MLP | +0.00 | +0.00 | +0.00 |
| SwiGLU | -0.55 | +0.95 | +1.31 |
| MoE | +2.56 | +3.79 | +4.90 |
| PEER | +1.08 | +5.01 | +4.12 |
| sgatlin | +8.08 | +7.88 | +8.13 |
[1] He (2024), Mixture of A Million Experts
[2] Hoffmann et al. (2022), Training Compute-Optimal Large Language Models
[3] Krajewski et al. (2024), Scaling Laws for Fine-Grained Mixture of Experts
Q1. Design of the circuit discrimination quiz.
Thank you for this insightful question. Following your and Reviewer 6z15’s suggestion, we have modified the distractors in the quiz to ensure they contain the activating token and reproduced our finding in this setting. In addition, we compare discrimination accuracy on the circuit discrimination quiz for other architectures and units of analysis to better contextualize the high accuracy we find in sgatlin. Please consider our general response for the details of this analysis. Your intuition is correct that the circuits are highly monosemantic, that is, many neighboring circuits activate for the same or very similar input tokens. We will clarify this point in the paper and add more sample items from the circuit discrimination quiz to the appendix.
Q2. Interpretation at the level of individual neurons.
This is a very interesting and important question, thank you! It could be the case that circuits are interpretable because each neuron they consist of is already by itself interpretable. We can repeat the circuit discrimination quiz using neurons as the unit of analysis and empirically find that the neurons by themselves are much less monosemantic, i.e. a single neuron often activates in many different contexts (see our general response for the details). This confirms that focusing on the circuits is a productive unit of analysis for the interpretability analysis.
Q3. Discrepancy between Table 1 and Figure 3.
Thank you for pointing out this inconsistency in the manuscript. The ablations in Table 1 were run with 8 instead of 16 channels for the isoflop comparison (this difference exists only for legacy reasons). We will clarify this in the paper.
Q4. PEER baseline results.
Indeed the biggest difference in our setting to the one considered by the original PEER paper is that we replace all feedforward layers in the transformer with (separate) PEER layers rather than only a single layer and that we compare to a different, possibly stronger MoE implementation (we use the MoE from GPT-OSS). We use PEER without query BatchNorm since the original paper reports minimal differences in terms of language modelling perplexity and this avoids leaking non-causal information of future time steps during training, we will clarify this in the paper. The ablation you suggested of matching the number of neurons between PEER and sgatlin at the same depth establishes that differences are not simply due to PEER having fewer neurons.
Q5. Clarification of the experimental setup.
Thank you for pointing out these two typos in the manuscript: We will remove the wrong statement that sgatlin removes “the softmax nonlinearity applied to the gating scores”, and will clarify in Appendix A.1 that the query projection is shared across channels (not layers, that was a typo). The isoflop comparison on Slimpajama applies no softmax nonlinearity to the gating scores but the ClimbMix results do. The TinyStories model and the Climbmix model we use for the revised feedforward circuit analysis reported in our general response use sgatlin with 16 channels and apply the softmax nonlinearity to the gating scores. We will clarify this in the paper.
Thank you for the close reading of our work and the helpful feedback. We believe the additional experiments you suggested have meaningfully improved our work. Please kindly check our general response with an overview of the new experiments, in the following we will respond to your individual comments.
Thank you for suggesting these ablations which we have now conducted and present in the general response. We copy them here for convenience: We ran these additional ablations for the compute-optimal model size on a compute budget of 6e17 FLOPs on the ClimbMix dataset and report the resulting Validation BPB (bits per byte) below:
These ablations confirm that the improvements in sgatlin are not simply due to a larger number of total experts/neurons and that not sharing neurons across channels is indeed beneficial.
We apologize for accidentally omitting PEER from the Core Evals comparison in Figure 4. We will update the manuscript to add the missing number and already copy it here for convenience:
Thank you for pointing out this unclear statement in Appendix A.2, it was intended to clarify the (possibly subtle) difference between the time and memory complexity between PEER’s and sgatlin’s router, connecting to the sentence before. We will clarify this in the paper. You are correct that one of the main differences between the two architectures is that sgatlin further increases compute sparsity (e.g. see L82-L84 and Figure 1). If we match the number of neurons available per channel, sgatlin has more total parameters than PEER but still less active parameters (due to the gating network). If we match the number of neurons between both architectures as you suggested above, PEER has slightly more total parameters than sgatlin due to the difference in their routers. Concretely for the ablations you suggested this results in the following training memory:
Please note that for simplicity our implementation is not optimized to leverage gradient sparsity which could potentially noticeably decrease peak memory requirements for both architectures. We will explicitly discuss the memory trade-off in sparse MoEs and sgatlin in our related work section citing the work you mentioned. Thank you in particular for pointing out Monet -- the horizontal or vertical expert decomposition it suggests can be combined with sgatlin which is an interesting avenue for future work.