acceptodds
Under review as a conference paper at ICLR 2027

Sparsely gated tiny linear experts

Abstract

Sparsity allows scaling model parameters without proportionally increasing the size of active circuits. While mixture of experts models are made increasingly sparse, individual experts typically remain large and dense. Here, we demonstrate that further increasing sparsity by shrinking each expert to consist of a single neuron and selecting a tiny fraction of many available neurons can improve mechanistic interpretability and compute-matched performance. Counterintuitively, the key to achieving both is removing the nonlinearity typically applied to the experts, resulting in a network of sparsely gated linear neurons (sgatlin). In an isoflop comparison, we find that replacing all transformer feedforward layers with sgatlin improves perplexity in language models across different compute budgets. Crucially, the sparsity and linearity of the resulting feedforward circuits simplifies mechanistic analyses. In a small-scale case study, we demonstrate that the sparse feedforward circuits in sgatlin can be interpreted without having to train additional replacement models (e.g., sparse autoencoders). We find that feedforward circuits live in a semantically structured metric space and can be selectively intervened on to probe their causal roles. Our findings paint a possible path towards interpretable transformers without sacrificing modelling quality.

Primary area:
interpretability and explainable AI
Keywords:
transformer feedforward layer, sparsity, mechanistic interpretability, language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_imp7OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes sgatlin, a feedforward layer that replaces each transformer FFN with a large pool of single-neuron linear experts, of which only a small subset is selected per token by a product-key top-k gating network [1]. Compared to PEER [2], sgatlin removes the nonlinear activation of the experts; whereas PEER lets multiple heads share a single expert pool, sgatlin gives each of its CC channels a separate pool of neurons and shares only the query projection across channels to reduce the routing cost. Given the gating weights, each token is processed by an exact low-rank linear map, which the authors call a sparse, linear "feedforward circuit". In isoflop comparisons on SlimPajama across compute budgets from 1e17 to 6e18 FLOPs, sgatlin achieves lower test perplexity than MLP, SwiGLU, GPT-OSS-style MoE, and PEER. In an additional experiment trained on a different dataset (ClimbMix) with up to 1e19 FLOPs, sgatlin also achieves higher downstream scores than MLP, SwiGLU, and MoE. In a case study on a 4-layer TinyStories model, the authors show that the gating vectors form a semantically organized metric space (UMAP; nearest-neighbour-based automated interpretation accuracy of 0.98--0.99), and that patching the gating weights has a measurable causal effect on factual recall.

References
[1] Lample, Guillaume, et al. "Large memory layers with product keys." Advances in Neural Information Processing Systems 32 (2019).
[2] He, Xu Owen. "Mixture of a million experts." arXiv preprint arXiv:2407.04153 (2024).

Critical strengths

S1. Systematic compute-matched comparison. Using an isoflop design that jointly varies model size and the number of training tokens across four compute budgets (1e17--6e18 FLOPs), the paper compares two dense variants (MLP, SwiGLU) and two MoE variants (GPT-OSS-style MoE, PEER) on the same scaling ladder. Because the comparison is made between the compute-optimal points of each budget rather than at a single model size, the results are more reliable.

S2. A counterintuitive ablation result. In an architecture where single-neuron experts are selected via top-k, adding back an element-wise activation function substantially worsens perplexity (18.10 -> 20.6--21.1). This is an interesting result that supports the authors' interpretation that top-k selection alone provides sufficient nonlinearity for the layer.

S3. A concise modification of PEER with measured efficiency gains. Without making the architecture substantially more complex than PEER, sgatlin reduces the routing cost by sharing the query projection, and the H100 prefill/decode throughput measurements show that this advantage translates into actual speed. The stated release of all experimental code also benefits reproducibility in this line of work.

S4. Treating the gating vector as an identifier of the circuit. Since the FFN computation is fully determined once the gating weights are fixed, the gating vector itself can serve as a label for the circuit. This makes it possible, without a replacement model, to define distances between circuits and to intervene by swapping only the circuit applied to an otherwise unchanged input. The observation in the Discussion that fixing the gating weights, attention patterns, and normalization statistics turns the entire transformer into an exact affine transformation is also theoretically interesting.

S5. An attempt at quantitative evaluation of automated interpretation. The circuit discrimination quiz, which generates an explanation from the contexts of the nearest-neighbour circuits and then uses that explanation to identify the target circuit's actual context, is a meaningful attempt to evaluate automated interpretation quantitatively.

Critical weaknesses

W1. It is not disentangled whether the performance gain comes from the architecture or from the number of parameters. In the scaling ladder (Appendix B.2), sgatlin uses a separate pool of (16+6L)2(16+6L)^2 neurons for each of its 16 channels, whereas PEER uses (32+12L)2(32+12L)^2 neurons in a single shared pool. Hence, at the same depth, sgatlin has 4× as many total neurons as PEER, while the number of active neurons per token (128) is identical. Since PEER [1] has already shown that perplexity improves when only the total number of experts is increased with the number of active experts fixed, it is unclear whether the gain over PEER comes from the architectural changes or simply from having more neurons. There is also no ablation of "eliminating expert parameter sharing across channels", which the introduction presents as a key change. A comparison with PEER matched in total number of neurons and sparsity is needed. In addition, PEER is missing from the Core Evals comparison (Figure 4).

W2. The parameter memory cost is not discussed, and relevant prior work is missing. In Table 2, the neuron storage of sgatlin is nchanneldffwdmodeln_{\text{channel}} d_{\text{ffw}} d_{\text{model}}, i.e., nchanneln_{\text{channel}} times that of PEER, yet Appendix A.2 attributes the difference between the two methods only to the shared query projection, and training memory is not reported. Monet [2] pointed out this memory bottleneck of PEER and reduced it to O(N)O(\sqrt{N}) via product key composition. By integrating sparse dictionary learning into end-to-end MoE pretraining, Monet also made the same argument earlier as the positioning in Section 5.2 ("conceptually very similar to building a sparse autoencoder into the architecture from the outset"). The trade-off that linear experts require storing all neurons independently per channel should be discussed, citing Monet [2], MoE-X [3], Memory Layers at Scale [4], and Bilinear MLPs [5].

W3. The interpretability evidence does not rely on properties unique to sgatlin, and there are no baselines. Although the paper puts linearity at its center, all three analyses in Section 4 (UMAP, the discrimination quiz, and gating patching) use only the gating vectors and never analyse the linear circuit matrix WcircuitW^{\text{circuit}}. The separation of circuit formation and application is also shared by sparse routing in general, such as PEER, so the same analyses could be applied to PEER directly. Moreover, the quiz accuracy (0.98--0.99) and the NIE (about 0.4) are reported without any point of comparison, such as PEER routing, dense MLPs, or SAEs, which makes them hard to interpret. As a result, the claim of interpretability without replacement models currently reads as a property of sparse routing in general.

W4. The training scale is small, making it difficult to assess the gain at practical scales. The maximum compute is 6e18 FLOPs (isoflop) and 1e19 FLOPs (ClimbMix), the number of training tokens is not reported, and the interpretability analysis is limited to a single 4-layer TinyStories model. Read roughly from Figure 1, the gain over coarse-grained MoE tends to shrink from about 11% at 1e17 FLOPs to about 6% at 6e18 FLOPs, and the isoflop design alone cannot show what happens to this gain at larger budgets or with longer training. Compared with Monet [2], which trained 850M--4.1B models on 100B tokens, the experimental scale is small, and the practical implications of the current results are somewhat limited.

References
[1] He, Xu Owen. "Mixture of a million experts." arXiv preprint arXiv:2407.04153 (2024).
[2] Park, Jungwoo, et al. "Monet: Mixture of monosemantic experts for transformers." International Conference on Learning Representations. Vol. 2025. 2025.
[3] Yang, Xingyi, et al. "Mixture of experts made intrinsically interpretable." ICML 2025.
[4] Berges, Vincent-Pierre, et al. "Memory layers at scale." ICML 2025.
[5] Pearce, Michael T., et al. "Bilinear MLPs enable weight-based mechanistic interpretability." ICLR 2025.

Questions for authors

Q1. Design of the circuit discrimination quiz. How were the target circuits and the distractor options selected? In the example in Figure 7, the highlighted tokens of the distractors (They, they, an, and) all differ from the token of the correct option ('t), so the quiz appears solvable from the identity of the input token alone rather than from the circuit. What is the accuracy when the distractors are occurrences of the same token in different contexts? How does the pipeline compare with a baseline that retrieves, as neighbours, positions with the same input token instead of using the gating distance?

Q2. Interpretation at the level of individual neurons. The unit of the current analysis is the entire gating vector, composed of C⋅kC \cdot k neurons per layer. Is a meaningful interpretation also possible at the level of individual neurons (rank-1 components)? Does the co-activation of neurons caused by sub-key sharing (Figure 9) make this difficult?

Q3. Discrepancy between Table 1 and Figure 3. According to Appendix B.1.1, the isoflop comparison in Figure 3 uses identity gating, so it should share the setting of the identity gating result in Table 1 (1.2B model, 6e18 FLOPs, 18.10). However, in the 6e18 FLOPs panel of Figure 3, the sgatlin model of apparently the same size lies clearly below a perplexity of 18, and appears to be lower even than the best softmax gating result in Table 1 (17.82). Where does this difference come from?

Q4. PEER baseline results. In Figures 1 and 3, PEER has higher perplexity than coarse-grained MoE at all four budgets, which is the opposite of the trend reported in the original PEER paper [1]. Unlike the original paper, PEER is applied to all layers here, the number of experts is much smaller, and the use of query BatchNorm is not stated. Are these differences in setup the cause?

Q5. Clarification of the experimental setup. (1) Which gating function was used in each experiment, and how is the statement in Section 5.1 that sgatlin removes "the softmax nonlinearity applied to the gating scores" consistent with the finally recommended setting (softmax)? Do the results in Section 4 hold for a softmax-gated model? (2) Is the query projection shared across channels or across layers? Section 2 and Appendix A.1 differ on this point. (3) What are the number of channels, dffwd_{\text{ffw}}, and the gating function of the TinyStories model?

References
[1] He, Xu Owen. "Mixture of a million experts." arXiv preprint arXiv:2407.04153 (2024).

5 comments

Thank you for the close reading of our work and the helpful feedback. We believe the additional experiments you suggested have meaningfully improved our work. Please kindly check our general response with an overview of the new experiments, in the following we will respond to your individual comments.

W1: Matching total neurons in PEER

Thank you for suggesting these ablations which we have now conducted and present in the general response. We copy them here for convenience: We ran these additional ablations for the compute-optimal model size on a compute budget of 6e17 FLOPs on the ClimbMix dataset and report the resulting Validation BPB (bits per byte) below:

AblationValidation BPB
PEER0.9262
PEER w/ total neurons matched0.9155
sgatlin w/ neuron sharing0.8913
sgatlin0.8855

These ablations confirm that the improvements in sgatlin are not simply due to a larger number of total experts/neurons and that not sharing neurons across channels is indeed beneficial.

W1: PEER is missing from the Core Evals comparison

We apologize for accidentally omitting PEER from the Core Evals comparison in Figure 4. We will update the manuscript to add the missing number and already copy it here for convenience:

Feedforward architectureCore Evals score
MLP0.1592
SwiGLU0.1727
MoE0.1944
PEER0.1879
sgatlin0.2028

W2: Discussing memory cost and relevant prior work

Thank you for pointing out this unclear statement in Appendix A.2, it was intended to clarify the (possibly subtle) difference between the time and memory complexity between PEER’s and sgatlin’s router, connecting to the sentence before. We will clarify this in the paper. You are correct that one of the main differences between the two architectures is that sgatlin further increases compute sparsity (e.g. see L82-L84 and Figure 1). If we match the number of neurons available per channel, sgatlin has more total parameters than PEER but still less active parameters (due to the gating network). If we match the number of neurons between both architectures as you suggested above, PEER has slightly more total parameters than sgatlin due to the difference in their routers. Concretely for the ablations you suggested this results in the following training memory:

ArchitecturePeak memory (MiB)Val BPBTotal paramsActive params
PEER w/ neurons per channel matched10,0240.9262180.4M47.2M
PEER w/ total neurons matched14,9120.9155587.2M51.4M
sgatlin12,5310.8855564.7M28.8M

Please note that for simplicity our implementation is not optimized to leverage gradient sparsity which could potentially noticeably decrease peak memory requirements for both architectures. We will explicitly discuss the memory trade-off in sparse MoEs and sgatlin in our related work section citing the work you mentioned. Thank you in particular for pointing out Monet -- the horizontal or vertical expert decomposition it suggests can be combined with sgatlin which is an interesting avenue for future work.

W3: Contextualizing and refining the interpretability analysis

Thank you for this suggestion. In our general response, we extend our feedforward circuit analysis to other architectures to better contextualize our findings. Our results here demonstrate that the semantic regularity we find for sgatlin circuits is not found for individual neurons (e.g. in an MLP, sgatlin or PEER). For the coarse-grained MoE, we find that both the routing patterns and individual experts are semantically also noticeably less structured. As you correctly point out however, the sparse routing property is shared by both sgatlin and PEER and we find that both have high semantic regularity in their circuit space as a result. The linearity of sgatlin’s circuits is an additional theoretical property that opens interesting future research directions such as tracing an entire transformer forward pass as a single linear map. We will make this distinction of how sparsity and linearity affect interpretability more clearly in the paper.

W4: Training scale

As you point out in the strengths “S1. Systematic compute-matched comparison”, we follow prior work [1,2,3] and invest our compute budget into a systematic isoflop analysis, to ensure a fair comparison across architectures. For each compute budget we consider, this requires sweeping each architecture across multiple model sizes / number of training tokens. The total wall-clock time for running the isoflop comparison alone amounts to thousands of H100 GPU hours. In this compute-optimal setting, we find that sgatlin consistently improves language modelling performance.

It is true that the performance gain of the coarse-grained MoE over the MLP improves over the scales we consider but relative gain of sgatlin to the MLP remains consistently high. To make this point possibly more convincing for you, we trained the compute-optimal model sizes, identified in the isoflop comparison on SlimPajama, on the ClimbMix dataset. To mirror your sample calculation, we report the relative improvement over the dense MLP baseline in % in the following:

Feedforward architecture1e+17 FLOPs1e+18 FLOPs1e+19 FLOPs
MLP+0.00+0.00+0.00
SwiGLU-0.55+0.95+1.31
MoE+2.56+3.79+4.90
PEER+1.08+5.01+4.12
sgatlin+8.08+7.88+8.13

[1] He (2024), Mixture of A Million Experts

[2] Hoffmann et al. (2022), Training Compute-Optimal Large Language Models

[3] Krajewski et al. (2024), Scaling Laws for Fine-Grained Mixture of Experts

Q1. Design of the circuit discrimination quiz.

Thank you for this insightful question. Following your and Reviewer 6z15’s suggestion, we have modified the distractors in the quiz to ensure they contain the activating token and reproduced our finding in this setting. In addition, we compare discrimination accuracy on the circuit discrimination quiz for other architectures and units of analysis to better contextualize the high accuracy we find in sgatlin. Please consider our general response for the details of this analysis. Your intuition is correct that the circuits are highly monosemantic, that is, many neighboring circuits activate for the same or very similar input tokens. We will clarify this point in the paper and add more sample items from the circuit discrimination quiz to the appendix.

Q2. Interpretation at the level of individual neurons.

This is a very interesting and important question, thank you! It could be the case that circuits are interpretable because each neuron they consist of is already by itself interpretable. We can repeat the circuit discrimination quiz using neurons as the unit of analysis and empirically find that the neurons by themselves are much less monosemantic, i.e. a single neuron often activates in many different contexts (see our general response for the details). This confirms that focusing on the circuits is a productive unit of analysis for the interpretability analysis.

Q3. Discrepancy between Table 1 and Figure 3.

Thank you for pointing out this inconsistency in the manuscript. The ablations in Table 1 were run with 8 instead of 16 channels for the isoflop comparison (this difference exists only for legacy reasons). We will clarify this in the paper.

Q4. PEER baseline results.

Indeed the biggest difference in our setting to the one considered by the original PEER paper is that we replace all feedforward layers in the transformer with (separate) PEER layers rather than only a single layer and that we compare to a different, possibly stronger MoE implementation (we use the MoE from GPT-OSS). We use PEER without query BatchNorm since the original paper reports minimal differences in terms of language modelling perplexity and this avoids leaking non-causal information of future time steps during training, we will clarify this in the paper. The ablation you suggested of matching the number of neurons between PEER and sgatlin at the same depth establishes that differences are not simply due to PEER having fewer neurons.

Q5. Clarification of the experimental setup.

Thank you for pointing out these two typos in the manuscript: We will remove the wrong statement that sgatlin removes “the softmax nonlinearity applied to the gating scores”, and will clarify in Appendix A.1 that the query projection is shared across channels (not layers, that was a typo). The isoflop comparison on Slimpajama applies no softmax nonlinearity to the gating scores but the ClimbMix results do. The TinyStories model and the Climbmix model we use for the revised feedforward circuit analysis reported in our general response use sgatlin with 16 channels and apply the softmax nonlinearity to the gating scores. We will clarify this in the paper.

Reviewer_CXCaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Based on PEER, this paper presents a scalable ultra fine-grained sparse expert network, which takes one neuron as an expert, removes expert activations, and shares the query projection. From the experimental results, this method shows scalable properties compared with other baselines with better downstream task performance and faster inference throughput.

Critical strengths
  • Better reported ppl across four compute budgets.
  • An interesting attempt to scale the ultra fine-grained experts, which pushes the boundary of extreme small experts.
Critical weaknesses
  • In sparse MoE, sparsity is a critical factor that affects performance, and there’s no corresponding controlled experiments to investigate this in sgatlin, and we don’t know if the performance gain comes from granularity or sparsity.
  • For softmax gating, there’s no way the neuron could be trained in balance if we don’t take special actions to control this. Counting how many neurons are activated at least once is not enough.
  • The training budget is relatively small (1e17/1e18), and there’s no given estimated scaling laws, which does not provide sufficient evidence to support its scaling abilities.
  • The paper does not include an ablation specifically isolating the shared vs. unshared query projection.
Questions for authors

see weaknesses above.

2 comments

Thank you for taking the time to review our work which has resulted in several additional experiments that we think improve the paper and that we hope address your concerns. Please kindly take a look at the general response in which we present these additional results. In the following we will respond to the points raised one-by-one.

Weakness 1: Sparsity and granularity ablation

Thank you for this suggestion, indeed it is an interesting question how sparsity relates to compute-optimal performance. In Figure 1, we demonstrate the relationship between compute sparsity (the ratio of parameters per FLOP) and performance compute-optimal language modelling performance across feedforward architectures. But of course it is similarly interesting to understand how sparsity affects performance within sgatlin. Following your suggestion, we investigate the effect of different levels of granularity at different sparsity levels. In sgatlin, the number of activated neurons is determined by the number of selected neurons per channel, kk, times the number of channels, CC. Since queries in the gating network are shared across channels, the number of channels effectively corresponds to the granularity. We run an additional ablation where we vary kk and CC for two different numbers of total active neurons (k×Ck \times C) on a compute budget of 6e17 FLOPs on the ClimbMix dataset and report the resulting Validation BPB (bits per byte):

Active neurons1 channel4 channels16 channels
1280.91830.89100.8857
2560.93160.88660.8767

We find that having multiple channels, i.e. higher granularity, improves performance. Increasing the total number of active neurons only slightly increases performance. This is in line with a similar experiment conducted by PEER [1], the prior work sgatlin builds on, finding diminishing returns for further increasing the total number of active neurons (see their Figure 3b).

[1] He (2024), Mixture of A Million Experts

Weakness 2: Heterogeneous neuron activation and expert balancing

Thank you for raising this possible point of confusion, we will clarify this point in the paper: In Figure 9, we show that almost all neurons are being used by the model in general, i.e. there are almost no dead neurons (center panel) while activation frequency is clearly heterogeneous (right panel). We agree that in classical MoE architectures, expert balancing is important to achieve a high device utilization since experts are typically sharded across devices [e.g., 2,3]. This is however not directly applicable to sgatlin where individual experts reduce to single neurons that are not sharded across devices. A clearer analogy for this case is to think of sgatlin as a kind of ReLU MLP with a fixed sparsity: For each input only a subset of the neurons is activated. Heterogeneous activation in MLPs like we observe can reflect useful specialization [4,5] and even be used to improve inference efficiency [6].

[2] Zhou et al. (2022), Mixture-of-Experts with Expert Choice Routing

[3] Wang et al. (2024), Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

[4] Voita et al. (2024), Neurons in Large Language Models: Dead, N-gram, Positional.

[5] Geva et al. (2021), Transformer Feed-Forward Layers Are Key-Value Memories

[6] Song et al. (2024), PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU

Weakness 3: Training scale

We would like to clarify that we use the training budgets (1e17, 6e17, 1e18, 6e18 FLOPs) to conduct a comprehensive isoflop comparison which requires training each architecture across several model sizes / token budgets per FLOP budget. This is a compute-intense experiment which by itself consumed thousands of H100 GPU hours. We find that across compute budgets, sgatlin achieves better compute-optimal performance (Figure 3). We then conduct a larger pretraining run on a budget of 1e19 FLOPs on a separate dataset (Figure 4), confirming the improved language modeling perplexity is reflected in downstream performance.

Weakness 4: Missing ablation of shared query projection.

Thank you for making us aware that this ablation is currently not sufficiently clearly described in the paper. Sharing vs. not sharing the query projection is the difference between the router used by sgatlin and the router used by PEER. When we replace sgatlin’s router with the PEER router we run exactly this ablation. The result is reported in Table 1: not sharing the query as done by PEER slightly decreases performance.

Dear Reviewer CXCa, thank you again for your constructive feedback. With the end of the discussion period coming closer, we wanted to kindly ask whether our additional experiments and clarifications have helped to address your concerns? Please let us know if you have any further questions!

Reviewer_6z15OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper introduces a feed-forward layer, sgatlin, built from a big pool of single-neuron linear experts. A cheap gating network chooses a small number of neurons per token, and the neurons have no activation function, so for a given token, the layer behaves as a small linear map. In compute-matched language modeling experiments, sgatlin outperforms both dense and mixture-of-experts baselines, and the inclusion of a nonlinearity in sgatlin adversely affects performance. A case study on TinyStories shows that gating patterns have some semantic organization and can be influenced to probe their causal roles.

Critical strengths

-Easily interpreted by design, at no performance cost. Designs for interpretability usually trade-off model performance. Here, sgatlin is on par with standard feedforward layers for compute, and better for some smaller architectures. This weighs most heavily in my assessment: having a layer which is efficient and easy to analyze, increases the chance that it will be adopted.

-A clear structure for analysis. For a given token, feedforward computation is a small linear map, and which circuit is used is determined by gating. This makes circuits easy to compare and/or intervene upon without the need to train a replacement model. The observation that introducing non-linearities hurts is surprising and supports the case for design.

-The feedforward circuit analysis it felt creative and well executed too. Although I always have doubts about autointerp approaches (see weaknesses).

Critical weaknesses

-The case study focuses on a single small model, and no other models are compared. So, it is still unclear if sgatlin is easier to interpret than other models (e.g., a fine-grained MoE or a dense model) with transcoders or similar architectures. This analysis is at the level of gating patterns. The individual neurons are not analyzed, and the gating network (which does the nonlinear computation) is not analyzed. Interpretability is one of the two main contributions of this paper, so this is my biggest issue with the paper.

-The evidence for the efficiency claim is also thin. The comparison is on a small scale perplexity, and all the models use the same hyperparameters which were tuned on a dense baseline. The downstream evaluation is also done for a single setting. Reporting wall-clock time and memory costs of training are not done. The authors also claim that sgatlin outperforms the baselines; however, this is not fully justified with the evidence provided.

-Misleading autointerp scores. As mentioned, automatic interpretability scores can easily be manipulated or missleading. High accuracy in this case doesn't necessarily mean that the interpretations are correct. The near-perfect accuracy results in Section 4.2 are also misleading because they are reported without a baseline. For example, in Figure 7, the interpretation displays a single token, and the judge is able to identify the correct answer because it is the only option containing that token. If this is the norm, the quiz largely becomes a token matching exercise and I'd guess most architectures would likely pass the quiz.

Questions for authors

-(regarding weakness 3) I'd suggest a human review of sample explanations, baseline runs on other architectures, and the presence of distractors that contain the same token, but in a different role, would make the result more convincing.

-How should the gating network be interpreted? With the nonlinearity moved into the router, much of the computation now happens there. Is the routing decision itself understandable?

-This paper made me wonder about sparsity or linearity? Some work show weight-sparse transformers (https://arxiv.org/abs/2511.13653) also produce self-interpretable circuits and weights, through sparse weights rather than sparse routing. Wondering the authors thoughts on this.

5 comments

Thank you for your actionable feedback on our paper. Given your suggestions, we have run several additional experiments/analysis to address your concerns. Please kindly consider our general response in which we summarize the additional results. We will answer your specific questions point-by-point below.

Weakness 1: Extending the feedforward circuit analysis

Thank you for this suggestion. In our general response, we extend our feedforward circuit analysis to other architectures to make it easier to appreciate the semantic regularity we find for the feedforward circuits of sgatlin. Our results here demonstrate that the semantic regularity we find for sgatlin circuits is not found for individual neurons (e.g. in an MLP, sgatlin or PEER). For the coarse-grained MoE, we find that both the routing patterns and individual experts are semantically also noticeably less structured. For the fine-grained MoE, PEER, on which sgatlin builds, we find that its sparse circuits are similarly semantically structured. This shows that sparse routing creates a semantically structured space. The linearity of sgatlin’s circuits (but not PEER’s) is an additional theoretical property that opens interesting future research directions such as tracing an entire transformer forward pass as a single linear map. We will make this distinction of how sparsity and linearity affect interpretability more clearly in the paper.

Given your suggestion, we furthermore scale the feedforward circuit analysis to a larger model pretrained and evaluated on ClimbMix rather than Tinystories, as detailed in our general response. Our findings replicate in this setting, with highly semantically structured feedforward circuits in this model with 20x more parameters.

Weakness 2: Efficiency claim and scale

We would like to kindly clarify that an isoflop comparison as we conduct in this study requires significant computational resources: For each compute budget and for each feedforward architecture we sweep the model size and number of training tokens. The total wall-clock time for running the isoflop comparison alone took 2,400 H100 GPU hours. Following prior work [1,2,3], we invest this compute budget into an isoflop comparison rather than, e.g., running larger models with a fixed size, to ensure a fair comparison across architectures. Across this comprehensive analysis we find that sgatlin outperforms the baselines. We tune hyperparameters on a dense baseline to ensure we do not unfairly bias the results to our architecture but generally speaking the chosen hyperparameters reported in Appendix B represent fairly standard choices (e.g. compare with [4]).

To address your concern that the downstream evaluation has only been done in a single setting, we pretrained the compute-optimal models identified in our isoflop comparison on the 1e18 FLOP budgets on ClimbMix, confirming that the perplexity advantage translates into downstream performance in this setting as well.

Feedforward architecture1e+18 FLOPs1e+19 FLOPs
MLP0.09100.1592
SwiGLU0.10060.1727
MoE0.11700.1944
PEER0.11910.1879
sgatlin0.13130.2028

[1] He (2024), Mixture of A Million Experts

[2] Hoffmann et al. (2022), Training Compute-Optimal Large Language Models

[3] Krajewski et al. (2024), Scaling Laws for Fine-Grained Mixture of Experts

[4] Wortsman et al. (2023), Small-scale Proxies for Large-scale Transformer Training Instabilities

Weakness 3: Misleading autointerp scores. + Question 1: human review

We agree that automatic interpretability methods need to be carefully scored to ensure that they reflect human interpretability. As mentioned in our response to Weakness 1 above, we follow your suggestion, and ensure that the distractor options in the circuit discrimination quiz contain the activating token, run the quiz across different architectures and conduct a human review of circuit explanations to confirm that the explanations are human-interpretable, achieving similar discrimination accuracy as the Gemini 3.8 Flash judge.

Question 2: Interpreting the gating network

The gating network which selects the feedforward circuits can be interpreted as querying a circuit lookup table for a given context: For a given input, the gating network applies a linear map to calculate a query which is compared to the keys corresponding to the neurons via a dot-product. Each neuron corresponds to a specific key. This part is linear and the nonlinearity is applied afterwards to select the top-k most similar keys. The routing decision can therefore be understood in terms of the subspaces spanned by the input queries and the neuron keys. For a given circuit (a subset of neurons), we can for example use the corresponding keys to find the query subspace that activates it.

Question 3: Prior work on interpretability of sparse models

Thank you for highlighting this related work, it is indeed highly relevant to our study, since it shows how sparsity can make a model inherently interpretable. While this work uses L0 regularization during training to achieve weight sparsity, sgatlin achieves weight sparsity in the feedforward layers by activating only a sparse subset of neurons. We will make sure to discuss this work in our related work section.

Dear Reviewer 6z15, thank you again for your thoughtful review. As the discussion period is nearing its end, we wanted to kindly ask whether our additional experiments and clarifications have helped to address your concerns? Please let us know if you have any additional questions!

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

We would like to thank all reviewers for comprehensively engaging with our work and providing clear and actionable feedback. To address shared concerns by multiple reviewers, we have run the following additional experiments and ablations.

Extending the feedforward circuit analysis

Other feedforward architectures and units of analysis

Reviewer 6z15 and Reviewer imp7 suggested running the feedforward circuit analysis with other feedforward architectures and also evaluating individual neuron interpretability in addition to the circuit-level we presented. Furthermore, they suggested modifying the distractor items in the circuit discrimination quiz (Figure 7) to contain the activated token in the context. The following table summarizes the discrimination accuracies across layers for the different architectures using different units of analysis for the same circuit discrimination quiz (chance accuracy is 20%):

ArchitectureUnit of analysisLayer 0Layer 1Layer 2Layer 3
MLPNeuron41.0%29.0%39.0%32.0%
MoERouting pattern58.0%83.0%61.0%70.0%
MoEExpert68.8%56.2%42.9%81.8%
PEERCircuit99.0%100.0%100.0%95.0%
PEERNeuron80.0%75.0%66.0%76.0%
sgatlinNeuron74.0%76.0%73.0%61.0%
sgatlinCircuit100.0%99.0%99.0%97.0%

We find that the sparse circuits in sgatlin (and the closely related PEER on which our work builds) are semantically much more regular than individual neurons (in sgatlin, PEER or an MLP) or than individual experts or the routing patterns in the (coarse-grained) MoE. We further unpack this result in our individual responses to the respective reviewers.

Scaling the feedforward circuit analysis

We then scaled up this revised pipeline to a 20x larger sgatlin model trained and evaluated on the ClimbMix dataset (1e18 FLOPs, 1.2B parameters, 3.5B tokens) confirming that the semantic regularity of the circuits extends beyond the TinyStories setting.

ArchitectureUnit of analysisLayer 0Layer 1Layer 2Layer 3Layer 4Layer 5Layer 6Layer 7Layer 8Layer 9
sgatlinCircuit98.0%100.0%100.0%100.0%100.0%100.0%98.0%99.0%98.0%94.0%

Human review

Finally, in response to Reviewer 6z15, we conducted a human review of circuit explanations for two example layers (100 questions each) on this scaled-up model to confirm that the explanations are human-interpretable, achieving similar discrimination accuracy as the Gemini 3.8 Flash judge:

ArchitectureLevel of analysisLayer 0Layer 9
sgatlinCircuit97.0%92.0%

Additional architecture ablations

Several reviewers asked for architectural ablations which we summarize in the following:

  • Sharing neurons across channels: Reviewer imp7 asked for an ablation of sgatlin in which the neuron pools are shared across channels (like in PEER) rather than having separate neuron pools per channel. We match the total number of available neurons to the number of available neurons in the non-shared case.
  • Matching total neurons in PEER: Reviewer imp7 asked to match the total number of available neurons in PEER (shared across channels) to the total number of available neurons across channels in sgatlin. This allows PEER to select from a larger pool of neurons per channel than sgatlin and results in PEER having slightly more total parameters due to its different router.

We ran these additional ablations on a compute budget of 6e17 FLOPs on the ClimbMix dataset and report the resulting Validation BPB (bits per byte):

Architecture                   Total neuronsTotal paramsActive paramsVal BPB
PEER                                 131,072      180.4M        47.2M  0.9262
PEER w/ total neurons matched                 524,288      587.2M        51.4M  0.9155
sgatlin w/ neuron sharing across channels      524,288      571.0M        35.1M  0.8913
sgatlin                              524,288      564.7M        28.8M  0.8855

These results confirm that not sharing neurons across channels is indeed beneficial and the general improvement of sgatlin over PEER goes beyond simply having more total neurons available.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.