acceptodds
Withdrawn from ICLR 2027

Design and Training Shape Component Reliance in Small Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Architectural choices can change which components a Transformer learns to rely on. We compare dense and mixture-of-experts (MoE) feedforward networks (FFNs) on three algorithmic tasks and character-level language modeling. MoE models retain higher accuracy after FFN removal on carry-based addition and TinyStories. This advantage persists in deeper models on both tasks and in wider TinyStories models. Controlled one-layer addition experiments show that increasing expert count at fixed total FFN width improves no-FFN performance and increases relative attention attribution, while intact accuracy remains near perfect. We identify restricted active capacity and evolving expert assignments as contributing factors. Critically, allowing assignments to evolve increases no-FFN performance even when router weights remain frozen. Dropout and shuffled assignments further show that training conditions can change reliance. The effect varies across tasks and training stages: the gap grows with continued training in one-layer modular addition, while histogram counting shows little separation. Fitted-readout comparisons distinguish recoverable answer information from its use by the trained model, helping interpret these differences. Together, these findings suggest ways to steer component reliance through FFN design and training.

Primary area:
interpretability and explainable AI
Keywords:
mixture-of-experts, learning dynamics, component reliance, transformers

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nUJdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the effect of architecture (primarily the dense vs. MoE FFN choice, but also other things like width and model depth) on "component reliance". Component reliance is a concept introduced by the authors that measures a model's resilience to the deletion of a certain (type of) component; in this case, the authors are interested in the model's reliance on its FFNs. They measure this metric across small models trained on a few toy tasks and language modeling. They conclude that in general, models that use MoEs, and particularly those with more experts, are more resilient to FFN deletion. They perform various other ablations to determine which aspects of FFNs (e.g. their width) matter most. There is no clear conclusion, besides that FFN design choices affect the algorithm that the model learns, and its "component reliance".

Critical strengths

This paper uses causal interventions to show what models rely on, which is in general a solid technique in interpretability.

Critical weaknesses

Weak Motivation: It's not really clear what this paper is trying to prove. Component reliance isn't an intrinsically interesting thing to study; nobody is going around and ablating FFNs from transformer models, so we don't need to be robust to this. Moreover, to my knowledge, most efforts to sparsify models post-hoc by ablating whole components also fail. This seems true for this study, too; while ablating MoEs is less harmful than ablating their dense counterparts, doing either is harmful enough that it is best avoided.

Another argument, then, is that this paper provides a view into how models that achieve the same training loss implement different algorithms, in line with the grokking literature. But this paper is also weak from that point of view: if showing this was the point, then the paper could have provided a much finer-grained view into the models' algorithms. With a one or two layer transformer, it's pretty trivial to reverse-engineer by hand the algorithm that is being used, and show that it (or its implementation) differs depending on the model's architecture. This would be much more interesting and compelling evidence than a simple coarse-grained zero or mean ablation of whole components, which tells us little about how the model performs the task.

Weak Conclusions: With weak motivations and methods come correspondingly weak conclusions. The authors conclude that

Implications for model design. Expert count, top-k, shared experts, and active width may offer controls over learned reliance alongside their familiar effects on parameter count and inference cost. A concrete prediction for larger models is that widening an always-active FFN pathway will weaken the no-FFN advantage under comparable training conditions. Testing this would determine whether the controls studied here can inform larger-model design.

This is true, but not especially novel. We already knew that even models that share the exact same architecture can learn different algorithms, which generalize differently, based on the amount of training they undergo and other factors (see Nanda et al. on grokking, or Zhong et al.). It's unsurprising that larger architectural differences will lead to larger differences between the learned algorithms. That this manifests itself as differences in "component reliance" on FFNs does not seem especially relevant to me; this is not an important quality.

The authors also state that:

Implications for interpretability. Component roles should be reassessed after architectural or training changes, even when task accuracy stays similar. A fitted decoder can reveal recoverable information, while interventions test how the original model uses its components. Treating these as complementary questions helps distinguish information present in a representation from the solution actually learned.

But I'm not sure who this lesson is for. I don't think people claim that component roles will stay the same after architectural or training changes?

Questions for authors
  • What was the intended contribution of this paper?
  • Why should we care about FFN / component reliance?
  • What does this paper contribute beyond past work that has shown how one model can learn multiple algorithms for a task?
2 comments
AuthorsOpenReview ↗

Reply to nUJd

Thank you for raising the question of motivation. We agree that removing FFNs is not itself a practical goal. We use removal as a way to ask what the rest of the model learned during training. Two models can give the same answers with their FFNs present, yet differ sharply in what they can do without them.

Our contribution is to identify a specific training choice that produces this difference. In the paired Add-7 experiment, the models had the same architecture, initial weights, training batches, and training budget. Both solved the intact task perfectly. When expert assignments stayed fixed, accuracy after FFN removal was 25%. When assignments could change during training, it was 50%.

This complements work that identifies different algorithms for the same task. We do not identify two complete addition algorithms. What we show is that changing how tokens are assigned to experts changes how much the trained model depends on its FFN. The readout experiment explains why this distinction matters for interpretability, a new output head recovers nearly perfect answers from both dense and MoE models after FFN removal. A larger ablation drop therefore does not, by itself, mean the remaining representation lacks the answer.

Thanks for this response! Ultimately this does not change my opinion - the answers to "what does the model learn during training?" that this paper provides are not concrete enough to be interesting to interpretability, nor actionable enough to be useful otherwise. I encourage the authors to work further on this submission to achieve one of these two goals, or another one.

Reviewer_J477OpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2(−1)
Read review
Paper summary

This paper studies how architecture and training change the roles of attention and feedforward layers (FFNs) in small Transformers. It compares dense FFNs with sparse mixtures of experts on algorithmic tasks and TinyStories. It removes FFNs and fits new output layers to test what information remains. The results link component reliance to active FFN size and changing expert assignments.

Critical strengths
  1. Table 15 gives clear evidence for a routing effect. It fixes the router weights and matches initialization, minibatches, and training budget, with no balancing loss. All five seeds favor changing expert assignments over fixed assignments. This adds useful evidence beyond earlier studies of component roles [1] and sparse expert training [2].
  2. Table 2 helps explain the ablation results. On add-7, a new output layer recovers over 99% accuracy for both models, compared with about 11% and 44% using the original readout. Low ablation accuracy does not always mean that answer information has been lost.
Critical weaknesses

The TinyStories comparison leaves model quality as a possible explanation. In Table 25, MoE starts roughly four percentage points below the dense model before ablation at both main widths. Measuring changes from each model's own baseline helps, but does not fully remove this difference. The result is less clear than a comparison between models with similar starting quality. This concern mainly affects the language-model extension.

Questions for authors
  1. Does the TinyStories result hold when dense and MoE checkpoints have similar accuracy before ablation?
  2. What data split and fitting procedure were used for the Table 2 output layers? How much do the gains vary across seeds?

References

[1] Behrens et al. Counting in Small Transformers: The Delicate Interplay between Attention and Feed-Forward Layers. 2025.

[2] Chen et al. Sparse MoE as the New Dropout: Scaling Dense and Self-Slimmable Transformers. 2023.

2 comments
AuthorsOpenReview ↗

Reply to J477

Thank you for highlighting the routing and readout experiments.

Q1. Does the TinyStories result hold at matched starting accuracy? We did not select checkpoints to match intact accuracy, so we cannot answer that directly. The closest comparison is at six layers, dense and MoE models reach 72.4% and 70.1% accuracy before FFN removal, but 1.9% and 8.0% afterward. MoE also shows a smaller loss increase relative to its own baseline, but that could also be a confounder to shore up.

Q2. How were the readouts fitted, and how consistent were the gains? We froze each model and trained only a new linear output head on the representation before the FFN. We split whole input sequences approximately 60/20/20 for fitting, validation, and testing. We chose regularization on validation data and compared both heads on the same test examples. These examples were held out from fitting the head, not necessarily from training the Transformer.

The new head improved accuracy in every seed. On Add-7, all fitted heads reached at least 98.5%, with gains of 84 to 93 percentage points for dense and 37 to 77 for MoE. On histogram, the gains were 56 to 76 points for dense and 78 to 85 for MoE.

Thanks for the response. Your answer to Q2 clears up the Table 2 readout setup. I also appreciate that the paper states its claims clearly and provides clean controlled tests.

However, after reading the comments from Reviewers RaTn and nUJd, I carefully rechecked the appendix controls and found that the contribution issues they raised are real. The Table 15 result only covers hard top-1 routing without router task gradients. Weighted top-2 replay is inconclusive (Tables 16-17), gated top-1 (Table 19) and shared experts (Table 10) shrink the gap, and untied random output heads already raise dense zero-FFN accuracy from 12.13% to 78.75% (Table 33).

Above all, I am lowering my rating from 3 to 2.

Reviewer_RaTnOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper explores the role and necessity of certain transformer components on various algorithmic and language modeling tasks. The authors experiment with removing the FFN component of MoE and dense transformer models, and record the retain performance of these models after ablations. They find that on multi-digit addition (Add-7) and language modelling (TinyStories), MoE models retain substantially higher performance after FFN removal compared to dense models. Diagnostic probes suggest that dense models still rely on the FFN to reformat some task information for the readout head, although the strength of this finding varies between tasks (modular addition and histogram counting).

Critical strengths

The experiments are thoroughly carried out and statistically sound. The figures are well presented, and sufficient detail is provided after the Abstract and Introduction to fully understand the motivation for each experiment.

Critical weaknesses

While the experiments are throughly carried out and largely well-motivated, the significance and usefulness of the conclusions are unclear. For instance, the first sentence of the conclusion “Changing the FFN changes how the trained model uses its components” is an obvious statement; it would be surprising if the reverse were true. Further, showing that “active capacity, changing expert assignments, and training conditions each matter” is also obvious. The authors themselves state that this is a “basic question” in their Introduction.

W1. Many of the tasks and terms used in the paper are not adequately explained. For instance, the Introduction introduces the term “component reliance” and refers to it as “the pattern” without defining this concretely. The Add-7 task is also not properly introduced…it is unclear what better/worse “performance” on Add-7 means in the Introduction. I understand that more details are provided in the Setup and the Appendix, but this hinders readability. Therefore it is challenging to understand what the authors are trying to show and what their conclusions are.

W2. Architectural choices are insufficient to fully test the claims. The authors train/test admittedly extremely small models (1-6 layers, with d_model=256 or less) on very simple tasks. More useful experiments and reliable conclusions could be drawn from larger pre-trained models on more real-world tasks (question answering, simple math, summarization, etc). As well, the models omit normalization, and when normalization is introduced in Appendix G.3, the authors state that the performance differences are compressed (although Figure 11b seems to show the opposite from the text in G.3, making things a bit unclear).

W3. There is some confounding between the architecture and the four tasks: causal vs bidirectional attention, tied vs untied readouts). Therefore it is challenging to isolate if the failure of the MoE pattern to appear on certain tasks is a result of the task itself or the experimental setup or architectural choices in each task.

W4. There is a hard top-1 routing scheme imposed on the MoEs…the selected gate is fixed to one. As a result of this choice in the experiments, task gradients are blocked from reaching the router, which may encourage the model to push task-relevant computations onto the upstream components like the attention block. Therefore it is possible that their task-relevance is inflated in the MoE-ablation settings compared to the dense network. Gated top-1 with router task gradients retains an MoE–dense advantage, but it is smaller. When the authors do test a top-2 weighted setting in the Appendix, the dynamic-versus-fixed expert membership difference is statistically inconclusive. Therefore the routing mechanism appears to play a large role in the results.

Questions for authors

Q1. Why did the authors choose not to use normalization for the transformer architectures?

Q2. What would the results look like if larger pre-trained models were tested?

Q3. Could the authors clarify how important the routing mechanism is for their results?

2 comments
AuthorsOpenReview ↗

Reply to RaTN

Thank you for the careful review. We agree that the introduction could define component reliance more clearly. On Add-7, we measure how many digits of x+7x+7 the model still predicts correctly after removing the FFN, without retraining. Increasing expert count at fixed total FFN width raises this accuracy from 10.5% to 89.3%, even though the intact models all solve the task nearly perfectly. That difference is invisible if we look only at intact accuracy.

Q1. Why omit normalization? We chose a simple architecture to study attention and the FFN together. Two related modular-addition studies also omitted LayerNorm: Nanda et al., §3 and Zhong et al., Appendix F.1. You are right about Fig. 11(b): on Add-7, the MoE advantage after FFN removal is larger with RMSNorm, not smaller as the text states.

Q2. What about larger pretrained models? We have not tested them. The advantage persists in deeper models on Add-7 and TinyStories, and in wider TinyStories models. We also agree that differences across tasks cannot be attributed to the task alone, since attention masking and readout tying can vary across settings. Both are held fixed in each within-task comparison.

Q3. How important is routing? The MoE advantage is smaller when the top-1 router learns from the task loss, but it remains positive at both expert counts tested. The result about changing assignments is more specific, allowing tokens to switch experts during training improves performance after FFN removal under hard top-1 routing. The corresponding weighted top-2 comparison is inconclusive.

I appreciate that the authors responded to each of my questions.

While I understand that the authors had to make some experimental simplifications, I still feel that these tradeoffs severely limit the utility and impact of the paper. The experiments are simply not comprehensive enough, and the implications of "component reliance" are barely explored. Indeed, based on the author responses to my comments, I do not have any additional insight beyond the initial paper.

Therefore I retain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.