- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the effect of architecture (primarily the dense vs. MoE FFN choice, but also other things like width and model depth) on "component reliance". Component reliance is a concept introduced by the authors that measures a model's resilience to the deletion of a certain (type of) component; in this case, the authors are interested in the model's reliance on its FFNs. They measure this metric across small models trained on a few toy tasks and language modeling. They conclude that in general, models that use MoEs, and particularly those with more experts, are more resilient to FFN deletion. They perform various other ablations to determine which aspects of FFNs (e.g. their width) matter most. There is no clear conclusion, besides that FFN design choices affect the algorithm that the model learns, and its "component reliance".
Critical strengths
This paper uses causal interventions to show what models rely on, which is in general a solid technique in interpretability.
Critical weaknesses
Weak Motivation: It's not really clear what this paper is trying to prove. Component reliance isn't an intrinsically interesting thing to study; nobody is going around and ablating FFNs from transformer models, so we don't need to be robust to this. Moreover, to my knowledge, most efforts to sparsify models post-hoc by ablating whole components also fail. This seems true for this study, too; while ablating MoEs is less harmful than ablating their dense counterparts, doing either is harmful enough that it is best avoided.
Another argument, then, is that this paper provides a view into how models that achieve the same training loss implement different algorithms, in line with the grokking literature. But this paper is also weak from that point of view: if showing this was the point, then the paper could have provided a much finer-grained view into the models' algorithms. With a one or two layer transformer, it's pretty trivial to reverse-engineer by hand the algorithm that is being used, and show that it (or its implementation) differs depending on the model's architecture. This would be much more interesting and compelling evidence than a simple coarse-grained zero or mean ablation of whole components, which tells us little about how the model performs the task.
Weak Conclusions: With weak motivations and methods come correspondingly weak conclusions. The authors conclude that
Implications for model design. Expert count, top-k, shared experts, and active width may offer controls over learned reliance alongside their familiar effects on parameter count and inference cost. A concrete prediction for larger models is that widening an always-active FFN pathway will weaken the no-FFN advantage under comparable training conditions. Testing this would determine whether the controls studied here can inform larger-model design.
This is true, but not especially novel. We already knew that even models that share the exact same architecture can learn different algorithms, which generalize differently, based on the amount of training they undergo and other factors (see Nanda et al. on grokking, or Zhong et al.). It's unsurprising that larger architectural differences will lead to larger differences between the learned algorithms. That this manifests itself as differences in "component reliance" on FFNs does not seem especially relevant to me; this is not an important quality.
The authors also state that:
Implications for interpretability. Component roles should be reassessed after architectural or training changes, even when task accuracy stays similar. A fitted decoder can reveal recoverable information, while interventions test how the original model uses its components. Treating these as complementary questions helps distinguish information present in a representation from the solution actually learned.
But I'm not sure who this lesson is for. I don't think people claim that component roles will stay the same after architectural or training changes?
Questions for authors
- What was the intended contribution of this paper?
- Why should we care about FFN / component reliance?
- What does this paper contribute beyond past work that has shown how one model can learn multiple algorithms for a task?
2 comments
Thanks for this response! Ultimately this does not change my opinion - the answers to "what does the model learn during training?" that this paper provides are not concrete enough to be interesting to interpretability, nor actionable enough to be useful otherwise. I encourage the authors to work further on this submission to achieve one of these two goals, or another one.
Reply to nUJd
Thank you for raising the question of motivation. We agree that removing FFNs is not itself a practical goal. We use removal as a way to ask what the rest of the model learned during training. Two models can give the same answers with their FFNs present, yet differ sharply in what they can do without them.
Our contribution is to identify a specific training choice that produces this difference. In the paired Add-7 experiment, the models had the same architecture, initial weights, training batches, and training budget. Both solved the intact task perfectly. When expert assignments stayed fixed, accuracy after FFN removal was 25%. When assignments could change during training, it was 50%.
This complements work that identifies different algorithms for the same task. We do not identify two complete addition algorithms. What we show is that changing how tokens are assigned to experts changes how much the trained model depends on its FFN. The readout experiment explains why this distinction matters for interpretability, a new output head recovers nearly perfect answers from both dense and MoE models after FFN removal. A larger ablation drop therefore does not, by itself, mean the remaining representation lacks the answer.