- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The authors propose to distill models trained on several tasks, including modular addition, into student models and analyze the features of the students. In the case of modular addition, the teachers are confirmed to rely on input embeddings whose fourier frequency sets (derived from all number embeddings) are required to perform the models computation. They then ask whether the students use the same frequencies. Three types of distillation are done, direct output distribution, hidden state matching, and using direct prediction outputs (for DFA trained teachers) from the teacher as training data for the students.
The main empirical contribution of the work is showing that the mass distribution on incorrect answers impacts what frequency features the student number embeddings use. This is shown by taking two teachers, A and B, and combining their distributions such that the rescaled wrong answer distribution from B combined with the right answer probability for A leads to a student model whose frequencies resemble those of B. And they show that deterministically shuffling the probabilities of the non-correct answer can predict what fourier features a student picks up in the number embeddings. Subtracting those frequencies leads to a performance drop.
Critical strengths
The core experiments seem reasonably clear and comprehensive as described in the summary, confirming that for an otherwise performing model that the frequencies associated with the tail distribution (bar the mass on the correct answer) have a causal relationship with the choice of embedding frequencies. Connecting the distillation itself, i.e. the output distribution of the parent model with the finding that the internal frequencies of the trained student model match those of the parent is sound and interesting.
Critical weaknesses
The paper could be better written in many places. In particular the abstract immediately stands out as particularly opaque to someone not already familiar with the work. There is also a bit of an overstatement (“A trained network solves this task by rotation”) in terms of asserting that modular addition always happens with the same procedure (a paper is even cited where this is not the case, Zhong et al.)
Questions for authors
None