- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper attempts to introduce a principled way to perform distillation of an attention layer into linear-attention variants, with particular focus on GLA / GDN / KDA linear recurrent cells. The procedure aims at initialising the newly-added student parameters specific to the recursion (gates, decay, write strength, and so on) with statistics coming from the teacher. The description of the exact mechanism is rather confused: the paper first establishes a connection between an online update view of softmax attention, approximated via Taylor expansion, and a delta-rule style RNN, but this does not seem to be directly leveraged for the initialisation, which instead relies on teacher statistics such as mean look-back and per-head entropy, although the underlying justification eludes me. This somehow forms phase 1 of the distillation procedure. Phase 2 then follows, which is standard per-layer distillation. The evaluation procedure covers ablations on Phase1 parameters initialisation, zero-shot scores for 4 teachers, recovery speed for distillation, and distillation a 975B MoE.
Critical strengths
- The problem is relevant
Critical weaknesses
- The paper in its current form, is extremely hard to parse, to the point that it hinders my ability to express a clear judgment on it
Questions for authors
- Clarity: The paper is extremely hard to parse, for a number of reasons. Necessary terminology is not properly introduced when needed, and must be either reverse-engineered from other parts of the paper or plainly guessed. Similarly with a number of mathematical quantities or acronyms introduced, which oftentimes are not properly described. Sentences are often borderline nonsensical, or employing obscure terms, at great cost to clarity. The paper lacks a coherent structure, failing to properly introduce the key aspects of the method. Figures, rather than helping highlighting the particularities of the method, are quite frankly uninformative. The fact that the paper keeps on switching terminology to refer to given components further adds to the confusion. In short, the paper is in severe need of a thorough rewrite.