- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes evaluation rules for the setting where a segment of computation in a deep or recursive network is replaced by a cheaper substitute. It argues that reporting how similar the substitute is to the original computation is not enough. Such experiments should also report end-to-end downstream task performance and OOD performance. On GPT-2, LoopFormer, and a synthetic prototype, the paper shows that similarity and task performance do not move in the same direction. The paper itself is not a methods paper and does not propose a new compression method.
Critical strengths
The paper puts a lot of effort into spelling out the boundaries and limitations of its claims, and this kind of candor deserves credit.
Critical weaknesses
- Fidelity fitting plus downstream benchmark evaluation is already what ReplaceMe and LLM-Streamline do, and using the reduced network trained from scratch as a control is also existing practice. What the paper actually does is closer to consolidating these evaluation practices.
- The writing puts a heavy burden on the reader. There are very few figures, the theoretical part feels disconnected from the experiments, and the paper introduces many self-defined terms that are used heavily before they are ever defined.
Questions for authors
Proposition 1 lists six conditions in the main text, but four of them are marked as vacuous, not applicable, or untested. After reading the whole paper, I still don't understand what role these conditions actually play in this work.