- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
The paper proposed to use affine mappings to approximate and compress transformer layers. They derive the Local Nonlinearity Amplification, which is the product of the normalized local nonlinearity and the empirical response estimate. This produces ReLoNA, which optionally applies Operator-Aligned Recovery.
Critical strengths
I think the critical strength is the proposed method has decent theoretical motivation, even though the math is quite simple.
Critical weaknesses
- Writing is one of the main weakness. Some of the key terms are not really defined such as layer (it could we a MLP layer, a linear layer/matrix, the entire transformer layer), subnetwork, and layer compression (it seems to be affine approximation here while this term usually mean other things). Thus, notations in Sec 2.1 is either unclear or incorrect, like F is a layer, B is also a layer, yet G and B consist of F (so F is larger than B? the two layer mean different things?). Fig1, suggest that F is a sequence of layers, yet the writing is still confusing.
The math part like the propositions also do not provide much insight. Without further explanation, the proposition seems to be quite straightforward and I am not sure the importance of them.
- While in the strength part, I mention the method has decent theoretical motivation, the empirical observation seem to strongly undermine it. Fig 2 shows the quantities are rather monotone, so the method is simply mostly pruning later layers? A fixed compression ratio of 25% also feel in sufficient for the experiments. Plus, the proposed method might have a O(L^2) complexity while the main baseline only require O(L).
Questions for authors
- Could the authors clarify the notations and the terminology in Sec 2.1?
- How different is the layer selection method compared to simply selecting later layers or the existing layer selection method but do linear layer approximation instead of pruning?
I thank the authors for their patient responses during the rebuttal, as well as for providing additional experiments—including budget-matched recovery comparisons and end-to-end ablations—and for thoroughly explaining the policy differences induced by regularization. I appreciate the considerable effort and time invested in these revisions, which have partially addressed my concerns and questions. However, the authors still primarily emphasize the performance of the complete pipeline in their experiments, without validating the contribution of LNA layer selection. The authors also acknowledge a theoretical gap with respect to practical deployment. Therefore, I wish to maintain my rating of 2. Thank you once again for your careful and detailed responses.