- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes WaLeM, which is an LLM depth compression technique that takes inspiration from Neural ODEs. The method finds similar adjacent layers via CKA which corresponds in the ODE case to "smoothness." It then uses dynamic programming to determine which blocks to combine, and combines blocks via best-fit coefficients between these layers before merging. LoRA fine-tuning is applied to heal the model when fine-tuning to match predictive distributions, and fit the coefficients.
Critical strengths
- The layer-merging method seems to have strong performance compared to baselines. However, it is not clear to me which methods are training free and which include recovery training which affect the comparison present.
Critical weaknesses
-
Presentation: The writing in this paper is dense, convoluted, and uses wording that does not unambigously delineate what is actually done in the method. It reads as if it was completely written by AI, which is not disclosed in the AI statement which just points to polishing. There are several instances of overly-complicated, undefined terms that are not provided with sufficient context ("component-wise quadrature", "fine-layer parameter matrix", "micro-level approximation errors").
-
Connection to ODEs: The paper as it stands is a highly empirical method of using CKA, Dynamic programming, and best-fit coefficients to perform layer merging. I find the use of the ODE framing to detract from the paper, including the error bound which is not very interpretable or useful to know given the empirical nature of the rest of the paper. I think a clearer straightforward description of the method for what it is would have strengthened this paper significantly.
Questions for authors
- line 323 talks about released code, but I cannot find a pointer or link to the code. Is it available?