acceptodds
Under review as a conference paper at ICLR 2027

WaLeM: Weight-Aware Learnable Merging for Large Language Model Depth Compression

Abstract

The residual connections in modern Large Language Models (LLMs) can be formulated as forward-Euler discretizations of continuous-depth dynamical systems. From this Neural ODE perspective, heuristic depth-compression techniques—such as layer pruning and merging—mathematically correspond to step-size coarsening operations over discrete trajectories. However, existing methods implicitly apply uniform coarsening schemes that ignore the heterogeneous smoothness of distinct Transformer operators, thereby incurring unmitigated local truncation errors. To address this, we introduce WaLeM, a principled, smoothness-aware operator-split coarsening framework for LLM depth compression. Specifically, WaLeM estimates local trajectory smoothness via Centered Kernel Alignment (CKA) and employs dynamic programming to construct an adaptive temporal grid that minimizes a global truncation-error surrogate. Furthermore, it learns component-wise quadrature weights to strictly eliminate operator-coupling errors. Theoretically, we establish an end-to-end approximation error bound of and demonstrate that our scheduling strategy optimally minimizes this bound. Across five mainstream LLM families, WaLeM outperforms state-of-the-art pruning and merging baselines with or without post-compression recovery, isolating the gain to the coarsening itself rather than to subsequent fine-tuning. Notably, it exhibits exceptional robustness on long-chain reasoning tasks (e.g., GSM8K), where early-stage discretization errors typically compound exponentially. Our work formally bridges heuristic layer reduction with rigorous numerical analysis, establishing a unified design space for continuous-depth LLM compression.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Compression, Layer Pruning, Merging

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_t2soOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes WaLeM, which is an LLM depth compression technique that takes inspiration from Neural ODEs. The method finds similar adjacent layers via CKA which corresponds in the ODE case to "smoothness." It then uses dynamic programming to determine which blocks to combine, and combines blocks via best-fit coefficients between these layers before merging. LoRA fine-tuning is applied to heal the model when fine-tuning to match predictive distributions, and fit the coefficients.

Critical strengths
  1. The layer-merging method seems to have strong performance compared to baselines. However, it is not clear to me which methods are training free and which include recovery training which affect the comparison present.
Critical weaknesses
  1. Presentation: The writing in this paper is dense, convoluted, and uses wording that does not unambigously delineate what is actually done in the method. It reads as if it was completely written by AI, which is not disclosed in the AI statement which just points to polishing. There are several instances of overly-complicated, undefined terms that are not provided with sufficient context ("component-wise quadrature", "fine-layer parameter matrix", "micro-level approximation errors").

  2. Connection to ODEs: The paper as it stands is a highly empirical method of using CKA, Dynamic programming, and best-fit coefficients to perform layer merging. I find the use of the ODE framing to detract from the paper, including the error bound which is not very interpretable or useful to know given the empirical nature of the rest of the paper. I think a clearer straightforward description of the method for what it is would have strengthened this paper significantly.

Questions for authors
  1. line 323 talks about released code, but I cannot find a pointer or link to the code. Is it available?
Reviewer_67AcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes WaLeM, a depth-compression method for LLMs built on the observation that a residual block is one forward-Euler step of a continuous-depth ODE, so removing or merging blocks amounts to coarsening the integration grid and incurs a truncation error that existing heuristic methods neither measure nor control. The method has two parts. A dynamic program selects which consecutive layers to merge, using CKA between layer representations as a stand-in for local smoothness and penalizing long intervals and shallow depths; within each selected interval, the weight matrices of each operator type (attention projections, MLP gates) are averaged with separately learned coefficients fitted to reproduce the dense model’s token-level predictions, with an optional LoRA stage afterwards. Across Llama2/3 and Qwen2.5 models at roughly 20–30% depth reduction, WaLeM reports lower perplexity and higher downstream accuracy than pruning and merging baselines, with the clearest separation on GSM8K; a controlled experiment on Llama3-8B suggests the gain survives when every method receives the same recovery budget, though by a smaller margin.

Critical strengths
  1. The method separates the two decisions that depth compression involves, where to merge and how to combine the merged layers, and ablates each against sensible controls: the DP schedule against Block-Influence ranking and fixed-interval placements (Tables 3, 9), and the per-operator learned coefficients against the LaCo and MKA merge rules under an identical merging scope (Table 4). The CKA proxy that drives the schedule is also checked against direct Jacobian norms (Table 8), so the central modelling assumption is not left unexamined.
Critical weaknesses
  1. The numerical-analysis framing contributes little beyond vocabulary. The quadrature term in Proposition 1 rests on an unproven assertion that an endpoint CKA threshold bounds the variation of the vector field inside the interval; the bound does not depend on the learned coefficients, since it uses the worst-case weight deviation; the e^{L_f L} factor is vacuous at L = 32; and “operator splitting” and “deferred correction” denote per-matrix-type averaging and LoRA fine-tuning. The claim that the DP schedule “optimally minimizes this bound” is not established, as DP minimizes a surrogate of a different form.
  2. The main comparison is not conducted under a uniform protocol. In Tables 1-2 WaLeM includes LoRA recovery while the LaCo, ShortGPT and MKA entries coincide with their no-recovery values in Table 7, and LLM-Pruner appears with its own official fine-tuning; the controlled comparison covers a single model and ratio, where WaLeM’s advantage under matched recovery is about one point and its average without recovery is below LaCo’s. The headline margins in the main tables therefore overstate what the controlled experiment supports.
Questions for authors
  1. The quadrature term in Proposition 1 relies on the step “the highly aligned adjacent representations imply sup_l ‖f_l − f̄_k‖ ≤ (1−γ)M_f”, which is asserted with citations rather than derived; CKA is computed between the interval endpoints on calibration tokens, whereas the inequality concerns the vector fields inside the interval at a fixed state. In addition, the bound uses the worst-case weight deviation and therefore does not depend on the learned coefficients α, and the DP minimizes a surrogate of a different form from the bound. Could the authors provide a derivation of the CKA-to-vector-field inequality under stated assumptions, and clarify in what sense the schedule “optimally minimizes this bound” as claimed in the abstract?
  2. The main tables do not appear to follow a uniform protocol: at 25% on Llama3-8B, the LaCo, SLEB and MKA entries in Table 1 coincide with their no-recovery values in Table 7, LLM-Pruner appears with its official fine-tuning, and WaLeM is reported with recovery at a different operating point from the matched-recovery row of Table 7. Could the authors state, for every baseline in Tables 1, 2 and 5, whether post-compression recovery was applied, and report the full set of models and ratios under both conditions of Table 7?
  3. Table 7 reports WaLeM’s no-recovery average on Llama3-8B at 25% as 53.53, while Tables 9 and 10 report 46.98 for the same schedule and the same perplexity of 37.01; the Table 9 caption refers to a seven-task average but the Table 10 text refers to eight tasks. Could the authors reconcile these figures, and also explain why SLEB and LLM-Pruner are omitted from the Qwen experiments and why FuseGPT, identified in Appendix A.2 as the closest concurrent method, is not compared?
Reviewer_Uhc8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes depth compression by first selecting adjacent layer groups using a CKA-based dynamic-programming objective, then learning separate convex merging coefficients for Transformer parameter matrices through output-distribution distillation. Optional LoRA recovery is applied afterwards. The method is interpreted as adaptive coarsening of a continuous-depth system, with a claimed approximation bound. Experiments cover five Llama/Qwen backbones, language modeling, downstream accuracy and GSM8K, with additional schedule, recovery and computational-cost studies.

Critical strengths
  1. The combination of a global layer-group search with low-dimensional, component-specific coefficient learning is practically attractive. Tables 1, 2 and 5 report useful compression results across five Llama/Qwen backbones and several compression levels, supporting the empirical value of the pipeline.

  2. The appendix includes substantial relevant controls: Table 7 reports both no-recovery and matched-LoRA comparisons; Table 9 tests fixed schedules with an identical fusion stage and recovery disabled; and Tables 12-13 disclose the extra compute. At 25% reduction on Llama3-8B, matched-recovery WaLeM reaches 59.59 average accuracy versus LaCo's 58.60, with similar PPL (20.25 versus 20.30). Without recovery it retains comparable average accuracy to LaCo (53.53 versus 53.77) with lower PPL (37.01 versus 89.62). These are meaningful controls for the practical method, although they do not isolate every proposed mechanism.

Critical weaknesses
  1. The theoretical guarantee has not been connected to the implemented parameter-merging rule. Equation (4) constructs one convexly averaged matrix for each component, whereas Equation (10) analyzes a weighted sum of complete fine-layer residual functions. Footnote 1 supplies a block-size factor, but does not justify moving parameter averaging through nonlinear attention/MLP computations, especially with distinct coefficients for each matrix. The proof also invokes small-residual/high-alignment conditions on p.17; I acknowledge these qualifications, but they are not specified quantitatively and do not establish the required absolute vector-field variation bound from CKA. Appendix A.7 is useful empirical support for a scheduling proxy, as Appendix A.1 itself recognizes, rather than a proof of that inequality. Finally, the DP correctly optimizes the additive surrogate in Equation (3), but this is not shown to minimize the stated end-to-end bound. These issues require either a justified bridge and explicit assumptions or a clear restriction of the theoretical claims; they do not invalidate the reported empirical compression results.

  2. The component-specific mechanism is less isolated than the text claims. Table 4 usefully fixes the merge scope, but compares learned coefficients with heuristic MKA/LaCo coefficients, so the effects of learning and coefficient granularity remain combined. Table 9 controls the schedule, and Table 7 controls LoRA recovery; neither varies coefficient granularity under the same coefficient-learning budget. A learned shared-coefficient control would test this attribution directly. In its absence, the claims of necessity and strict elimination of operator-coupling errors should be narrowed. This is a limitation on the mechanism claim, not evidence that the practical pipeline is ineffective.

Questions for authors
  1. Can you specify the exported merged-block forward computation, including block-size scaling and normalization, and either justify the Equation (4)-to-(10) connection and the quantitative extra condition needed for the p.17 CKA inequality, or explicitly restrict Proposition 1 to an abstract surrogate model? Please distinguish the valid DP optimum for Equation (3) from an optimum of a proved error bound.

  2. Under the same groups, initialization, calibration data, coefficient-learning steps and recovery budget, how does learning one shared coefficient vector per merged layer compare with the component-specific vectors? If this control is unavailable, which mechanism claims can the existing Tables 4, 7 and 9 support without it?

  3. Please clarify the aggregation/configuration used in Tables 6 and 9. For Qwen2.5-14B with PPL 11.95, Table 2 reports 59.87, whereas Table 6 reports 55.65, which matches including GSM8K in an eight-task mean. The DP row with PPL 37.01 is 53.53 in Table 7 but 46.98 in Table 9, whose caption specifies the seven-task definition. Identical rounded PPL alone does not establish identical configurations; per-task results and corrected aggregate definitions would make these controls interpretable.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.