acceptodds
Under review as a conference paper at ICLR 2027

Revealing Local Nonlinearity Amplification for Large Language Model Compression

Abstract

The growing computational cost of large language models (LLMs) has driven increasing interest in structured compression to reduce inference overhead. Existing structured compression methods primarily assess layer importance or redundancy, but rarely characterize how errors introduced by compression propagate through subsequent computation. We observe a consistent layer-wise pattern in LLMs: across inference activation distributions, layers exhibit distinct levels of local nonlinear deviation, with many being well approximated by affine mappings. Meanwhile, the response of downstream computation to layer outputs exhibits stable distributional patterns across layer positions. We therefore introduce Local Nonlinearity Amplification (LNA), a quantitative measure that combines a layer's local nonlinear deviation with the downstream response to its output change, providing a direct estimation of the degradation risk associated with layer compression. Guided by LNA-based layer selection, we propose ReLoNA, which adaptively determines an affine-operator replacement strategy based on each layer's local nonlinearity. After layer replacement, ReLoNA applies Operator-Aligned Recovery to reduce the approximation errors introduced by replacement and maintain consistency with the original computation. Across 18 benchmarks and multiple models, ReLoNA outperforms existing structured compression methods at 25% compression ratios, retaining 94% of dense-model performance on multiple-choice benchmarks and achieving better performance on generative benchmarks, with gains of up to 2 over existing methods. Moreover, LNA-guided pruning consistently yields substantially lower degradation than existing criteria under the same compression ratio.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Large Language Models, Model Compression, Structural Pruning, Layer Pruning, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AuHHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper proposed to use affine mappings to approximate and compress transformer layers. They derive the Local Nonlinearity Amplification, which is the product of the normalized local nonlinearity and the empirical response estimate. This produces ReLoNA, which optionally applies Operator-Aligned Recovery.

Critical strengths

I think the critical strength is the proposed method has decent theoretical motivation, even though the math is quite simple.

Critical weaknesses
  1. Writing is one of the main weakness. Some of the key terms are not really defined such as layer (it could we a MLP layer, a linear layer/matrix, the entire transformer layer), subnetwork, and layer compression (it seems to be affine approximation here while this term usually mean other things). Thus, notations in Sec 2.1 is either unclear or incorrect, like F is a layer, B is also a layer, yet G and B consist of F (so F is larger than B? the two layer mean different things?). Fig1, suggest that F is a sequence of layers, yet the writing is still confusing.

The math part like the propositions also do not provide much insight. Without further explanation, the proposition seems to be quite straightforward and I am not sure the importance of them.

  1. While in the strength part, I mention the method has decent theoretical motivation, the empirical observation seem to strongly undermine it. Fig 2 shows the quantities are rather monotone, so the method is simply mostly pruning later layers? A fixed compression ratio of 25% also feel in sufficient for the experiments. Plus, the proposed method might have a O(L^2) complexity while the main baseline only require O(L).
Questions for authors
  1. Could the authors clarify the notations and the terminology in Sec 2.1?
  2. How different is the layer selection method compared to simply selecting later layers or the existing layer selection method but do linear layer approximation instead of pruning?
Reviewer_nQhgOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ReLoNA, a framework for structured LLM compression through layer replacement and recovery. Its central metric, Local Nonlinearity Amplification (LNA), combines a layer’s local affine approximation error with an empirical estimate of downstream perturbation amplification to assess compression risk. ReLoNA uses this metric to select layers for compression and applies adaptive affine absorption or residual folding. It further introduces Operator-Aligned Recovery (OAR), which updates selected layers using output distillation, hidden-state alignment, and auxiliary objectives to mitigate compression-induced degradation.

Critical strengths
  1. The paper offers a clear and interpretable perspective on layer compressibility by combining local approximation error with downstream error amplification to assess compression risk.

  2. The authors conduct comprehensive experiments and evaluations across multiple dimensions, providing a thorough assessment of the proposed method’s effectiveness.

Critical weaknesses
  1. Without OAR, ReLoNA underperforms ReplaceMe on four of the six models; its performance improves after OAR is introduced. This suggests that ReLoNA’s final advantage may depend substantially on recovery training, while the current results do not sufficiently establish the effectiveness of LNA itself. Ablation studies with a matched recovery-training budget are needed to disentangle the respective contributions of LNA and OAR.

  2. When using the same data and unregularized affine least-squares fitting, directly fitting B(x)−xB(x)-x is mathematically equivalent to fitting B(x)B(x) and then subtracting the identity map. Moreover, under nondegenerate conditions, the optimal residual scaling coefficient is 1. Are the two proposed strategies therefore effectively equivalent under these assumptions? If they differ in practice, the authors should specify the additional constraints or implementation details responsible for the difference.

  3. The theoretical analysis assumes that an entire layer is replaced by a complete affine map. In contrast, the implementation modifies only the Q/K/V projections in the following layer, without making the corresponding replacement in the residual path. Consequently, the error bound in Equation (12) cannot be directly interpreted as a theoretical guarantee for the final compressed model. The authors should revise the analysis to reflect the actual implementation and experimentally quantify how this discrepancy affects model error and performance.

  4. The authors argue that existing methods primarily base pruning decisions on intrinsic properties of the structures being pruned, without adequately considering error propagation or its effects on the model as a whole. However, prior work, including BlockPruner [1] and LLM-Pruner [2], uses changes in global loss or model performance to assess pruning decisions. Although these methods may not explicitly model error propagation, they already account for the effects of structural modifications on overall model behavior. The authors should therefore narrow this claim and clearly distinguish between evaluating pruning effects through global objectives and explicitly modeling error propagation.

[1] Zhong, L., Wan, F., Chen, R., Quan, X., & Li, L. (2025). BlockPruner: Fine-grained pruning for large language models. In Findings of the Association for Computational Linguistics: ACL 2025 (pp. 5065–5080). [2] Ma, X., Fang, G., & Wang, X. (2023). LLM-Pruner: On the structural pruning of large language models. In Advances in Neural Information Processing Systems (NeurIPS).

Questions for authors

A minor question: Model compression seems more closely aligned with primary subject areas such as “Optimization” or “Foundation or Frontier Models.” Could the authors please clarify their rationale for selecting “Other Topics in Machine Learning” instead?

1 comment

I thank the authors for their patient responses during the rebuttal, as well as for providing additional experiments—including budget-matched recovery comparisons and end-to-end ablations—and for thoroughly explaining the policy differences induced by regularization. I appreciate the considerable effort and time invested in these revisions, which have partially addressed my concerns and questions. However, the authors still primarily emphasize the performance of the complete pipeline in their experiments, without validating the contribution of LNA layer selection. The authors also acknowledge a theoretical gap with respect to practical deployment. Therefore, I wish to maintain my rating of 2. Thank you once again for your careful and detailed responses.

Reviewer_o7CqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a structured LLM compression method addressing local nonlinearity and error amplification. It introduces LNA to rank layer compression risk by combining local affine error with downstream perturbation responses. Based on this ranking, ReLoNA adaptively selects affine absorption, residual folding, or retention according to local nonlinearity, and folds transformations into subsequent layers. Additionally, Operator-Aligned Recovery (OAR) updates boundary layers to recover model performance. Evaluations on six 7B-8B models across multiple tasks show the complete method outperforms baselines, whereas removing OAR underperforms ReplaceMe.

Critical strengths

1: An interpretable method for assessing compression risk. The method jointly considers a layer's local approximability and the amplification of approximation errors by the subsequent network. This provides a more comprehensive assessment than treating an individual layer in isolation.

2: Effective performance recovery via targeted localized updates. The evaluation covers multiple models and diverse task types, including both multiple-choice and generative tasks. The proposed method recovers a substantial portion of the dense model's performance. These results demonstrate the practical value of combining layer compression with localized recovery targeted at the compression sites.

Critical weaknesses
  1. Unexplained data inconsistencies in some experimental tables make the results difficult to verify: Tables 3 and 4 both report the Dense baseline for Llama2-7B, but the CS, CP, and SM scores differ. In particular, the CP scores are 38.89 and 45.19, respectively, while AVG is 47.99 in both tables.

  2. The rationale for distinguishing the two adaptive replacement strategies and their independent benefits is insufficiently established: The paper applies Affine absorption to layers with low local nonlinearity and Residual folding to layers with intermediate local nonlinearity. However, I consider both resulting transformations to be affine mappings. Therefore, the claim that residual folding is more suitable for layers with intermediate nonlinearity does not follow directly from the current definitions. I would like to see a proof or ablation experiments clarifying the necessity and benefits of the adaptive branch design compared with a single replacement strategy.

Questions for authors
  1. Data consistency between Tables 3 and 4: Both tables report the Dense baseline for Llama2-7B, but the CS, CP, and SM scores differ. In particular, the CP scores are 38.89 and 45.19, respectively, while AVG is 47.99 in both tables. Please explain why these differences occur.

  2. Necessity of the adaptive replacement strategy: Affine absorption and Residual folding both ultimately produce affine mappings. Why is Residual folding more suitable for layers with intermediate local nonlinearity than directly using Affine absorption? Please clarify the roles of regularization and residual scaling, and provide analysis or ablation experiments supporting the benefits of choosing the replacement branch according to the degree of local nonlinearity.

Comments on the paper

1 comment

I thank the authors for the responses.

  1. Writing Issues

To me, the writing issues require careful revision instead of just simple clarifications. Like F_i is a layer, while F is a chain of suffix (that depends on B, but do not have B as a subscript). In retrospect, F_i does not really seem to be used meaningfully in that paragraph...

The use of proposition remains weird after the authors explanation. Currently, it still appear to me that the proposition is straightforward from the definition, so it is better merged with the definition like changing equation 5, 7 to Definition 1,2, and then have the original proposition immediately following them as equations.

  1. An existing selector plus a linear approximation

If ReplaceMe is an existing linear approximation method, to me, this poses more problem to the writing of this paper. It should be discussed more earlier in the paper. In the current form, this paper gives the feeling that they are the first to propose linear approximation for pruning.

  1. Simply selecting later layers

The authors additional results are appreciated and partially resolved my concern. Still, the method appear to be selecting later layers, and the overlap might increase or decrease depending on the selection ratio. One experiment the authors can potentially do is to conduct one-layer replacements, for each of the layers selected by your method, iterating all one-layer substitutions and see how close is the proposed method to the best performing one (So basically consider all sets of candidate layers that is only differ by two to your proposed set). This is more or less just an advice to strengthen the paper, not necessarily a big concern that the authors have to address.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.