acceptodds
Under review as a conference paper at ICLR 2027

CurvProbe: Damage Has Two Factors, and the Checkpoint Sets One

Abstract

Post-training compression first has to decide where: which layers to delete, factorise or quantise. The criterion the field deploys ranks layers by displacement, how far an operation moves a layer's output, and never asks how much that move raises the loss. Expanding the loss where these operations inject their change splits the damage at a layer into that displacement times the loss's curvature along it. Second-order salience already estimates this curvature, one operation at a time. That curvature belongs to the frozen checkpoint, not to the operation, so it can be read once, before any operation is named. CurvProbe reads it as a curvature profile, one number per layer from forward passes, with no backward pass and no Hessian. The factorisation predicts that displacement ranks layers below chance on checkpoints whose curvature is highest where displacement is lowest, as on Llama-3-8B, where the attention sublayer ranked safest to delete costs the most; the profile flags such failures before any damage is measured. One profile, never shown the operation it is judged on, repairs the damage ranking across eighteen checkpoints and eleven operations. On Llama-3-8B its layer sets keep that gain once a quarter of the layers are compressed. After low-rank approximation they hold 65.8% LAMBADA accuracy, against Block Influence's 55.8%. A checkpoint can therefore ship with its profile, and a compressor that arrives later ranks its layers with one forward pass where measuring their damage would take .

Primary area:
foundation or frontier models, including LLMs
Keywords:
model compression, layer selection, large language models, layer pruning, curvature, Hessian, quantization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_gDbcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper proposes CurvProbe, a layer-selection method for post-training compression. It estimates an output-space curvature profile using forward finite differences and combines this profile with operation-specific output displacements to predict layer damage. The intended benefit is to reuse the same curvature profile across different compression operations.

Critical strengths
  1. The empirical question is useful: local output displacement can be a poor indicator of the final loss increase, and testing whether a reusable output-space curvature profile improves this indicator is worthwhile. The Llama-3-8B example provides a clear motivation.

  2. The ranking experiments are substantial and include meaningful controls: the reported improvements on 53/55 core pairs and 121/128 extended pairs are supported by comparisons with shuffled profiles, depth priors, scale normalization, and each factor alone. The statistical analysis groups operations by checkpoint rather than treating all pairs as independent.

Critical weaknesses
  1. Notation and experimental terminology are not introduced where they are needed. Figure 1 uses quantities such as ϵℓ\epsilon_\ell and yℓy_\ell before their introduction in Section 3, and its caption does not adequately define them. Terms such as “cell” and the different experimental cohorts are also used before their meaning is established. Several headline correlations and percentages appear before the relevant variables, experimental units, and calculation procedures are sufficiently explained. As a result, I need to read the later sections and then I can read some figures at previous sections. The main text and figure captions need to be substantially more self-contained.

  2. The current manuscript is not ready to be a final submission for ICLR. At the appendix, there are internal verification records or responses to earlier feedback. Examples include “A reviewer noted...” in Appendix D.7 and “a reviewer was right...” in Appendix E.11, followed by accounts of what was changed. Similar passages recur elsewhere. Where does these reviewers come from?

  3. The inconsistencies affect substantive conclusions and presentation. Appendix D.10 provides a concrete example. Its corrected accounting reports a valid fp32 profile cost of 705.1s and a break-even point at the eighth operation when compared with bf16 direct measurement. The section also explains that the earlier 203.1s estimate relied on a bf16 curvature computation that does not produce a valid profile. Nevertheless, the same section retains the 203.1s cost and claims that the method is ahead from the third operation for deletion. Table D.9 also retains the earlier accounting. These are materially different conclusions about when the proposed method is worth using. I'm confusing about it.

  4. The decomposition into perturbation magnitude and curvature is established in prior work. In particular, HAWQ-V2 already combines squared perturbation magnitude with average Hessian trace in Eq. (5), extends its analysis to activations in Section 2.2, and compares trace-weighted sensitivity with perturbation-only sensitivity. From my perspective, I think the most plausible contribution here is the forward-only estimator at a common sublayer output site, together with the empirical study of reuse across compression operations. This should be the central claim. Moreover, the limitations of the particular HAWQ-V2-style weight-space score evaluated in this paper should not be presented as limitations of the full activation-space second-order framework. One can also compute the Hessian component before the target perturbation is chosen in either framework; the complete score requires the perturbation in both.

[1] HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks https://arxiv.org/abs/1911.03852

Questions for authors

Which configuration should be regarded as the primary method: reference-norm or clean-norm calibration, and isotropic or directional probing? Can you provide a single algorithm and a consistent set of headline results for that configuration?

1 comment

Thank you again for reading the revision so closely. The paper was updated once more on 14 November, with the three points from your follow-up corrected and the rest of the appendix and the supplementary appendix checked for the same kind of mismatch. If time allows, we would be grateful to know whether any concern remains, and we will address it promptly.

Reviewer_WiEdOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Existing layer selection methods for post-training compression typically rely solely on the local output change caused by compression; however, the same magnitude of displacement can lead to vastly different impacts on the final loss depending on the layer. CurvProbe decomposes "layer damage" into the product of an operation-dependent displacement and a checkpoint-dependent curvature, constructing a curvature profile for each layer in advance using a forward-only finite difference approach.

Critical strengths
  1. The experimental evaluation is very extensive, and the details are clearly presented.
  2. CurvProbe addresses a missing factor by introducing a precomputable curvature profile—reusable across different compression methods—which enables more accurate decisions regarding which layers to compress.
Critical weaknesses
  1. The motivation and the problem addressed are not particularly novel; in fact, an increasing number of papers are currently exploring the use of curvature. That said, the engineering implementation of the curvature profile is valuable.
  2. How does CurvProbe perform under extreme compression scenarios—such as ultra-low bit-widths, extremely low-rank approximations, or high pruning ratios? Is it particularly well-suited for such high-compression regimes? Furthermore, if the performance gap between different selection strategies is negligible when using methods like GPTQ or AWQ—which already achieve very low error—does CurvProbe still offer any tangible benefit?
  3. For larger models (e.g., 32B/70B parameters), how does the cost of loading/accessing the curvature profile compare to the cost of approximating the Hessian?
Questions for authors

See the "Weaknesses" for further details. Overall, while the paper’s novelty is modest, it features a substantial experimental scope and provides highly detailed descriptions. My current assessment leans toward a "weak accept."

1 comment

Thank you again for your review and for the encouraging assessment. Our response above covers extreme compression, GPTQ and AWQ, and the cost at 32B, and the revised paper reports these results in Appendix J. If you have a moment before the discussion closes, we would be glad to hear whether any of your questions remains open.

Reviewer_Gym9OpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This work investigates how to identify the least important layers in transformers for an arbitrary compression operation (e.g., attention-layer removal, weight pruning, linearization, quantization). Some widely used selection criteria rank layers only by how much the operation changes each layer's output, while second-order methods also estimate the loss curvature, but do so separately for each operation, using backpropagated Hessian-vector products. The authors show that compression damage depends both on how much the layer output changes and on how sensitive the loss is to this change. Their method, CurvProbe, estimates an isotropic curvature profile of the loss (one value per layer) once per checkpoint and reuses it for all compression operations. The profile is computed with forward passes only, which makes the method more efficient than backpropagation-based approaches.

Critical strengths
  1. No backward pass is required, which makes the method cheaper than backpropagation-based curvature estimation.
  2. The method can be applied to a wide range of compression techniques (with the caveat in Weakness 5).
  3. I appreciate the extensive experiments on many different models, although most of the checkpoints belong to older generations of LLMs.
  4. Separating how much compression changes the layer output from how sensitive the loss is to that change is an interesting insight. It helps explain why ranking layers only by output changes can be worse than random.
Critical weaknesses
  1. In my opinion, a per-layer score cannot fully capture the damage of compressing several layers at once: it ignores interactions between simultaneously compressed layers, so part of the error is missed and accumulates as more layers are compressed.
  2. After LoRA healing, the method's NLL advantage disappears. The isotropic variant is also weaker on reasoning (GSM8K), and it would be interesting to understand why.
  3. The experiments are extensive, but they are performed on older and relatively small models (operations are measured only up to 13B). Since the method relies on forward passes only, I would like to see at least one experiment on a substantially larger model.
  4. The method also requires fp32 to estimate curvature reliably. In bf16, the estimates fail. This increases memory usage, especially for larger models.
  5. A single isotropic profile does not serve all operations equally well. It is weakest on the operations that remove the sublayer's function (mean rho +0.42 for linearization and +0.56 for deletion, against 0.66-0.73 for quantization and low-rank), where the isotropy assumption is least accurate. The strongest deletion results come from the directional read, which coincides with the deletion direction, so in practice the best results still depend on knowing the operation.
Questions for authors
  1. The probe step size is set from one specific operation, attention linearization (Sec. 4). With the operation-free clean-output step, the layer ordering stays almost the same (rho >= 0.98), but the selected sets cost up to 0.15 nats more (e.g., 0.13 instead of 0.03 nats; Table E.11). Why not report the operation-free variant as the main result, so that the claim that the profile is never shown the operation holds literally?
  2. All evaluated checkpoints are attention-only. Do the factorization and the isotropic profile hold for hybrid models that interleave attention with linear-attention or SSM layers, which are increasingly common among modern models (e.g., Qwen3.5 combines Gated DeltaNet and full-attention layers)?
  3. Is full fp32 really necessary, or would bf16 weights with fp32 loss accumulation, or a larger probe step, make the bf16 read usable?
2 comments

Thank you again for your detailed review. We have responded to each of your points above and updated the paper, most recently on 14 November. If you have a moment before the discussion closes, we would be glad to know whether the response addresses your concerns, and we are happy to clarify anything that remains unclear.

Thank you for the detailed responses and additional experiments. They addressed most of my concerns, and I particularly appreciated the analysis showing where per-layer scores become less reliable under aggressive compression. I'm keeping my rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.