- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper proposes CurvProbe, a layer-selection method for post-training compression. It estimates an output-space curvature profile using forward finite differences and combines this profile with operation-specific output displacements to predict layer damage. The intended benefit is to reuse the same curvature profile across different compression operations.
Critical strengths
-
The empirical question is useful: local output displacement can be a poor indicator of the final loss increase, and testing whether a reusable output-space curvature profile improves this indicator is worthwhile. The Llama-3-8B example provides a clear motivation.
-
The ranking experiments are substantial and include meaningful controls: the reported improvements on 53/55 core pairs and 121/128 extended pairs are supported by comparisons with shuffled profiles, depth priors, scale normalization, and each factor alone. The statistical analysis groups operations by checkpoint rather than treating all pairs as independent.
Critical weaknesses
-
Notation and experimental terminology are not introduced where they are needed. Figure 1 uses quantities such as and before their introduction in Section 3, and its caption does not adequately define them. Terms such as “cell” and the different experimental cohorts are also used before their meaning is established. Several headline correlations and percentages appear before the relevant variables, experimental units, and calculation procedures are sufficiently explained. As a result, I need to read the later sections and then I can read some figures at previous sections. The main text and figure captions need to be substantially more self-contained.
-
The current manuscript is not ready to be a final submission for ICLR. At the appendix, there are internal verification records or responses to earlier feedback. Examples include “A reviewer noted...” in Appendix D.7 and “a reviewer was right...” in Appendix E.11, followed by accounts of what was changed. Similar passages recur elsewhere. Where does these reviewers come from?
-
The inconsistencies affect substantive conclusions and presentation. Appendix D.10 provides a concrete example. Its corrected accounting reports a valid fp32 profile cost of 705.1s and a break-even point at the eighth operation when compared with bf16 direct measurement. The section also explains that the earlier 203.1s estimate relied on a bf16 curvature computation that does not produce a valid profile. Nevertheless, the same section retains the 203.1s cost and claims that the method is ahead from the third operation for deletion. Table D.9 also retains the earlier accounting. These are materially different conclusions about when the proposed method is worth using. I'm confusing about it.
-
The decomposition into perturbation magnitude and curvature is established in prior work. In particular, HAWQ-V2 already combines squared perturbation magnitude with average Hessian trace in Eq. (5), extends its analysis to activations in Section 2.2, and compares trace-weighted sensitivity with perturbation-only sensitivity. From my perspective, I think the most plausible contribution here is the forward-only estimator at a common sublayer output site, together with the empirical study of reuse across compression operations. This should be the central claim. Moreover, the limitations of the particular HAWQ-V2-style weight-space score evaluated in this paper should not be presented as limitations of the full activation-space second-order framework. One can also compute the Hessian component before the target perturbation is chosen in either framework; the complete score requires the perturbation in both.
[1] HAWQ-V2: Hessian Aware trace-Weighted Quantization of Neural Networks https://arxiv.org/abs/1911.03852
Questions for authors
Which configuration should be regarded as the primary method: reference-norm or clean-norm calibration, and isotropic or directional probing? Can you provide a single algorithm and a consistent set of headline results for that configuration?
Thank you again for reading the revision so closely. The paper was updated once more on 14 November, with the three points from your follow-up corrected and the rest of the appendix and the supplementary appendix checked for the same kind of mismatch. If time allows, we would be grateful to know whether any concern remains, and we will address it promptly.