- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces an interesting perspective on low-rank compression of LLMs by studying functional changes during post-compression recovery. It introduces the concept of ‘recovery pressure’ which reflects a reproducible compression-induced functional structure. Recovery pressure is characterized through dense-referenced sensitivities to channel-wise perturbations, while fixed-coordinate recovery preserves compression-selected subspaces. The paper provides substantial empirical analysis across six compression methods and two backbones, for which it investigates characteristics such as endpoint-sensitivity, evolution during recovery and their potential contribution to the stopping decision.
Critical strengths
- The paper proposes an interesting concept termed 'recovery pressure', which is motivated by the observation that compressed endpoint ranking is not necessarily an indication of post-recovery capability of compressed models. This provides a useful perspective on the evaluation of compressed models beyond their immediate post-compression performance.
- A shared recovery protocol enables systematic comparisons across compression methods. The extensive empirical analysis provides useful insights into the structure and evolution of post-compression recovery through the proposed characterization of ‘recovery pressure’.
Critical weaknesses
- The stopping comparison: Section 6 describes 'Loss+P' as adding pressure-stability requirements to a loss criterion. If the loss thresholds and patience are identical, it is unclear how it could lead to earlier stopping than the 'Loss' itself. Yet Table 4 reports it. Let's assume the pressure would be satisfied earlier, in order for 'Loss+P' to stop earlier, the Loss would not be satisfied at a different step than before, correct? Thus, could you please specify any misconception here, or provide the differences and establish whether pressure improves the stopping trade-off beyond the threshold calibration. Additionally, although Section 6 acknowledges the trade-off, the higher perplexity and similar Avg7 limits the interpretation that earlier stopping preserves model quality.
- The characterization’s added value remains incompletely established: Sections 4–5 document reproducible, evolving sensitivity patterns, but do not clearly establish their explanatory value beyond simple diagnostics. Since Section 4 does not establish incremental predictive value for subsequent recovery gain beyond endpoint loss, makes clarification of the unresolved stopping comparison in Section 6 more important. Moreover, the overall contribution remains primarily empirical, with limited theoretical support linking recovery pressure to recoverability.
**Note to authors: There appears to be a slight error in table indexing. Table 3 appears before Table 2 does.
Questions for authors
- Could you please provide clarification on the stopping comparison in Section 6? Moreover, Section 3.3 introduces dense-reference subtraction and separates pressure magnitude from module-wise structure. In the stopping experiment (Section 6, Table 4), what additional benefit does each component provide?
- Is higher initial absolute module pressure associated with larger validation-loss reductions when that module is adapted under matched budgets? Such evidence would strengthen the connection between pressure and recoverability (cf. Section 5).