- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies post-training compression of large neural networks and observes that conventional low-rank factorization minimizes weight reconstruction error, which controls only the magnitude of the perturbation but not whether it pushes the model toward higher- or lower-loss regions, which often causes unnecessary performance degradation at a given compression ratio. To address this, the authors propose Direction-Aware Low-Rank factorization (DA-LR): under a first-order view, the loss variation is dominated by the inner product between the loss gradient and the perturbation, so they shift the original weights along the gradient direction as W̃ = W + (λ/2)G and then apply rank-r truncated SVD, biasing the low-rank perturbation toward descent directions while preserving the compression constraint, with a closed-form solution that requires no multi-epoch post-factorization finetuning. Two practical variants are further introduced—DA-LR-P for performance preservation and DA-LR-C for stronger compression, both of which needing only a small calibration set and a single gradient pass.
Critical strengths
- The paper identifies a clean and well-motivated gap in post-training low-rank compression: conventional factorization controls only the magnitude of the weight perturbation and ignores its direction relative to the loss landscape, an observation that is simple yet largely overlooked in prior work.
- Building on this, the resulting update W̃ = W + (λ/2)G followed by rank-r truncated SVD is concise and can be dropped into existing pipelines such as SVD-LLM or Denton-style factorization with minimal engineering effort.
- The method requires no post-factorization finetuning, which makes it practically attractive for deployment settings where training resources are unavailable.
- The calibration overhead is also modest: a few hundred samples and a single backward pass on a single GPU within minutes are sufficient, a clear advantage over methods that demand thousands of calibration steps.
Critical weaknesses
- The closed-form solution rests on a first-order approximation and a per-layer independence assumption, neither of which provides a global optimality guarantee or an error bound for deep nonlinear networks. As a result, the scope of the theoretical claim is narrower than what the main text suggests, and the presentation would benefit from a more explicit acknowledgement of this limitation.
- The formulation is mathematically close to LoRA with gradient-based initialization, yet the paper positions itself as a compression method without providing a matched-rank comparison against the PEFT line of work. This ambiguity weakens both the novelty claim and the clarity of where the method sits in the existing landscape.
- The selection and robustness of λ are under-examined. The performance variant relies on a validation-set grid search, but the paper offers little systematic study of how sensitive the result is to the choice of validation set and evaluation metric in LLM settings, nor of how performance degrades when λ overshoots the regime in which the first-order approximation remains valid.
Questions for authors
Please see my comments on weaknesses.
3 comments
Q1. Theoretical scope & Q2. Relationship to LoRA
Q1. Theoretical scope and higher-order effects
The submitted manuscript provides a local theoretical basis for directional rectification. As specified in §3.3, DA-LR fixes the calibration gradient at the pretrained model and applies the iterative factor-space updates in Eq. (6). Proposition 3.1 establishes descent of the signed first-order alignment for sufficiently small steps while preserving the factor dimensions and rank upper bound. Under the stated smoothness assumptions, Eq. (3) bounds the layer-wise Taylor remainder, and Appendix B.2 extends the analysis to simultaneous multi-layer perturbations, explicitly accounting for cross-layer interactions without requiring layer independence.
To complement this analysis, we evaluate the final rectified LLaMA-7B checkpoint at nominal parameter retention 0.8. All loss changes are measured relative to the original model. We compare independent projection-matrix replacements with joint replacement, using the original-model calibration gradients for the first-order prediction.
Table J1. First-order prediction and measured loss changes on LLaMA-7B at nominal retention 0.8.
| Measurement | Calibration | Held-out |
|---|---|---|
| First-order prediction | 0.08107 | — |
| Sum of independent replacements | 0.10290 | 0.13060 |
| Joint replacement | 0.13430 | 0.15910 |
| Interaction difference: joint − sum | 0.03140 | 0.02850 |
The measured interaction increments are 0.03140 on the calibration set and 0.02850 on the held-out set, directly quantifying the additional loss change associated with joint replacement. These measurements complement the multi-layer expansion in Appendix B.2 and connect the local analysis with network-level loss changes. Within this framework, directional rectification reduces the signed first-order term, while the neighborhood constraint controls the perturbation magnitude. We will make the connection between this local theoretical guarantee, the algorithm, and the empirical evaluation more explicit in the revised manuscript.
Q2. Relationship to LoRA and low-rank fine-tuning
LoRA retains the pretrained weight and learns an additive low-rank update, . DA-LR directly replaces with , using a fixed-gradient surrogate under the local-neighborhood constraint. For an weight matrix, this compressed representation stores parameters, with compression-effective ranks satisfying .
To evaluate the contribution of directional rectification beyond updating low-rank factors, we compare against Factor-FT, which directly optimizes the calibration loss over the compressed factors. The supplementary LLaMA-7B experiment at nominal retention 0.8 matches initialization, ranks, calibration data, evaluation, and compressed parameter budget. The matched-cost control additionally uses the same construction-time budget.
Table J2. Comparison with direct factor fine-tuning on LLaMA-7B at nominal retention 0.8.
| Method | WikiText-2 PPL ↓ | PTB PPL ↓ | Construction time (h) |
|---|---|---|---|
| DA-LR (fixed gradient) | 6.70 | 11.79 | 0.62 |
| Factor-FT (matched cost) | 6.77 | 12.02 | 0.62 |
| Factor-FT (extended optimization) | 6.59 | 11.77 | 4.01 |
At the same 0.62 h construction cost, DA-LR achieves lower perplexity on both datasets. Extended Factor-FT reaches 6.59/11.77 but requires 4.01 h. These results support the efficiency of directional rectification under the tested limited construction budget. We will clarify the relationship to low-rank adaptation and document this comparison in Related Work and the experimental section.
Q3. Hyperparameter selection and robustness
Q3. Hyperparameter selection and robustness
The default settings are , , , and . The main procedure uses these controls without task-specific validation-set grid search; appears only in the optional second-order diagnostic in Appendix B.4. When Compact Mode must meet a prescribed compression budget, §3.5 performs an outer search over based on the resulting parameter count. This search controls the compression budget rather than selecting parameters by validation performance.
Backtracking accepts an update only when the neighborhood constraint holds and the signed alignment does not increase. The existing ResNet-50 studies examine the relevant controls: Table 13 varies from to , yielding Top-1 accuracy of 76.19%–76.61%; Table 14 reports 76.62%, 76.63%, and 76.60% for , respectively. Table 6 shows that increasing from 0.05 to 0.15 raises parameter reduction from 30.0% to 48.8%, while Top-1 accuracy changes from 76.62% to 70.94%, demonstrating the compression–accuracy trade-off as the neighborhood expands.
Under the original manuscript’s LLM protocol, Table 8 reports WikiText-2 PPL of and at retention 0.8 and 0.6 across five calibration seeds. These original-protocol statistics are reported separately from the supplementary Factor-FT results in Section Q2. They show consistent LLM performance across independently sampled calibration subsets. Together with the parameter sweeps on ResNet-50, they provide complementary evidence for calibration stability and the behavior of the rectification controls. We will clarify the default settings, the budget-based selection of , and the experimental setting of each analysis in the revised manuscript.
Response overview
We thank the reviewer for the constructive assessment. DA-LR combines a local theoretical basis for directional rectification with a compressed representation stored directly as low-rank factors. It estimates the calibration gradient once at the original model and performs a limited number of factor-space updates within a relative Frobenius neighborhood, targeting effective post-training compression under a limited construction budget. Below, we clarify the theoretical scope, the relationship to low-rank fine-tuning, and the selection and stability of the method’s controls.