acceptodds
Under review as a conference paper at ICLR 2027

Direction-Aware Low-Rank Weight Factorization for Post-Training Compression

Abstract

Low-rank factorization is a widely used post-training compression technique that typically minimizes weight reconstruction error. However, preserving reconstruction quality alone does not necessarily preserve model performance: it describes how much the weights are changed, but not whether the induced perturbation moves the model toward lower-loss or higher-loss regions. In neural networks, this distinction is crucial. Under a local first-order view, the loss variation is determined by the interaction between the loss gradient and the factorization perturbation, indicating that perturbation direction is a key factor overlooked by reconstruction-only methods. Motivated by this observation, we propose a direction-aware low-rank factorization framework that encourages low-rank perturbations to follow descent-aligned directions while maintaining compression constraints. We further develop two practical variants that emphasize either performance preservation or stronger compression. As a post-training factorization method, our approach does not require multi-epoch post-factorization finetuning. Experiments on vision models, NLP models, and large language models show that our method consistently achieves a better performance-compression trade-off than existing low-rank baselines.

Primary area:
optimization
Keywords:
Direction-Aware, Low-rank factorization, Post-Training Compression

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JfH9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies post-training compression of large neural networks and observes that conventional low-rank factorization minimizes weight reconstruction error, which controls only the magnitude of the perturbation but not whether it pushes the model toward higher- or lower-loss regions, which often causes unnecessary performance degradation at a given compression ratio. To address this, the authors propose Direction-Aware Low-Rank factorization (DA-LR): under a first-order view, the loss variation is dominated by the inner product between the loss gradient and the perturbation, so they shift the original weights along the gradient direction as W̃ = W + (λ/2)G and then apply rank-r truncated SVD, biasing the low-rank perturbation toward descent directions while preserving the compression constraint, with a closed-form solution that requires no multi-epoch post-factorization finetuning. Two practical variants are further introduced—DA-LR-P for performance preservation and DA-LR-C for stronger compression, both of which needing only a small calibration set and a single gradient pass.

Critical strengths
  1. The paper identifies a clean and well-motivated gap in post-training low-rank compression: conventional factorization controls only the magnitude of the weight perturbation and ignores its direction relative to the loss landscape, an observation that is simple yet largely overlooked in prior work.
  2. Building on this, the resulting update W̃ = W + (λ/2)G followed by rank-r truncated SVD is concise and can be dropped into existing pipelines such as SVD-LLM or Denton-style factorization with minimal engineering effort.
  3. The method requires no post-factorization finetuning, which makes it practically attractive for deployment settings where training resources are unavailable.
  4. The calibration overhead is also modest: a few hundred samples and a single backward pass on a single GPU within minutes are sufficient, a clear advantage over methods that demand thousands of calibration steps.
Critical weaknesses
  1. The closed-form solution rests on a first-order approximation and a per-layer independence assumption, neither of which provides a global optimality guarantee or an error bound for deep nonlinear networks. As a result, the scope of the theoretical claim is narrower than what the main text suggests, and the presentation would benefit from a more explicit acknowledgement of this limitation.
  2. The formulation is mathematically close to LoRA with gradient-based initialization, yet the paper positions itself as a compression method without providing a matched-rank comparison against the PEFT line of work. This ambiguity weakens both the novelty claim and the clarity of where the method sits in the existing landscape.
  3. The selection and robustness of λ are under-examined. The performance variant relies on a validation-set grid search, but the paper offers little systematic study of how sensitive the result is to the choice of validation set and evaluation metric in LLM settings, nor of how performance degrades when λ overshoots the regime in which the first-order approximation remains valid.
Questions for authors

Please see my comments on weaknesses.

3 comments
AuthorsOpenReview ↗

Response overview

We thank the reviewer for the constructive assessment. DA-LR combines a local theoretical basis for directional rectification with a compressed representation stored directly as low-rank factors. It estimates the calibration gradient once at the original model and performs a limited number of factor-space updates within a relative Frobenius neighborhood, targeting effective post-training compression under a limited construction budget. Below, we clarify the theoretical scope, the relationship to low-rank fine-tuning, and the selection and stability of the method’s controls.

AuthorsOpenReview ↗

Q1. Theoretical scope & Q2. Relationship to LoRA

Q1. Theoretical scope and higher-order effects

The submitted manuscript provides a local theoretical basis for directional rectification. As specified in §3.3, DA-LR fixes the calibration gradient at the pretrained model and applies the iterative factor-space updates in Eq. (6). Proposition 3.1 establishes descent of the signed first-order alignment for sufficiently small steps while preserving the factor dimensions and rank upper bound. Under the stated smoothness assumptions, Eq. (3) bounds the layer-wise Taylor remainder, and Appendix B.2 extends the analysis to simultaneous multi-layer perturbations, explicitly accounting for cross-layer interactions without requiring layer independence.

To complement this analysis, we evaluate the final rectified LLaMA-7B checkpoint at nominal parameter retention 0.8. All loss changes are measured relative to the original model. We compare independent projection-matrix replacements with joint replacement, using the original-model calibration gradients for the first-order prediction.

Table J1. First-order prediction and measured loss changes on LLaMA-7B at nominal retention 0.8.

MeasurementCalibration ΔL\Delta\mathcal LHeld-out ΔL\Delta\mathcal L
First-order prediction0.08107—
Sum of independent replacements0.102900.13060
Joint replacement0.134300.15910
Interaction difference: joint − sum0.031400.02850

The measured interaction increments are 0.03140 on the calibration set and 0.02850 on the held-out set, directly quantifying the additional loss change associated with joint replacement. These measurements complement the multi-layer expansion in Appendix B.2 and connect the local analysis with network-level loss changes. Within this framework, directional rectification reduces the signed first-order term, while the neighborhood constraint controls the perturbation magnitude. We will make the connection between this local theoretical guarantee, the algorithm, and the empirical evaluation more explicit in the revised manuscript.

Q2. Relationship to LoRA and low-rank fine-tuning

LoRA retains the pretrained weight and learns an additive low-rank update, Weff=W0+BAW_{\mathrm{eff}}=W_0+BA. DA-LR directly replaces W0W_0 with LR⊤LR^\top, using a fixed-gradient surrogate under the local-neighborhood constraint. For an m×nm\times n weight matrix, this compressed representation stores r(m+n)r(m+n) parameters, with compression-effective ranks satisfying r(m+n)<mnr(m+n)<mn.

To evaluate the contribution of directional rectification beyond updating low-rank factors, we compare against Factor-FT, which directly optimizes the calibration loss over the compressed factors. The supplementary LLaMA-7B experiment at nominal retention 0.8 matches initialization, ranks, calibration data, evaluation, and compressed parameter budget. The matched-cost control additionally uses the same construction-time budget.

Table J2. Comparison with direct factor fine-tuning on LLaMA-7B at nominal retention 0.8.

MethodWikiText-2 PPL ↓PTB PPL ↓Construction time (h)
DA-LR (fixed gradient)6.7011.790.62
Factor-FT (matched cost)6.7712.020.62
Factor-FT (extended optimization)6.5911.774.01

At the same 0.62 h construction cost, DA-LR achieves lower perplexity on both datasets. Extended Factor-FT reaches 6.59/11.77 but requires 4.01 h. These results support the efficiency of directional rectification under the tested limited construction budget. We will clarify the relationship to low-rank adaptation and document this comparison in Related Work and the experimental section.

AuthorsOpenReview ↗

Q3. Hyperparameter selection and robustness

Q3. Hyperparameter selection and robustness

The default settings are η=10−4\eta=10^{-4}, γ=0.05\gamma=0.05, τ=0\tau=0, and Tmax⁡=50T_{\max}=50. The main procedure uses these controls without task-specific validation-set grid search; λ\lambda appears only in the optional second-order diagnostic in Appendix B.4. When Compact Mode must meet a prescribed compression budget, §3.5 performs an outer search over γ\gamma based on the resulting parameter count. This search controls the compression budget rather than selecting parameters by validation performance.

Backtracking accepts an update only when the neighborhood constraint holds and the signed alignment does not increase. The existing ResNet-50 studies examine the relevant controls: Table 13 varies η\eta from 10−510^{-5} to 10−310^{-3}, yielding Top-1 accuracy of 76.19%–76.61%; Table 14 reports 76.62%, 76.63%, and 76.60% for Tmax⁡=50,100,200T_{\max}=50,100,200, respectively. Table 6 shows that increasing γ\gamma from 0.05 to 0.15 raises parameter reduction from 30.0% to 48.8%, while Top-1 accuracy changes from 76.62% to 70.94%, demonstrating the compression–accuracy trade-off as the neighborhood expands.

Under the original manuscript’s LLM protocol, Table 8 reports WikiText-2 PPL of 7.06±0.037.06\pm0.03 and 10.40±0.1410.40\pm0.14 at retention 0.8 and 0.6 across five calibration seeds. These original-protocol statistics are reported separately from the supplementary Factor-FT results in Section Q2. They show consistent LLM performance across independently sampled calibration subsets. Together with the parameter sweeps on ResNet-50, they provide complementary evidence for calibration stability and the behavior of the rectification controls. We will clarify the default settings, the budget-based selection of γ\gamma, and the experimental setting of each analysis in the revised manuscript.

Reviewer_eR4mOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the low rank factorization compression of neural networks in a post training setting. They leverage the first order sensitivity of the loss in respects to the weights to guide the compression. Using this framework, they propose two variants tailored to two different objectives, preserving performance or prioritizing large compression. They evaluate their proposed method on an extensive number of experiments on a variety of models and tasks against different baselines.

Critical strengths

The idea is interesting, the observation that methods rarely leverage first order sensitivity output is interesting and valuable.

The empirical comparisons cover a wide and varied range of models and tasks, which is very valuable to evaluate the generality of the improvement.

Critical weaknesses

The paper omits recent strong baselines (eg. gfwsvd, kfac svd [1-2]), that also leverage the sensitivity of the output of the model to the weight. For example, Kronecker-factored Fisher weighting is a stronger version of the FWSVD baseline the paper already includes.

The choice of baselines and metrics for the experiments does not seem consistent across experiments, and stronger (loss-aware) baselines (ERC-SVD, ZS SVD and OBD-LLM) are only compared with once on the extreme compression setting and absent from Table 1 and Table 4, where most claims are made. For example: On Table 1: llama-7b (wiki2 and PTB) compares against ASVD SVD-LLM Dobi-SVD Dip-SVD, but at 0.6, Dip-SVD is dropped without explanation

Table 3 the methods are evaluated on two other models (Llama 2-7B and Llama3-8B), but the baselines are different from table 1 and between the two models: ERC-SVD and ZS SVD for llama 2-7B, OBD-LLM for llama3-8B.

More importantly, Zero sum svd [3] already leverages first order calibration loss sensitivity for low rank compression, but it is not discussed in the related work and is compared with in only one setting, at an extreme compression (third column of table 3).

[1] Chekalina, V. A., Moskovskiy, D., Matveeva, T., Kuznetsov, A., & Frolov, E. Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. In Forty-third International Conference on Machine Learning. [2] Thoma, M., Groezinger, M., Forstenhäusler, M., Aghajanzadeh, E., Vemparala, M. R., Anagnostopoulos, C., ... & Schlichtmann, U. (2026, April). Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. In Forty-third International Conference on Machine Learning. [3] Abbasi, A., Thrash, C., Qin, H., Sharma, S., Seifi, S. & Kolouri, S.. (2026). Zero Sum SVD: Balancing Loss Sensitivity for Low Rank LLM Compression, Proceedings of the 43rd International Conference on Machine Learning.

Questions for authors

Can you clarify if the baselines were run under the same protocol, or are they reported from other papers? Can you provide a unified comparison against SVD-LLM, ZS-SVD, and OBD-LLM (or KFAC SVD that provides public code) under the same models, ratios, and calibration protocol?

Could you explain the conceptual difference between your approach with zero-sum svd?

While Table 6 (w/ Dir.) row agrees with Table 5 Ours(L), (w/o Dir) (which I understand is the SVD of Table 5 per Appendix D.1) gives different results from the values in Table 5. Could you clarify the apparent discrepancy? With the numbers of Table 5, the gain would be +0.25 / +0.01 points rather than about +2.5.

https://github.com/mint-vu/Zero-Sum-SVD https://github.com/lems-svd/lems https://github.com/sayankotor/FisherKronecker

5 comments
AuthorsOpenReview ↗

Response overview

Thank you for the constructive comments. DA-LR combines a calibration gradient computed once at the pretrained model with direct correction of the compressed factors. Its relative Frobenius neighborhood controls the perturbation magnitude, while directional rectification improves the signed first-order criterion within a limited construction budget. Below, we provide unified comparisons, clarify the distinction from ZS-SVD, and explain the configurations underlying Tables 5–6.

AuthorsOpenReview ↗

Q1. Unified comparison & Q2.Conceptual differences between DA-LR and ZS-SVD

Q1. Unified comparison with recent low-rank compression methods

To address comparability across methods, we conducted additional experiments on LLaMA-7B at nominal parameter retention ratios of 0.8 and 0.6. All entries in Table E1 were obtained from our experiments under the unified ZS-SVD protocol, with matched checkpoints, calibration settings, and evaluation procedures. The calibration set consists of 256 WikiText-2 sequences of length 2048.

Table E1. Unified comparison on LLaMA-7B.

MethodNominal retention 0.8: WikiText-2 PPL ↓Nominal retention 0.6: WikiText-2 PPL ↓
SVD-LLM7.9413.11
ZS-SVD6.7411.44
OBD-LLM7.4211.80
KFAC-SVD (uniform allocation)7.2711.12
DA-LR (fixed gradient)6.7010.38

DA-LR achieves lower WikiText-2 perplexity than the listed configurations at both retention ratios. KFAC-SVD uses uniform allocation without the LEMS rank-search stage. The OBD-LLM entries are our LLaMA-7B reproductions, reported separately from the LLaMA3-8B result cited in the original Table 3. Comparisons with ZS-SVD 1X are provided in Section Q2.

We also filled the missing Dip-SVD comparison at 0.6 retention under the same unified protocol. Dip-SVD achieves 12.76/46.95/0.40 on WikiText-2/PTB/average accuracy, compared with 10.38/43.25/0.42 for DA-LR. The original Table 1 results remain reported under their original experimental setting.

We will incorporate these comparisons into the main experimental discussion, identify each baseline’s implementation and configuration, and expand Related Work to discuss GFWSVD and KFAC-SVD/LEMS. Performance claims will correspond to the models, metrics, and configurations evaluated.

Q2. Conceptual differences between DA-LR and ZS-SVD

Both methods use signed first-order loss information. The distinction lies in how this information is used to construct and correct the low-rank representation. ZS-SVD uses singular-component sensitivities for global rank allocation, and its 1X variant additionally computes a gradient at the truncated model, applies a correction, and re-truncates the weights.

DA-LR computes the gradient once at the original model and directly updates the selected low-rank factors to reduce their residual’s signed alignment with that gradient. Factor dimensions remain fixed during rectification, while a relative Frobenius neighborhood and explicit acceptance and stopping criteria control the updates. Although Compact Mode also selects ranks, directional rectification operates on the factors after rank selection.

To assess these design choices, we performed ablations with matched initialization, selected ranks, and evaluation settings. At the same 0.62-hour construction cost, DA-LR (fixed gradient) achieves 6.70 WikiText-2 PPL, compared with 6.77 for Factor-FT (matched cost). Correction followed by re-truncation yields 6.76, and removing the neighborhood constraint yields 6.85. These controls support the contributions of factor-space correction and neighborhood control.

We further evaluated gradient recomputation while retaining DA-LR’s factor-space update mechanism.

Table E2. Comparison with ZS-SVD 1X on LLaMA-7B at nominal retention 0.8.

MethodWikiText-2 PPL ↓PTB PPL ↓Average accuracy ↑
ZS-SVD 1X6.6111.260.51
DA-LR (fixed gradient)6.7011.790.51
DA-LR (recomputed gradients)6.6011.770.52

The recomputed-gradient variant achieves WikiText-2 perplexity comparable to ZS-SVD 1X and higher average accuracy, while ZS-SVD 1X has lower PTB perplexity. Within DA-LR, gradient recomputation increases construction time from 0.62 to 3.03 hours. At 0.6 retention, ZS-SVD 1X achieves 9.96/37.13/0.42, compared with 10.38/43.25/0.42 for DA-LR (fixed gradient).

Together with the matched-cost ablations, these comparisons support constrained factor-space rectification as an effective approach under a limited construction budget and quantify the additional cost of gradient recomputation within DA-LR. We will clarify this positioning and the gradient-evaluation schedules in the revised Related Work and Method sections.

AuthorsOpenReview ↗

Q3. Discrepancy between Tables 5 and 6

Q3. Discrepancy between Tables 5 and 6

The two tables come from independent experiments with different baseline configurations. The SVD baselines in Table 5 use conventional SVD followed by one epoch of fine-tuning. Table 6 uses SVD-LLM-based initialization; its w/o Dir. variant applies neither directional rectification nor post-factorization fine-tuning. The w/o Dir. and w/ Dir. variants share the initialization, selected ranks, compression budget, calibration data, and evaluation protocol, with directional rectification as their only difference.

Accordingly, the 0.246/0.006 percentage-point gains in Table 5 measure improvement over the fine-tuned SVD baseline, whereas the 2.64/2.50-point gains in Table 6 measure directional rectification’s effect under paired ablation settings. These gains correspond to different comparisons and should be interpreted within their respective configurations.

We will label the Table 5 baselines as SVD + FT (1 epoch) and document both tables’ initialization and fine-tuning settings in the experimental section and Appendix D.1. The reported results are retained.

Reviewer_XSVuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Direction-Aware Low-Rank Weight Factorization for Post-Training Compression estimates a task gradient G once at the original model, initializes low-rank factors L and R using SVD, and updates the factors to reduce the signed first-order term. The method imposes rank and relative Frobenius-neighborhood constraints and includes variants for preserving performance and obtaining a more compact model.

Critical strengths
  1. The computational procedure is simple and operates directly on low-rank factors without multi-epoch model training. The neighborhood constraint, acceptance criterion, and backtracking provide useful controls on the update. The appendix addresses infeasible ranks by increasing rank or retaining the original layer, and includes calibration-seed and hyperparameter analyses. These details strengthen the implementation and stability evidence.
  2. The evaluation spans decoder-only language models, vision models, and an encoder model. In several listed LLaMA-7B settings, the method improves over SVD-LLM, ASVD, and Dobi-SVD. Construction-time breakdowns for ResNet/BERT and selected inference measurements provide useful initial evidence of practicality.
Critical weaknesses
  1. The closest prior method and stronger public operating points are not adequately incorporated into the central comparison. ZS-SVD, which is cited, already includes signed first-order loss estimates, global budget allocation, and optional gradient correction followed by re-truncation. (Sections 4.2-4.3; arXiv:2602.02848v1). At a retained ratio of 0.8 on LLaMA-7B, its Table 1 reports Wiki2/PTB perplexities and average accuracy of 6.74/11.87/0.50, versus 7.06/14.69/0.49 here. At a retained ratio of 0.6, ZS-SVD 1X reports 9.96/37.13/0.42, versus 10.40/45.11/0.42 here, although the additional correction cost must be counted. Cross-paper protocols still require alignment, so these numbers do not establish full dominance. Nevertheless, they make a direct comparison essential. The paper must explain the independent value of fixing the original gradient, updating within the factor space, and imposing its neighborhood and stopping rules. A single extreme-compression ZS-SVD point in Table 3 is insufficient.
  2. Budgets and experimental consistency are not yet auditable. Table 3 compares OBD-LLM's 167.3 with 144.31 here at ratio=0.4. In the original OBD paper (arXiv:2604.00821), 167.3 appears in the 40%-removed column of Table 1, with compression defined as 1 - r(m+n)/(mn), whereas this submission's principal tables use retained-parameter ratios. This suggests a possible convention mismatch unless OBD was rerun; the source of the number needs clarification. There is also an important ablation discrepancy: Table 5 gives SVD accuracies of 80.761/78.154 for ViT-B/ResNeXt101, whereas Table 6's no-direction variant gives 78.37/75.66, despite similar directional endpoints of approximately 81.01/78.16. Appendix D.1's within-ablation controls do not explain this cross-table gap. Together with missing actual ranks, neighborhood feasibility statistics at gamma=0.05, post-backtracking budgets, and BERT compression budgets, these issues prevent attribution of the gains specifically to direction rather than configuration or operating point.
Questions for authors

Q1 Experiment affecting novelty and competitiveness. Please compare the proposed method with ZS-SVD and ZS-SVD 1X using the same checkpoint, calibration examples and sequence lengths, retained bytes, numerical precision, and evaluation scripts. Please separate fixed versus recomputed gradients, factor-space updates versus projection and re-truncation, and the proposed constraint versus ordinary factor fine-tuning at matched cost. The relevant ZS-SVD components already appear in v1, rather than only in its recent revision. Q2 Critical clarification. Was OBD-LLM's 167.3 copied from the original table or obtained by rerunning it? What exactly does ratio=0.4 mean for each method? Please provide actual parameter counts, storage bytes, and layer ranks. If the number comes from the original table, the removed-versus-retained convention must be corrected before interpreting this comparison. Q3 Critical clarification and experiment. Please explain the SVD/no-direction discrepancy between Tables 5 and 6 and provide each row's configuration. Also report SVD tail energy, the achieved relative error ||LR^T-W||_F/||W||_F, actual gamma values, infeasible layers, and rank increases. These quantities are necessary to verify that the claimed budget and neighborhood constraints hold simultaneously. Q4 Experiment affecting the theoretical interpretation. The appendix already examines higher-order effects for ResNet/BERT. Can you compare first-order predictions with actual calibration and held-out loss changes on LLMs, including the interaction error when all layers are replaced together? Local surrogate descent alone does not establish non-increasing actual loss, especially when the first-order term is close to zero or negative. Q5 Fairness and resource clarification. Please report total LLM calibration, SVD, rectification, and budget-search time, peak memory, and on-disk model size. Specify batch size, sequence length, dtype, hardware, prefill/decode workload, latency units, and timing boundaries. Please also define the Data unit in Table 7 and reconcile 256 calibration examples with 7.6M Data. The existing cost measurements are useful but insufficient to establish low overhead across all evaluated model families.

5 comments
AuthorsOpenReview ↗

Response overview

Thank you for the detailed and constructive comments. DA-LR combines a calibration gradient computed once at the pretrained model, direct updates of compressed low-rank factors, and a layer-relative neighborhood constraint. This design improves the signed first-order loss criterion while preserving the selected factor dimensions during rectification, without multi-epoch post-factorization fine-tuning. The additional comparisons, mechanism ablations, and budget and resource measurements below address the requested evidence and supplement the original submission.

AuthorsOpenReview ↗

Q1. Comparison with ZS-SVD and methodological differences

Q1. Comparison with ZS-SVD and methodological differences

A1: To assess competitiveness under a common protocol, we reproduced ZS-SVD using its official implementation and evaluated DA-LR on LLaMA-7B at nominal retention ratios of 0.8 and 0.6. These experiments use matched checkpoints, calibration settings, and evaluation procedures, including 256 WikiText-2 calibration sequences of length 2048.

Table R1. Comparison with standard ZS-SVD on LLaMA-7B.

Nominal retentionMethodWikiText-2 PPL ↓PTB PPL ↓Avg. accuracy ↑
0.8ZS-SVD6.7411.870.50
0.8DA-LR (fixed gradient)6.7011.790.51
0.6ZS-SVD11.4443.190.41
0.6DA-LR (fixed gradient)10.3843.250.42

DA-LR improves all three reported metrics at 0.8 retention. At 0.6, it achieves lower WikiText-2 perplexity and higher average accuracy, with comparable PTB performance.

Both methods use signed first-order loss information. The distinction lies in the optimization variables and correction mechanism: ZS-SVD uses singular-component sensitivities for global rank allocation, while its optional correction performs a gradient-based update followed by re-truncation. These components already appear in ZS-SVD v1. DA-LR instead fixes the original-model gradient and directly updates the low-rank factors, preserving their dimensions while enforcing the relative neighborhood constraint and alignment-based stopping criteria.

To examine the role of gradient recomputation, we separately compared ZS-SVD 1X with DA-LR (fixed gradient) and DA-LR (recomputed gradients).

Table R2. Gradient-recomputation comparison at nominal retention 0.8.

MethodWikiText-2 PPL ↓PTB PPL ↓Avg. accuracy ↑
ZS-SVD 1X6.6111.260.51
DA-LR (fixed gradient)6.7011.790.51
DA-LR (recomputed gradients)6.6011.770.52

The recomputed-gradient variant achieves comparable WikiText-2 perplexity to ZS-SVD 1X and higher average accuracy, while ZS-SVD 1X achieves lower PTB perplexity. Within DA-LR, recomputation increases construction time from 0.62 to 3.03 hours.

To isolate the contribution of factor-space updates and neighborhood control, we next compared correction mechanisms under matched initialization, selected ranks, compression budgets, and evaluation settings.

Table R3. Correction-mechanism ablations at nominal retention 0.8.

MethodWikiText-2 PPL ↓PTB PPL ↓Avg. accuracy ↑Construction time (h)
DA-LR (fixed gradient)6.7011.790.510.62
Correction + re-truncation6.7612.040.490.59
Factor-FT (matched cost)6.7712.020.490.62
Factor-FT (extended optimization)6.5911.770.524.01
DA-LR (without neighborhood constraint)6.8512.110.480.63

At the matched construction cost of 0.62 hours, DA-LR achieves lower WikiText-2 perplexity than factor fine-tuning, 6.70 versus 6.77. Re-truncation and removing the neighborhood constraint yield 6.76 and 6.85, respectively. Longer factor fine-tuning reaches 6.59 but requires 4.01 hours.

These results support constrained factor-space rectification under a limited construction budget. We will document the correction procedures and gradient-evaluation schedules and clarify these methodological differences in the revised manuscript.

AuthorsOpenReview ↗

Q2. Compression ratio and parameter-budget clarification

Q2. Compression ratio and parameter-budget clarification

For the LLaMA3-8B comparison between OBD-LLM and DA-LR in Table 3, ratio = 0.4 denotes the 40%-removal setting for the compressible weight matrices. This clarification applies specifically to that comparison.

OBD-LLM’s WikiText-2 perplexity of 167.3 is cited from Table 1 of its original paper, arXiv:2604.00821v1. DA-LR’s 144.31 is measured in our experiment, following the same calibration and evaluation settings, including 128 C4 sequences of length 2048. To make the compression setting auditable, we report our parameter counts, budget record, and storage measurements below.

Table R4. LLaMA3-8B parameter, budget, and storage statistics.

MeasurementValue
Original whole-model parameters8,030,261,248
Original BF16 weight payload16,060,522,496 bytes
Original checkpoint size16.1 GB
Compression-budget ratio in our experiment record0.599847
DA-LR final whole-model parameters5.2658B
Realized whole-model retention65.6%
Parameters removed from the whole model2.7645B
DA-LR parameter payload at 16-bit precision10.53 GB
Measured DA-LR checkpoint size15.9 GB

The final parameter count includes all low-rank factors and uncompressed components, including embeddings and normalization layers. The 65.6% whole-model retention is calculated from the final and original parameter counts; 0.599847 is listed separately as the compression-budget ratio recorded in our experiment.

Likewise, 10.53 GB denotes the parameter payload at two bytes per parameter, while 15.9 GB is the measured saved-checkpoint size. We will specify the ratio definitions and accounting scope, provide the corresponding LLaMA3-8B layer ranks, and document the saved-checkpoint contents in the revised Table 3 and accompanying implementation details.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal at a Glance

We appreciate the reviewers' positive recognition of several strengths of our work(DA-LR):

(i) the clear motivation for considering loss-relevant perturbation directions beyond reconstruction error;

(ii) the simple and practical factor-space rectification mechanism with explicit neighborhood and backtracking controls;

(iii) the low calibration overhead without multi-epoch post-factorization fine-tuning;

(iv) the extensive evaluation across LLMs, vision models, and encoder-based NLP models.

These strengths highlight the methodological motivation, practical efficiency, and broad applicability of DA-LR, which we further substantiate through the additional experiments and analyses conducted during the rebuttal.

During the rebuttal, we substantially strengthened the experimental and analytical evidence for DA-LR through new unified comparisons, targeted mechanism ablations, and detailed budget and efficiency checks. We evaluated recent low-rank baselines on LLaMA-7B under matched settings at nominal retention ratios of 0.8 and 0.6, added comparisons with ZS-SVD 1X and the previously missing Dip-SVD operating point, and isolated the effects of gradient recomputation, factor-space correction, re-truncation, neighborhood control, and matched-cost factor fine-tuning.

We also verified realized parameter budgets, factor ranks, and reconstruction statistics; clarified the LLaMA3-8B/OBD-LLM compression convention and the baseline configurations in the original Tables 5–6; compared first-order predictions with actual calibration and held-out loss changes; and reported complete construction-time and peak-memory measurements. These additions strengthen the manuscript's methodological positioning, the attribution of its gains, and the transparency of its compression and cost evaluation, while preserving the original reported results.

Guide to the Supplementary Evidence

TablesPurpose
R1, E1Unified comparisons with standard ZS-SVD and other recent baselines at two retention ratios.
R2, E2ZS-SVD 1X comparison and gradient-recomputation trade-offs.
R3, J2Mechanism ablations and matched-cost factor fine-tuning.
R4LLaMA3-8B compression convention, realized parameters, and storage statistics.
R5Baseline configurations and interpretation of the original Tables 5–6.
R6–R7Final ranks, neighborhood feasibility, and achieved LLaMA-7B parameter budget.
R8, J1First-order estimates, measured loss changes, and joint-replacement interactions.
R9–R10Construction costs, peak memory, and reported full-sequence forward efficiency.

R, E, and J refer to tables in the responses to XSVu, eR4m, and JfH9, respectively.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.