acceptodds
Under review as a conference paper at ICLR 2027

UDA-SVD: Upstream- and Downstream-Aware Low-Rank Approximation and Adaptive Rank Allocation for LLMCompression

Abstract

The rapid growth of large language models (LLMs) has intensified the demand for efficient compression, and low-rank factorization provides a simple and broadly compatible solution. However, in sequential low-rank compression, the effectiveness of compressing each module is shaped by two factors: 182 upstream compression perturbs the activations received by the current module; and 183 approximation errors introduced at the current module propagate downstream and can have unequal effects on the final prediction. Existing methods typically model these effects in isolation. Conventional layer-wise approaches account for neither explicitly, whereas cumulative error-compensation methods primarily address upstream deviations and loss-aware methods focus on downstream sensitivity. Moreover, uniform rank allocation ignores differences in rank sensitivity across modules, leading to inefficient capacity allocation. We propose UDA-SVD, an upstream- and downstream-aware framework for low-rank LLM compression. UDA-SVD captures both activation deviations accumulated through upstream compression and the sensitivity of the language-modeling objective to compression errors at the current module. By integrating these two signals into a unified objective, UDA-SVD provides a common optimization basis for module-wise low-rank approximation and adaptive rank allocation under a fixed parameter budget. Across multiple compression ratios and model architectures, UDA-SVD consistently achieves the highest downstream accuracy among the compared SVD-based methods while substantially reducing language-modeling perplexity, including a nearly 40% reduction on WikiText-2. These results demonstrate the effectiveness of combining upstream drift, downstream sensitivity, and adaptive rank allocation, highlighting UDA-SVD as a promising framework for effective low-rank LLM compression.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM Quantization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_1vjVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UDA-SVD, a post-training low-rank compression method for LLM. It jointly models upstream activation drift caused by earlier compression and downstream sensitivity to errors at the current module. Experimental results demonstrate substantial memory savings and inference speedup.

Critical strengths
  1. The paper clearly identifies two important factors in sequential low-rank compression: upstream activation drift and downstream sensitivity. It explains why modeling both factors jointly, together with adaptive rank allocation, may improve compression quality.

  2. The paper gives a clear mathematical formulation that combines current-path reconstruction and compensation for upstream drift, while weighting output directions by an estimated downstream sensitivity. The two-sided transformation provides a closed-form solution for the resulting fixed-statistics local problem.

  3. The experiments cover multiple model families, compression budgets, perplexity datasets, zero-shot tasks, ablations, and deployment-oriented memory/speed measurements. The reported gains over the included baselines are generally consistent.

Critical weaknesses
  1. UDA-SVD requires calibration forward/backward steps, statistic construction, SVD, and sequential path updates, but the paper reports neither compression time nor peak memory.

  2. Rank allocation is limited to predefined groups within each Transformer block, with no budget transfer across layers or attention/MLP groups. The paper does not sufficiently justify this restriction or evaluate its impact relative to cross-layer allocation.

  3. Some closely related works are missing. For example, [1] SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression, and [2] Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression.

  4. The design challenge and the critical design innovation in each solution component are not clear. They are reasonable but plain, lacking solution depth.

  5. The models in evaluation are relatively not large. It is questionable how the solution would perform in larger models.

Questions for authors
  1. What are the total offline compression time and peak memory usage of UDA-SVD, including calibration, gradient computation, and decomposition? How do they compare with the evaluated baselines under the same hardware and calibration settings?
  2. Why is rank allocation restricted to predefined groups within each Transformer block? Would allowing budget transfer across layers improve compression quality?
  3. How does UDA-SVD compare with [1] and [2], both of which incorporate adaptive rank or compression-ratio allocation? A comparison under matched parameter budgets and calibration settings would help establish the benefit of the proposed approach.
Reviewer_hGPvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a new method for post-training SVD compression of large language models. The method balances both compensating for errors induced from compressing earlier layers (upstream compensation) as well as minimizing the impact to the overall loss (downstream compensation). In addition, the authors use the singular value spectrum of the upstream/downstream whitened matrix to allocate ranks heterogeneously across layers. The authors validate their method empirically across both Llama and Qwen architectures and show consistent performance improvement when compared with other recent SVD-based compression methods.

Critical strengths

The proposed method has strong results outperforming both methods with downstream compensation and upstream compensation.

Critical weaknesses
  • The contribution seems to be fairly narrow. Upstream compensation I believe is studied in SAES-SVD. Downstream compensation has likewise been studied in works such at FW-SVD and GFW-SVD. As the proposed method is primarily a combination of existing works, the overall novelty of the method seems poor.
  • The ablation results seem to suggest that the proposed method is not necessarily superior to SAES-SVD for Qwen architectures. For Qwen3-8B, average task accuracy at a 40% parameter ratio is reported as 39.21 in Table 2 and 38.82 in Table 3. This seems to be the case for all parameter ratios. Most importantly, this contradicts the claim that the proposed method outperforms SAES-SVD on Qwen-3 8B. From the appendix it appears the two tables were generated with different random seeds which explains the mismatch. However, the concern that the method does not improve over SAES-SVD on Qwen remains.
Questions for authors
  • The method for selecting the coefficient βℓ\beta_\ell seems to not be described in the text. SAES-SVD provides a method for automatically selecting this at each layer. However, with the incorporation of downstream sensitivity it’s unclear if this strategy is still appropriate. Could the authors elaborate on how this is selected and provide justification for it?
Reviewer_aKBbOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This work proposes UDA-SVD, a post-training, fine-tuning-free low-rank compression method for LLM linear layers. UDA-SVD handles two effects of sequential compression at once: upstream drift from already-compressed modules, addressed by reconstructing toward the full-precision reference via a compensation coefficient, and downstream sensitivity, addressed by weighting output errors with a damped diagonal empirical-Fisher metric from calibration gradients. Its contributions are a unified quadratic objective that reduces through a two-sided transformation to a closed-form truncated SVD and a greedy rank allocation across modules driven by the same transformed spectrum. Extensive experiments show the effectiveness of the proposed method.

Critical strengths

The paper shows two previously separate ideas, reference-path drift compensation (SAES-SVD) and Fisher-weighted error metrics (GFW-SVD), compose into a single weighted least-squares objective with an exact closed-form truncated-SVD solution, and that the same transformed spectrum then gives a principled signal for rank allocation; this is a clean, correct piece of reasoning that unifies rather than merely stacks prior components, and the special-case reductions (β = 0 recovers SVD-LLM, L = I recovers SAES-SVD) make the contribution easy to verify and build on.

Critical weaknesses
  1. The ablation experiments for Qwen3-8B weaken the paper’s main argument: when ρ = 0.6, the adaptive allocation method without a unified metric achieved a score of 49.08, while the method using a unified metric scored 48.12; under other budgets, the full method won by only 0.3 points, a margin that falls within the standard deviation. The paper refers to this as a “subtle difference” but offers no explanation. Therefore, the argument that “both effects must be modeled jointly” is based on only one model.
  2. These results stem from existing components (SAES-SVD compensation, GFW-SVD Fisher weighting, and AdaSVD-style allocation), whose closed-form expressions can be directly derived from the previous objective function. Beyond LLaMA-3-8B, the improvement over SAES-SVD ranges from 0.05 to 0.2 points when ρ ≤ 0.6, while on LLaMA-2-13B, it performs on par with SVD-LLM when ρ = 0.8.
  3. Through effective, the proposed method is incremental and novelty of the paper is limited. Moreover, the evidence demonstrating the significant effectiveness of this combined approach is limited to a single model.
Questions for authors
  1. Why does the unified metric help on LLaMA-3-8B but not on Qwen3-8B at ρ = 0.6 (Table 3)? Is this traceable to the Fisher proxy (e.g., gradient magnitudes concentrated in a few output channels, or damping μ_l being mis-scaled for Qwen’s activation range)? A diagnostic such as the condition number or effective rank of L_l per model would help.
  2. How were α_l, μ_l, and λ_l chosen for UDA-SVD, and were SAES-SVD’s α and GFW-SVD’s Fisher damping tuned with equivalent effort on the same calibration set? If UDA-SVD’s hyperparameters were selected on any evaluation data, please say so.
  3. Appendix D says calibration uses a “stratified task-mixture protocol.” Does this mixture draw from the same task families as the zero-shot benchmarks (ARC, PIQA, HellaSwag, etc.)? If so, how much of the accuracy gain survives when calibrating on WikiText-2 or C4 only?
  4. Please report whole-model parameter retention and a compression-time breakdown. The method needs a backward pass through the full-precision model for L_l and paired forward passes for Δ_l, so its cost relative to SAES-SVD matters for the practicality claim.
  5. The 1.29–2.37× speedup in Fig. 3: was this measured on the factorized A_l B_l path or on the dense write-back used for quality evaluation (Appendix C.4)? If the latter, the speedup should be re-measured.
  6. Perplexity gains over SAES-SVD on LLaMA-3-8B are 19–40%, but accuracy gains are about one point. Do you have an explanation for why the perplexity improvement translates so weakly to downstream tasks?

Comments on the paper

2 comments

I thank the authors for their time and for their careful responses to the raised concerns.

Novelty

I appreciate that the work develops a framework for merging both upstream and downstream error, and I appreciate that this joint modeling seems to result in consistent performance improvement. However, my opinion of the overall novelty of the contribution remains unchanged. Prior work already explores upstream error compensation as well as downstream error compensation, and the double-sided whitening mechanism used for the unified objective and closed-form solution is already well established within the literature.

Qwen ablation clarification

I appreciate the authors clarifying the discrepancies in the Qwen results, and I appreciate the additional results with larger calibration sets. My main concern regarding the Qwen results is not that I believe the results are incorrect. Rather, I am concerned that the gains at 0.4 and 0.6 are not statistically significant.

Selection of βℓ\beta_\ell

I thank the authors for both clarifying how this parameter is chosen as well as the additional analysis into the effect of selecting different values. If I understand correctly, UDA-SVD uses the same value of β\beta per layer. Is this value chosen per model or do all models use the same value? I think it's important to be clear about the method of selecting this parameter. One thing I am concerned about is whether this parameter is chosen specifically to maximize performance on the tasks the authors evaluate on. I am also curious if there have been any attempts to develop a method of selecting this value per layer as done in SAES-SVD.

Thank you for the detailed response and the additional experiments. The compression-time measurements and the larger-model results are helpful. A few points remain unclear and I keep my score.

  1. Compression overhead. Table E.8 reports peak inference memory of the compressed models, whereas my question concerned peak memory during compression (calibration forward/backward passes through the full-precision model, statistic construction, and SVD). This matters most for the 70B model.

  2. Comparison with Swift-SVD. The Swift-SVD numbers in Table E.9 (57/51/44/34) coincide with Table 1 of the Swift-SVD paper, so they appear to be taken from that paper rather than reproduced under matched calibration. In addition, the UDA-SVD six-task averages in Table E.9 (53.34/47.60/39.70) differ from the six-task averages implied by your own Table E.2(c) for LLaMA-1-7B (57.65/52.18/43.79).

  3. Larger models. In Table E.10, the per-task gains at ρ = 0.4 are identical on Qwen2.5-32B and LLaMA-3.1-70B (+0.5, +0.7, +0.4, +0.3, +0.3, +0.4, +0.3), and UDA-SVD wins all 42 task-level comparisons by 0.3–0.9 points, unlike in the main tables. The table also omits seeds and standard deviations.

  4. Rank allocation. The rationale for group-wise allocation is understandable, but it explains why cross-block allocation is difficult within your framework rather than whether it would help.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.