- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes UDA-SVD, a post-training low-rank compression method for LLM. It jointly models upstream activation drift caused by earlier compression and downstream sensitivity to errors at the current module. Experimental results demonstrate substantial memory savings and inference speedup.
Critical strengths
-
The paper clearly identifies two important factors in sequential low-rank compression: upstream activation drift and downstream sensitivity. It explains why modeling both factors jointly, together with adaptive rank allocation, may improve compression quality.
-
The paper gives a clear mathematical formulation that combines current-path reconstruction and compensation for upstream drift, while weighting output directions by an estimated downstream sensitivity. The two-sided transformation provides a closed-form solution for the resulting fixed-statistics local problem.
-
The experiments cover multiple model families, compression budgets, perplexity datasets, zero-shot tasks, ablations, and deployment-oriented memory/speed measurements. The reported gains over the included baselines are generally consistent.
Critical weaknesses
-
UDA-SVD requires calibration forward/backward steps, statistic construction, SVD, and sequential path updates, but the paper reports neither compression time nor peak memory.
-
Rank allocation is limited to predefined groups within each Transformer block, with no budget transfer across layers or attention/MLP groups. The paper does not sufficiently justify this restriction or evaluate its impact relative to cross-layer allocation.
-
Some closely related works are missing. For example, [1] SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression, and [2] Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression.
-
The design challenge and the critical design innovation in each solution component are not clear. They are reasonable but plain, lacking solution depth.
-
The models in evaluation are relatively not large. It is questionable how the solution would perform in larger models.
Questions for authors
- What are the total offline compression time and peak memory usage of UDA-SVD, including calibration, gradient computation, and decomposition? How do they compare with the evaluated baselines under the same hardware and calibration settings?
- Why is rank allocation restricted to predefined groups within each Transformer block? Would allowing budget transfer across layers improve compression quality?
- How does UDA-SVD compare with [1] and [2], both of which incorporate adaptive rank or compression-ratio allocation? A comparison under matched parameter budgets and calibration settings would help establish the benefit of the proposed approach.
I thank the authors for their time and for their careful responses to the raised concerns.
I appreciate that the work develops a framework for merging both upstream and downstream error, and I appreciate that this joint modeling seems to result in consistent performance improvement. However, my opinion of the overall novelty of the contribution remains unchanged. Prior work already explores upstream error compensation as well as downstream error compensation, and the double-sided whitening mechanism used for the unified objective and closed-form solution is already well established within the literature.
I appreciate the authors clarifying the discrepancies in the Qwen results, and I appreciate the additional results with larger calibration sets. My main concern regarding the Qwen results is not that I believe the results are incorrect. Rather, I am concerned that the gains at 0.4 and 0.6 are not statistically significant.
I thank the authors for both clarifying how this parameter is chosen as well as the additional analysis into the effect of selecting different values. If I understand correctly, UDA-SVD uses the same value of β per layer. Is this value chosen per model or do all models use the same value? I think it's important to be clear about the method of selecting this parameter. One thing I am concerned about is whether this parameter is chosen specifically to maximize performance on the tasks the authors evaluate on. I am also curious if there have been any attempts to develop a method of selecting this value per layer as done in SAES-SVD.