- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper introduces LLift, a framework that prepares a single pretrained checkpoint for SVD compression across multiple deployment budgets. It distills a low-rank approximation at one compression ratio and merges the learned updates into the original weights while preserving complementary components, enabling subsequent compression at different ranks without further fine-tuning. Experiments show improved accuracy across deployment budgets without separately fine-tuning each compressed model.
Critical strengths
-
The paper addresses the practically relevant problem of preparing a single checkpoint for multiple deployment budgets, reducing the need for separate optimization at each target compression ratio. The method is simple and well-motivated: it distills a low-rank student within fixed subspaces, merges the learned updates into the original weights while preserving the complementary components, and truncates shared SVD decompositions at different ranks. The residual-restoration ablation supports the benefit of retaining these components for subsequent compression.
-
The evaluation spans multiple model families, scales, and deployment budgets, with additional validation using a second SVD compression operator. This breadth provides useful evidence that the observed gains are not limited to a single model or compression configuration.
Critical weaknesses
- The evaluation is limited to multiple-choice benchmarks and language-modeling perplexity, leaving the preservation of generative capabilities insufficiently assessed. Including tasks that require multi-step reasoning, code generation, or instruction following would strengthen the evidence that the compressed models retain capabilities relevant to practical deployment.
- The performance advantage over strong elastic baselines is limited. In Table 2, LLift performs similarly to FlexRank at higher retention ratios but falls behind at 40% retention (38% vs. 41% average accuracy). Its performance is also close to ACIP with the additional correction term. The comparison with Low-Rank Prehab also remains insufficient, as the two methods use substantially different training token budgets. This limits the evidence that the proposed low-rank preparation mechanism outperforms existing model-preparation approaches under comparable conditions.
Questions for authors
- Does allowing the residual to participate in training affect the final compressed-model performance?
- How sensitive is the method’s performance to the choice of calibration data?
- The default preparation ratio of 0.6 is supported by the LLaMA-7B ablation, but its suitability across model families and deployment budgets remains unclear. How should the preparation ratio be selected when the target deployment budgets change?
4 comments
Response to Reviewer QKdc: Part 2 of 2
Q3: ...How should the preparation ratio be selected when the target deployment budgets change?
Anchor selection across models and budgets. Our LLaMA-7B and Qwen3-8B ablations show that all tested anchors () improve mean accuracy over direct SVD-LLM compression at all three budgets (Table R4). We use as a balanced default, while favors accuracy under aggressive compression. The same prepared checkpoint remains reusable across deployment budgets.
Table R4: Preparation-anchor ablations under SVD-LLM (manuscript Tables 9–10). Each budget cell reports WT2 PPL / C4 PPL / seven-task mean accuracy; lower PPL and higher accuracy are better. All checkpoints use fixed original-model mix75 calibration transforms.
| Model | Checkpoint | |||
|---|---|---|---|---|
| LLaMA-7B | Original | 8.03 / 13.89 / 0.44 | 14.27 / 35.03 / 0.40 | 63.25 / 218.52 / 0.31 |
| LLaMA-7B | LLift () | 9.72 / 12.78 / 0.50 | 10.89 / 17.08 / 0.47 | 14.07 / 30.78 / 0.39 |
| LLaMA-7B | LLift () | 7.49 / 11.15 / 0.50 | 8.40 / 14.77 / 0.46 | 13.48 / 32.24 / 0.38 |
| LLaMA-7B | LLift () | 7.70 / 11.22 / 0.50 | 9.53 / 16.91 / 0.45 | 20.20 / 52.31 / 0.35 |
| Qwen3-8B | Original | 14.21 / 28.97 / 0.52 | 31.79 / 89.05 / 0.38 | 239.87 / 896.79 / 0.31 |
| Qwen3-8B | LLift () | 17.95 / 37.03 / 0.54 | 21.09 / 54.99 / 0.47 | 32.40 / 120.76 / 0.37 |
| Qwen3-8B | LLift () | 13.95 / 29.55 / 0.55 | 17.85 / 46.78 / 0.47 | 44.60 / 173.05 / 0.34 |
| Qwen3-8B | LLift () | 12.66 / 26.23 / 0.54 | 18.76 / 51.12 / 0.43 | 93.18 / 434.36 / 0.32 |
We appreciate the reviewer's constructive feedback and hope our additional experiments and clarifications address the concerns. We would be happy to answer any further questions.
Thank you for your response and the additional experiments. Your response has addressed my concerns, and I will keep my original score.
Thank you for your positive feedback and for taking the time to review our response and additional experiments. We sincerely appreciate your thoughtful comments and consideration.
Response to Reviewer QKdc: Part 1 of 2
We thank the reviewer for recognizing the practical value of single-checkpoint preparation, our method's simplicity, and the broad evaluation. We address the concerns below.
Additional generation benchmarks. We added GSM8K generation results on Qwen3-8B (Table R1). LLift outperforms both baselines at r=0.8 and r=0.6, supporting improved preservation of multi-step mathematical generation.
Table R1: Qwen3-8B GSM8K strict exact-match accuracy (%, higher is better).
Comparison with elastic baselines. LLift provides reusable preparation across budgets and SVD compressors. It improves LEMS accuracy at r=0.6/0.4, and LLift + LEMS outperforms ACIP (+Δ) at all three budgets, while FlexRank remains stronger at r=0.4 (Table R2). With similar training exposure (about 11.5M vs. 12M token positions), LLift has an estimated preparation-stage subtotal of 2.07 A100 GPU-hours, versus about 27.0 for FlexRank (Appendix C.1).
Table R2: LLaMA-7B seven-task mean accuracy. Both LLift rows share one prepared checkpoint. The budget r excludes the ACIP correction parameters.
Matched training exposure with Prehab. LLift nearly retains its default-exposure accuracy when training exposure is matched to Prehab (Table R3). Our Prehab reimplementation loses substantial accuracy even before compression, so we report it as a reproduction diagnostic rather than a definitive comparison.
Table R3: LLaMA-7B seven-task mean accuracy at different training exposures. Dense denotes the uncompressed checkpoint.
Residual exclusion during training. Including R0 makes the initialized student match the teacher (Q0B0+R0=W0), giving zero KL loss and gradients under identical deterministic computation, whether R0 is frozen or trainable. The model would remain at initialization, yielding the original-model seven-task mean accuracy of 0.44/0.40/0.31 at r=0.8/0.6/0.4 under the same SVD-LLM settings. We therefore exclude R0 during training and restore it only when constructing the prepared checkpoint.
Calibration-data robustness. We added a controlled deployment-calibration study (Appendix B.10, Table 19), showing that LLift's gains are consistent across calibration datasets. With the prepared checkpoint fixed, mean accuracy varies by at most 0.03 across WT2, C4, and WT2/Alpaca calibration. At r=0.4, LLift achieves 0.35/0.36/0.38 accuracy, versus 0.31/0.30/0.31 for the original checkpoint. Across all evaluated budgets and calibration datasets, LLift reduces C4 perplexity in every setting and WT2 perplexity in all but one, where it changes from 12.01 to 12.02 (r=0.8, C4 calibration). Absolute perplexity remains sensitive to the calibration domain.