acceptodds
Under review as a conference paper at ICLR 2027

Learning to Be Truncated: Preparing LLM Checkpoints for SVD Compression

Abstract

Compression based on singular value decomposition (SVD) often requires subsequent fine-tuning or weight correction to mitigate performance degradation. Budget-specific optimization must be repeated for different compression ratios. We introduce LLift, which prepares a pretrained model for SVD compression across ratios without further fine-tuning. LLift fine-tunes a low-rank approximation at a single ratio using the original model as a teacher. We merge the learned updates into the original weights while preserving the complementary components excluded from the training forward pass. We apply SVD to the prepared model's target matrices and reuse the decompositions at different truncation ranks. On LLaMA-7B, we demonstrate accuracy gains across three compression ratios using a single prepared model. The gains extend across model families, model sizes, and SVD compression operators. We analyze gradients and an early training update to provide local evidence that updates learned at one ratio can benefit compression at other ratios. These results show that LLift can improve compression across deployment budgets without fine-tuning each compressed model separately.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Model Compression, SVD Compression

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QKdcOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LLift, a framework that prepares a single pretrained checkpoint for SVD compression across multiple deployment budgets. It distills a low-rank approximation at one compression ratio and merges the learned updates into the original weights while preserving complementary components, enabling subsequent compression at different ranks without further fine-tuning. Experiments show improved accuracy across deployment budgets without separately fine-tuning each compressed model.

Critical strengths
  1. The paper addresses the practically relevant problem of preparing a single checkpoint for multiple deployment budgets, reducing the need for separate optimization at each target compression ratio. The method is simple and well-motivated: it distills a low-rank student within fixed subspaces, merges the learned updates into the original weights while preserving the complementary components, and truncates shared SVD decompositions at different ranks. The residual-restoration ablation supports the benefit of retaining these components for subsequent compression.

  2. The evaluation spans multiple model families, scales, and deployment budgets, with additional validation using a second SVD compression operator. This breadth provides useful evidence that the observed gains are not limited to a single model or compression configuration.

Critical weaknesses
  1. The evaluation is limited to multiple-choice benchmarks and language-modeling perplexity, leaving the preservation of generative capabilities insufficiently assessed. Including tasks that require multi-step reasoning, code generation, or instruction following would strengthen the evidence that the compressed models retain capabilities relevant to practical deployment.
  2. The performance advantage over strong elastic baselines is limited. In Table 2, LLift performs similarly to FlexRank at higher retention ratios but falls behind at 40% retention (38% vs. 41% average accuracy). Its performance is also close to ACIP with the additional correction term. The comparison with Low-Rank Prehab also remains insufficient, as the two methods use substantially different training token budgets. This limits the evidence that the proposed low-rank preparation mechanism outperforms existing model-preparation approaches under comparable conditions.
Questions for authors
  1. Does allowing the residual R0R_0 to participate in training affect the final compressed-model performance?
  2. How sensitive is the method’s performance to the choice of calibration data?
  3. The default preparation ratio of 0.6 is supported by the LLaMA-7B ablation, but its suitability across model families and deployment budgets remains unclear. How should the preparation ratio be selected when the target deployment budgets change?
4 comments
AuthorsOpenReview ↗

Response to Reviewer QKdc: Part 1 of 2

We thank the reviewer for recognizing the practical value of single-checkpoint preparation, our method's simplicity, and the broad evaluation. We address the concerns below.

W1: The evaluation is limited to multiple-choice benchmarks and language-modeling perplexity, leaving the preservation of generative capabilities insufficiently assessed...

Additional generation benchmarks. We added GSM8K generation results on Qwen3-8B (Table R1). LLift outperforms both baselines at r=0.8r=0.8 and r=0.6r=0.6, supporting improved preservation of multi-step mathematical generation.

Table R1: Qwen3-8B GSM8K strict exact-match accuracy (%, higher is better).

Methodr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
SVD-LLM24.940.000.00
ZS-SVD13.190.000.00
LLift64.6717.130.38

W2: The performance advantage over strong elastic baselines is limited...

Comparison with elastic baselines. LLift provides reusable preparation across budgets and SVD compressors. It improves LEMS accuracy at r=0.6/0.4r=0.6/0.4, and LLift + LEMS outperforms ACIP (+Δ+\Delta) at all three budgets, while FlexRank remains stronger at r=0.4r=0.4 (Table R2). With similar training exposure (about 11.5M vs. 12M token positions), LLift has an estimated preparation-stage subtotal of 2.07 A100 GPU-hours, versus about 27.0 for FlexRank (Appendix C.1).

Table R2: LLaMA-7B seven-task mean accuracy. Both LLift rows share one prepared checkpoint. The budget rr excludes the ACIP correction parameters.

Methodr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
ACIP0.490.400.33
ACIP (+Δ+\Delta)0.480.450.38
FlexRank0.490.460.41
LLift + SVD-LLM0.500.460.38
LEMS0.520.440.35
LLift + LEMS0.520.460.39

Matched training exposure with Prehab. LLift nearly retains its default-exposure accuracy when training exposure is matched to Prehab (Table R3). Our Prehab reimplementation loses substantial accuracy even before compression, so we report it as a reproduction diagnostic rather than a definitive comparison.

Table R3: LLaMA-7B seven-task mean accuracy at different training exposures. Dense denotes the uncompressed checkpoint.

CheckpointExposure (M)Dense accuracyr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
Original—0.540.440.400.31
CE fine-tuning0.790.540.330.390.30
Prehab (reimpl.)0.790.350.330.300.30
LLift0.790.540.500.460.37
LLift11.530.540.500.460.38

Q1: Does allowing the residual R_0 to participate in training affect the final compressed-model performance?

Residual exclusion during training. Including R0R_0 makes the initialized student match the teacher (Q0B0+R0=W0Q_0B_0+R_0=W_0), giving zero KL loss and gradients under identical deterministic computation, whether R0R_0 is frozen or trainable. The model would remain at initialization, yielding the original-model seven-task mean accuracy of 0.44/0.40/0.310.44/0.40/0.31 at r=0.8/0.6/0.4r=0.8/0.6/0.4 under the same SVD-LLM settings. We therefore exclude R0R_0 during training and restore it only when constructing the prepared checkpoint.

Q2: How sensitive is the method’s performance to the choice of calibration data?

Calibration-data robustness. We added a controlled deployment-calibration study (Appendix B.10, Table 19), showing that LLift's gains are consistent across calibration datasets. With the prepared checkpoint fixed, mean accuracy varies by at most 0.03 across WT2, C4, and WT2/Alpaca calibration. At r=0.4, LLift achieves 0.35/0.36/0.38 accuracy, versus 0.31/0.30/0.31 for the original checkpoint. Across all evaluated budgets and calibration datasets, LLift reduces C4 perplexity in every setting and WT2 perplexity in all but one, where it changes from 12.01 to 12.02 (r=0.8r=0.8, C4 calibration). Absolute perplexity remains sensitive to the calibration domain.

AuthorsOpenReview ↗

Response to Reviewer QKdc: Part 2 of 2

Q3: ...How should the preparation ratio be selected when the target deployment budgets change?

Anchor selection across models and budgets. Our LLaMA-7B and Qwen3-8B ablations show that all tested anchors (ρ=0.4/0.6/0.8\rho=0.4/0.6/0.8) improve mean accuracy over direct SVD-LLM compression at all three budgets (Table R4). We use ρ=0.6\rho=0.6 as a balanced default, while ρ=0.4\rho=0.4 favors accuracy under aggressive compression. The same prepared checkpoint remains reusable across deployment budgets.

Table R4: Preparation-anchor ablations under SVD-LLM (manuscript Tables 9–10). Each budget cell reports WT2 PPL / C4 PPL / seven-task mean accuracy; lower PPL and higher accuracy are better. All checkpoints use fixed original-model mix75 calibration transforms.

ModelCheckpointr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
LLaMA-7BOriginal8.03 / 13.89 / 0.4414.27 / 35.03 / 0.4063.25 / 218.52 / 0.31
LLaMA-7BLLift (ρ=0.4\rho=0.4)9.72 / 12.78 / 0.5010.89 / 17.08 / 0.4714.07 / 30.78 / 0.39
LLaMA-7BLLift (ρ=0.6\rho=0.6)7.49 / 11.15 / 0.508.40 / 14.77 / 0.4613.48 / 32.24 / 0.38
LLaMA-7BLLift (ρ=0.8\rho=0.8)7.70 / 11.22 / 0.509.53 / 16.91 / 0.4520.20 / 52.31 / 0.35
Qwen3-8BOriginal14.21 / 28.97 / 0.5231.79 / 89.05 / 0.38239.87 / 896.79 / 0.31
Qwen3-8BLLift (ρ=0.4\rho=0.4)17.95 / 37.03 / 0.5421.09 / 54.99 / 0.4732.40 / 120.76 / 0.37
Qwen3-8BLLift (ρ=0.6\rho=0.6)13.95 / 29.55 / 0.5517.85 / 46.78 / 0.4744.60 / 173.05 / 0.34
Qwen3-8BLLift (ρ=0.8\rho=0.8)12.66 / 26.23 / 0.5418.76 / 51.12 / 0.4393.18 / 434.36 / 0.32

We appreciate the reviewer's constructive feedback and hope our additional experiments and clarifications address the concerns. We would be happy to answer any further questions.

Thank you for your response and the additional experiments. Your response has addressed my concerns, and I will keep my original score.

Thank you for your positive feedback and for taking the time to review our response and additional experiments. We sincerely appreciate your thoughtful comments and consideration.

Reviewer_pAueOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes LLift, which prepares a pre-trained LLM at a single compression anchor for deployment at multiple compression budgets without further fine-tuning. The method trains a fixed low-rank representation, merges this representation with the stored original residual components excluded during preparation and then recompresses the resulting checkpoint. The main experiments and analyses focus on Llama-7B, with additional evaluation provided for Qwen3-8B, Gemma-2-9B and larger Llama models (13B/30B).

Critical strengths
  • The proposition of preparing a single LLM checkpoint for multiple compression budgets is practical and reduces repeated recovery costs.
  • The paper presents thorough evaluations which supports the benefits of the proposed method across several budgets, model families, model sizes, and two different SVD compressions.
Critical weaknesses
  • Limited comparison against critical related work: Acknowledged related work [1,2] is absent from the main evaluations in the paper. Further, LLift displays no clear performance benefits compared to FlexRank [3], and only marginally outperforms ACIP (+Δ+\Delta) [4] (cf. Table 2). While Appendix C provides a preliminary cost analysis (setup across methods seem inconsistent, so comparison is slightly ambiguous) a clearer computational cost analysis in the main content would help establish LLift's benefits over related work.
  • Limited comparison with recent SVD compression baselines: Comparison with SVD baseline such as SVD-LLMv2 [5], GFWSVD [6], KFAC-SVD [7], Swift-SVD [8] would benefit the main evaluation.
  • Experimental coverage is substantial, though concentrated on Llama-7B. Given that Llama is somewhat outdated, concentrating the main evaluations on newer model families would be preferable to present the method's performance.

[1] Yaguchi et al., Decomposable-Net: Scalable Low-Rank Compression for Neural Networks. IJCAI, 2021.

[2] Qin et al., Low-Rank Prehab: Preparing Neural Networks for SVD Compression. arXiv, 2025.

[3] Zaccone et al., FlexRank: Nested Low-Rank Knowledge Decomposition for Adaptive Model Deployment. ICML, 2026.

[4] Genzel et al., Choose Your Model Size: Any Compression of Large Language Models Without Re-Computation. arXiv 2025

[5] Wang et al., SVD-LLM V2: Optimizing Singular Value Truncation for Large Language Model Compression. NAACL, 2025.

[6] Chekalina et al., Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. ICML, 2026.

[7] Thoma et al., Advancing SVD-based LLM Compression via Layer-Wise Error Model Search. ICML, 2026.

[8] Qi et al., Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compression. ICML, 2026.

Questions for authors
  1. Given that Prehab [2] is a particularly close related work, could you provide a more rigorous comparison? Specifically, why do the training token positions differ for the two methods in the evaluation (0.79M for Prehab versus 11.5M for LLift)? Would the training results presist for the same training exposure?

  2. My intuition or expectation would be that compression scales from bottom up, i.e., if a model is prepared to handle significant compression with an anchor ρ=0.4\rho=0.4, performance would more likely be recoverable for larger parameter retention ratios (r=0.6r=0.6 or r=0.8r=0.8). While I believe Fig. 3 (a) supports this intuition, Table 8 tells a different story. Thus, I am curious why anchor ρ=0.6\rho=0.6 seems to be the best preparation anchor to retain the best performance across different deployment budgets. Particularly, why is the performance better for ρ=0.6\rho=0.6 on r=0.4r=0.4 than a model that was prepared with the anchor that matches rr (here ρ=0.4\rho=0.4)? The same trend is observable at ρ=0.8\rho=0.8, could you provide your intuition on why this might occur?

  3. Do you have an explanation, why Lift's performance at r=0.8r=0.8 seems to be worse relative to other parameter ratios (specifically in Table 1)?

  4. Could you explain the choice of perplexity datasets, specifically the omission of PTB, and provide results if available?

  5. For the computational-cost comparison briefly provided in Section 5.2 and extended in Appendix C, to what extent are the preparation and post-compression fine-tuning procedures controlled for differences in training data, optimization procedures, etc.? Could you please clarify whether LLift’s reported computational advantage is recorded under matched conditions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer pAue: Part 1 of 3

We thank the reviewer for recognizing the value of single-checkpoint preparation and our evaluation across budgets, model families, and SVD compressors. We address the concerns with additional experiments and clarifications below.

W1: Limited comparison against critical related work...

Additional related-work comparisons. We added a matched-exposure comparison with Decomposable-Net. Under shared SVD-LLM deployment, LLift achieves higher accuracy and lower perplexity at all three budgets (Table R5). We discuss the matched-exposure Prehab diagnostic under Q1.

Table R5: LLaMA-7B comparison under shared SVD-LLM deployment. WT2 and C4 report perplexity, and Avg. reports seven-task mean accuracy.

BudgetPreparationDeploymentWT2 PPLC4 PPLAvg.
r=0.8r=0.8OriginalSVD-LLM8.0313.890.44
r=0.8r=0.8Decomposable-NetSVD-LLM9.8743.270.47
r=0.8r=0.8LLiftSVD-LLM7.4911.150.50
r=0.6r=0.6OriginalSVD-LLM14.2735.030.40
r=0.6r=0.6Decomposable-NetSVD-LLM12.6885.710.42
r=0.6r=0.6LLiftSVD-LLM8.4014.770.46
r=0.4r=0.4OriginalSVD-LLM63.25218.520.31
r=0.4r=0.4Decomposable-NetSVD-LLM34.89460.140.34
r=0.4r=0.4LLiftSVD-LLM13.4832.240.38

Elastic baselines. LLift + LEMS outperforms ACIP (+Δ+\Delta) at all three budgets, while FlexRank remains stronger at r=0.4r=0.4 (Table R2). For preparation cost, we now count calibration, decomposition, rank profiling, optimization, and merging where applicable for both methods, and we exclude evaluation. Under this accounting, FlexRank uses about 12M training token positions and takes about 27.0 A100 GPU-hours, of which about 24 are per-layer profiling for its rank search. LLift uses about 11.5M token positions with an estimated preparation-stage subtotal of 2.07 A100 GPU-hours. The subtotal combines recorded runtimes and log-based estimates, with loading and saving included only where timed. The methods use different training corpora (FineWeb-Edu and WT2); Appendix C.1 specifies the timing scope.

Table R2: LLaMA-7B seven-task mean accuracy. Both LLift rows share one prepared checkpoint. The budget rr excludes the ACIP correction parameters.

Methodr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
ACIP0.490.400.33
ACIP (+Δ+\Delta)0.480.450.38
FlexRank0.490.460.41
LLift + SVD-LLM0.500.460.38
LEMS on the original checkpoint0.520.440.35
LLift + LEMS0.520.460.39

W2: Limited comparison with recent SVD compression baselines...

Recent SVD baselines. We selected Swift-SVD and KFAC-SVD with LEMS for their publicly available implementations and expanded the related-work discussion. LLift achieves higher accuracy and lower perplexity on all three corpora at r=0.6/0.4r=0.6/0.4 (Table R6). The same prepared checkpoint improves accuracy across SVD-LLM, one-shot ZS-SVD, and LEMS at these budgets (Table R7).

Table R6: LLaMA-7B comparison with recent SVD baselines under their official calibration settings. LEMS denotes KFAC-SVD with LEMS rank allocation, and Swift-SVD$^*$ denotes the released configuration.

rrMethodWT2 PPLPTB PPLC4 PPLAvg.
0.8Swift-SVD$^*$11.6414.7411.760.46
0.8LEMS6.2310.279.150.51
0.8LLift7.4911.7211.150.50
0.6Swift-SVD$^*$36.8756.1525.200.35
0.6LEMS9.2221.4619.290.43
0.6LLift8.4015.9614.770.46
0.4Swift-SVD$^*$257.70400.56143.850.30
0.4LEMS21.93104.0583.940.34
0.4LLift13.4853.4732.240.38

Table R7: LLaMA-7B seven-task mean accuracy across compressors using one prepared checkpoint and WT2/Alpaca 75:25 calibration.

CompressorCheckpointr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
SVD-LLMOriginal0.440.400.31
SVD-LLMLLift0.500.460.38
ZS-SVDOriginal0.510.420.33
ZS-SVDLLift0.510.450.38
LEMSOriginal0.520.440.35
LEMSLLift0.520.460.39
AuthorsOpenReview ↗

Response to Reviewer pAue: Part 2 of 3

W3: ...concentrating the main evaluations on newer model families would be preferable to present the method's performance.

Evaluation on newer model families. Our main evaluation includes Qwen3-8B and Gemma-2-9B at three budgets. In the expanded Qwen3-8B comparison, LLift + LEMS improves accuracy over LEMS at every evaluated budget (Table R8).

Table R8: Additional compression results on Qwen3-8B. Seven-task mean accuracy (higher is better). Baselines use their official calibration settings, while LLift uses mix75.

Methodr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
SVD-LLM0.490.350.31
ZS-SVD0.550.400.31
Swift-SVD$^*$0.530.430.33
LEMS0.560.430.33
LLift + SVD-LLM0.550.470.34
LLift + LEMS0.590.510.38

Q1: Given that Prehab [2] is a particularly close related work, could you provide a more rigorous comparison...

Matched training exposure with Prehab. The original comparison used each method's prescribed training exposure. After matching sequences, epochs, batch size, and optimizer steps, LLift nearly retains its default-exposure accuracy, although perplexity increases (Table R3). Our Prehab adaptation already loses uncompressed accuracy, so this remains a reproduction diagnostic.

Table R3: LLaMA-7B seven-task mean accuracy at different training exposures. Dense denotes the uncompressed checkpoint.

CheckpointExposure (M)Dense accuracyr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
Original—0.540.440.400.31
CE fine-tuning0.790.540.330.390.30
Prehab (reimpl.)0.790.350.330.300.30
LLift0.790.540.500.460.37
LLift11.530.540.500.460.38

Q2: My intuition or expectation would be that compression scales from bottom up...

Understanding anchor-dependent performance. Figure 3(a) reports accuracy, whereas the original Table 8 reports WT2 perplexity. On LLaMA-7B at r=0.4r=0.4, ρ=0.6\rho=0.6 gives lower WT2 perplexity, while ρ=0.4\rho=0.4 gives higher accuracy and lower C4 perplexity. On Qwen3-8B, matched anchors give the lowest WT2/C4 perplexity among those tested (Table R4). Anchor rankings therefore depend on the metric and model, but all tested anchors improve mean accuracy over direct SVD-LLM compression at every evaluated budget on both models.

Table R4: Preparation-anchor ablations under SVD-LLM. Each budget cell reports WT2 PPL / C4 PPL / mean accuracy. All checkpoints use fixed original-model mix75 calibration transforms.

ModelCheckpointr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
LLaMA-7BOriginal8.03 / 13.89 / 0.4414.27 / 35.03 / 0.4063.25 / 218.52 / 0.31
LLaMA-7BLLift (ρ=0.4\rho=0.4)9.72 / 12.78 / 0.5010.89 / 17.08 / 0.4714.07 / 30.78 / 0.39
LLaMA-7BLLift (ρ=0.6\rho=0.6)7.49 / 11.15 / 0.508.40 / 14.77 / 0.4613.48 / 32.24 / 0.38
LLaMA-7BLLift (ρ=0.8\rho=0.8)7.70 / 11.22 / 0.509.53 / 16.91 / 0.4520.20 / 52.31 / 0.35
Qwen3-8BOriginal14.21 / 28.97 / 0.5231.79 / 89.05 / 0.38239.87 / 896.79 / 0.31
Qwen3-8BLLift (ρ=0.4\rho=0.4)17.95 / 37.03 / 0.5421.09 / 54.99 / 0.4732.40 / 120.76 / 0.37
Qwen3-8BLLift (ρ=0.6\rho=0.6)13.95 / 29.55 / 0.5517.85 / 46.78 / 0.4744.60 / 173.05 / 0.34
Qwen3-8BLLift (ρ=0.8\rho=0.8)12.66 / 26.23 / 0.5418.76 / 51.12 / 0.4393.18 / 434.36 / 0.32

Q3: Do you have an explanation, why Lift's performance at r=0.8 seems to be worse relative to other parameter ratios (specifically in Table 1)?

Performance at high retention ratios. LLift has less room to improve performance at r=0.8, where compression introduces relatively little distortion. This is consistent with its smaller relative gains at high retention and more pronounced benefits under aggressive compression (Table 1).

Q4: Could you explain the choice of perplexity datasets, specifically the omission of PTB, and provide results if available?

PTB perplexity evaluation. We added PTB results, in which LLift achieves the lowest perplexity among the compared methods at r=0.6/0.4r=0.6/0.4 (Table R9). The original WT2/C4 selection followed SVD-LLM's evaluation protocol.

Table R9: PTB perplexity on LLaMA-7B (lower is better). The uncompressed model has PTB perplexity 8.35. The dagger denotes budget-specific optimization.

Methodr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
ASVD16.55329245218
SVD-LLM16.2263.75438.58
Dobi-SVD$^\dagger$14.8346.38238.91
SAES-SVD$^\dagger$15.1645.13116.83
ZS-SVD11.8743.19334.85
ZS-SVD 5$\times^\dagger$11.1736.52163.58
Swift-SVD$^*$14.7456.15400.56
LEMS10.2721.46104.05
LLift11.7215.9653.47
AuthorsOpenReview ↗

Response to Reviewer pAue: Part 3 of 3

Q5: For the computational-cost comparison briefly provided in Section 5.2 and extended in Appendix C...

Computational-cost comparison. The original comparison used SVD-LLM's two-stage LoRA recovery on Alpaca and LLift's WT2 KL preparation. We added recovery controls matching LLift's data, objective, optimizer settings, deployment calibration, and training exposure per run (11.5M/10.2M token positions for LLaMA-7B/Qwen3-8B).

For N=3N=3 budgets (r=0.8/0.6/0.4r=0.8/0.6/0.4), LLift achieves broadly comparable mean accuracy at r=0.8/0.6r=0.8/0.6 on both models with one shared preparation run, while recovery trains each budget separately (Table R11). Including deployment calibration and SVD compression, the LLaMA-7B pipeline costs are 3.13 A100 GPU-hours for LLift and 5.10 for recovery (Table R10).

Table R10: LLaMA-7B training and pipeline costs (manuscript Table 24). Optimization and pipeline costs are in A100 GPU-hours.

MethodTraining runsTotal positions (M)OptimizationPipeline
LLift111.501.553.13
Recovery334.504.005.10

Table R11: Budget-specific SVD-LLM recovery and LLift (manuscript Table 16). Each cell reports WT2 PPL / C4 PPL / seven-task mean accuracy.

ModelMethodr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
LLaMA-7BLLift7.49 / 11.15 / 0.508.40 / 14.77 / 0.4613.48 / 32.24 / 0.38
LLaMA-7BRecovery6.75 / 10.52 / 0.507.96 / 14.05 / 0.4610.97 / 23.40 / 0.40
Qwen3-8BLLift13.95 / 29.55 / 0.5517.85 / 46.78 / 0.4744.60 / 173.05 / 0.34
Qwen3-8BRecovery12.08 / 28.45 / 0.5515.78 / 38.82 / 0.4826.26 / 79.88 / 0.39

We appreciate the reviewer's constructive feedback and hope the additional experiments and clarifications address the concerns. We would be happy to answer any further questions.

Reviewer_h86cOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2(+1)
Presentation
3
Soundness
2
Read review
Paper summary

LLift prepares a pretrained model for compression at multiple ratios with a single fine-tuning run, rather than optimizing each compressed version separately. It first constructs a low-rank approximation at a chosen compression ratio and fine-tunes it using KL divergence to match the original model’s predictions. The learned updates are then merged into the original weights while preserving the components excluded during training. The resulting checkpoint can be compressed at different ratios using SVD-based methods without additional fine-tuning. Experiments on LLaMA, Qwen, and Gemma models show improvements over training-free compression methods, particularly at higher compression levels, while achieving results comparable to budget-specific fine-tuning at a lower training cost.

Critical strengths

1. Practical relevance: Preparing a model once for deployment at multiple compression ratios addresses a practical deployment requirement. The proposed workflow is also straightforward to integrate with existing SVD-based compression methods.

2. Consistent performance across configurations: The authors evaluate LLift across multiple model families and sizes, as well as two compression operators, SVD-LLM and ZS-SVD. The results show relatively consistent improvements, particularly at more aggressive compression ratios.

3. Clarity: The paper is easy to follow, and the method is straightforward to understand.

Critical weaknesses

1. Limited novelty: The method combines existing components: activation-aware SVD compression, distillation from the original model, and a frozen residual as in PiSSA. The only new step is reattaching the residual and recompressing at other ratios. The supporting result (Eq. 9) is an orthogonal decomposition, which the authors acknowledge does not imply task-loss optimality.

2. Residual reattachment mixes two objectives: In W⋆=Q0B⋆+R0W^\star = Q_0B^\star + R_0, the head Q0B⋆Q_0B^\star is trained with KL divergence while the residual R0R_0 comes from the whitened reconstruction objective and is never trained. Since the head is optimized without R0R_0 in the forward pass, the paper offers no justification beyond empirical results for why the combination should work.

3. Heuristic anchor selection: The default ρ=0.6\rho=0.6 is chosen from three anchors tested only on LLaMA-7B, with no repeated seeds reported, and is reused for all other models. The anchor-matched model at r=0.4r=0.4 outperforms the default by 1.66 pp, so the "single preparation run" claim still depends on a model- and budget-specific choice of ρ\rho.

4. Unfair comparisons: None of the baselines in Table 1 fine-tunes the model weights, while LLift is fine-tuned on about 11.5M tokens. Against the only fine-tuned baseline (SVD-LLM + LoRA, Table 13), LLift ties or trails in accuracy: 0.50/0.46/0.38 vs. 0.50/0.46/0.39.

5. Risk of overfitting to WikiText-2: LLift also trains the embeddings, LM head, and norms on the small WikiText-2 training set, which is also an evaluation metric. Freezing them raises WikiText-2 perplexity by 12–13% but C4 by only 4–5%, with accuracy unchanged (Table 7), which suggests in-domain fitting.

6. Incomplete cost reporting: Preparation cost is reported only for LLaMA-7B and for LLaMA-13B optimization (Table 14). It is missing for Qwen3-8B, Gemma-2-9B, and LLaMA-30B, and the paper reports no peak memory, latency, or throughput.

Questions for authors

1. Matched fine-tuned baseline: How does LLift compare with budget-specific KL distillation using the same WikiText-2 data and 11.5M-token budget, or with the Table 1 baselines followed by the same recovery fine-tuning?

2. Anchor selection: How should ρ\rho be chosen for a new model without a sweep? Was ρ=0.6\rho=0.6 validated on Qwen3-8B, Gemma-2-9B, LLaMA-13B, or LLaMA-30B, or carried over from LLaMA-7B?

3. Generation tasks: All downstream evaluations are likelihood-scored multiple-choice tasks. Have the authors evaluated generation tasks such as GSM8K or instruction following, where compression often causes larger degradation?

5 comments
AuthorsOpenReview ↗

Response to Reviewer h86c: Part 1 of 3

We thank the reviewer for recognizing LLift's practical relevance, its improvements across models and compression settings, and the clarity of our presentation. We address the concerns below.

W1: Limited novelty: The method combines existing components...

Novelty and reusable preparation. LLift contributes a reusable preparation mechanism rather than a new SVD operator. Unlike PiSSA, it excludes the frozen residual during KL training to expose compression-induced errors, then restores it before applying the deployment compressor. The prepared checkpoint supports multiple budgets and SVD compressors (Tables R7 and R12).

Table R12: Model preparation and residual restoration on LLaMA-7B under matched WT2/Alpaca 75:25 calibration. Seven-task mean accuracy.

Checkpointr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
Original0.440.400.31
Spectral regularization0.480.410.33
Fine-tuned Q0B⋆Q_0B^\star only0.440.440.37
LLift: Q0B⋆+R0Q_0B^\star+R_00.500.460.38

Table R7: LLaMA-7B seven-task mean accuracy across compressors using one prepared checkpoint and WT2/Alpaca 75:25 calibration.

CompressorCheckpointr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
SVD-LLMOriginal0.440.400.31
SVD-LLMLLift0.500.460.38
ZS-SVDOriginal0.510.420.33
ZS-SVDLLift0.510.450.38
LEMSOriginal0.520.440.35
LEMSLLift0.520.460.39

W2: ...the paper offers no justification beyond empirical results for why the combination should work.

Residual restoration. Equation 9 establishes minimum-change completion, and Appendix A.2 (Eqs. 14–15) gives sufficient conditions for reducing teacher reconstruction error after recompression. Our paired diagnostic shows reduced teacher reconstruction error for all 224 target matrices at all three budgets (Appendix B.1), alongside downstream accuracy gains (Table R12). These results do not establish task-loss optimality.

W3: Heuristic anchor selection...

Q2: Anchor selection: How should rho be chosen for a new model without a sweep...

Anchor-selection robustness. We used ρ=0.6\rho=0.6 for all main experiments without model-specific tuning. Across LLaMA-7B and Qwen3-8B, all tested anchors (ρ=0.4/0.6/0.8\rho=0.4/0.6/0.8) improve mean accuracy over direct SVD-LLM compression at every evaluated budget (Table R4). The gains therefore persist across anchor choices, with ρ=0.4\rho=0.4 or 0.60.6 generally providing a favorable balance across budgets and metrics.

Table R4: Preparation-anchor ablations under SVD-LLM (accuracy excerpt; manuscript Tables 9–10). Seven-task mean accuracy (higher is better). All checkpoints use fixed original-model mix75 calibration transforms.

ModelCheckpointr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
LLaMA-7BOriginal0.440.400.31
LLaMA-7BLLift (ρ=0.4\rho=0.4)0.500.470.39
LLaMA-7BLLift (ρ=0.6\rho=0.6)0.500.460.38
LLaMA-7BLLift (ρ=0.8\rho=0.8)0.500.450.35
Qwen3-8BOriginal0.520.380.31
Qwen3-8BLLift (ρ=0.4\rho=0.4)0.540.470.37
Qwen3-8BLLift (ρ=0.6\rho=0.6)0.550.470.34
Qwen3-8BLLift (ρ=0.8\rho=0.8)0.540.430.32
AuthorsOpenReview ↗

Response to Reviewer h86c: Part 2 of 3

W4: Unfair comparisons: None of the baselines in Table 1 fine-tunes the model weights, while LLift is fine-tuned on about 11.5M tokens...

Q1: Matched fine-tuned baseline...

Comparison with budget-specific fine-tuning. We added SVD-LLM recovery on LLaMA-7B and Qwen3-8B, matching LLift's WT2 data, KL objective, frozen teacher, optimizer settings, trainable parameter groups, and deployment calibration. Each recovery run follows LLift's four-epoch training schedule.

For N=3N=3 budgets (r=0.8/0.6/0.4r=0.8/0.6/0.4), LLift achieves broadly comparable seven-task mean accuracy at r=0.8/0.6r=0.8/0.6 on both models with one shared preparation run, while recovery trains each budget separately (Table R11). Recovery yields lower WT2 and C4 perplexity. Including deployment calibration and SVD compression, the estimated LLaMA-7B pipeline costs are 3.13 A100 GPU-hours for LLift and 5.10 for recovery (Table R10).

Table R11: Budget-specific SVD-LLM recovery and LLift (manuscript Table 16). Each cell reports WT2 PPL / C4 PPL / seven-task mean accuracy.

ModelMethodr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
LLaMA-7BLLift7.49 / 11.15 / 0.508.40 / 14.77 / 0.4613.48 / 32.24 / 0.38
LLaMA-7BRecovery6.75 / 10.52 / 0.507.96 / 14.05 / 0.4610.97 / 23.40 / 0.40
Qwen3-8BLLift13.95 / 29.55 / 0.5517.85 / 46.78 / 0.4744.60 / 173.05 / 0.34
Qwen3-8BRecovery12.08 / 28.45 / 0.5515.78 / 38.82 / 0.4826.26 / 79.88 / 0.39

Table R10: LLaMA-7B training and pipeline costs (manuscript Table 24). Optimization and pipeline costs are in A100 GPU-hours.

MethodTraining runsTotal positions (M)OptimizationPipeline
LLift111.501.553.13
Recovery334.504.005.10

Table R13 reports Qwen3-8B GSM8K results. Per-budget exposure gives each recovery model the training exposure of one LLift run. Total exposure divides that exposure across N=3N=3 budgets (r=0.8/0.6/0.4r=0.8/0.6/0.4) or N=5N=5 budgets (r=0.8/0.7/0.6/0.5/0.4r=0.8/0.7/0.6/0.5/0.4). LLift uses the same prepared checkpoint throughout. The prepared checkpoint also supports ZS-SVD and LEMS without further training (Table R7).

Table R13: Qwen3-8B GSM8K strict exact-match accuracy (%, manuscript Table 17).

Methodr=0.8r=0.8r=0.7r=0.7r=0.6r=0.6r=0.5r=0.5r=0.4r=0.4
LLift64.6746.3217.132.050.38
Recovery, per-budget (N=3N=3)61.41—17.97—1.21
Recovery, total (N=3N=3)60.65—13.65—1.44
Recovery, total (N=5N=5)58.1522.826.972.350.76

W5: Risk of overfitting to WikiText-2...

Generalization beyond WikiText-2. Freezing embeddings, the LM head, and norms preserves LLift's accuracy gains and C4 perplexity improvements (Table R14). Training them further improves perplexity, especially on WT2. Full LLift also achieves the lowest PTB perplexity among the listed baselines at r=0.6/0.4r=0.6/0.4 (manuscript Table 1), despite PTB being excluded from preparation and calibration.

Table R14: Freezing non-target parameters Φ\Phi during LLaMA-7B preparation. All rows use SVD-LLM with shared fixed deployment calibration transforms.

rrCheckpointWT2 PPLC4 PPLAvg.
0.8Original8.0313.890.44
0.8LLift (frozen Φ\Phi)8.4011.580.50
0.8LLift (trainable Φ\Phi)7.4911.150.50
0.6Original14.2735.030.40
0.6LLift (frozen Φ\Phi)9.4615.460.46
0.6LLift (trainable Φ\Phi)8.4014.770.46
0.4Original63.25218.520.31
0.4LLift (frozen Φ\Phi)15.1233.650.38
0.4LLift (trainable Φ\Phi)13.4832.240.38
AuthorsOpenReview ↗

Response to Reviewer h86c: Part 3 of 3

W6: Incomplete cost reporting...

Preparation and deployment resources. We added preparation-cost and training-resource measurements for all five models (Tables R15–R16), plus inference benchmarks for dense and SVD-LLM factorized shapes with random FP16 weights (manuscript Table 25, Appendix C.3). Inference costs depend on the downstream compressor and deployment configuration.

Table R15: LLift preparation-stage costs in A100-80GB GPU-hours. Deployment costs are excluded. Subtotals are computed before rounding.

ModelWhiteningBasesOptimizationMergeSubtotal
LLaMA-7B0.400.091.560.022.07
Qwen3-8B0.550.101.750.012.42
Gemma-2-9B0.870.151.430.012.47
LLaMA-13B3.970.154.970.029.11
LLaMA-30B11.480.558.170.0520.25

Table R16: LLift optimization resources on four A100-80GB GPUs. Peak is allocated GPU memory on rank 0. Throughput is estimated from Trainer sample throughput.

ModelStepsTime (s)Positions/s/GPUPeak (GiB)CPU offload
LLaMA-7B1761405.4204532.18No
Qwen3-8B1561573.2160147.74No
Gemma-2-9B1521289.1188760.67No
LLaMA-13B1764472.464342.89Yes
LLaMA-30B1767350.239141.12Yes

Q3: Generation tasks: All downstream evaluations are likelihood-scored multiple-choice tasks...

Generation-task evaluation. We added GSM8K generation results on Qwen3-8B (Table R1). LLift outperforms both baselines at r=0.8r=0.8 and r=0.6r=0.6, supporting improved preservation of multi-step mathematical generation.

Table R1: Qwen3-8B GSM8K strict exact-match accuracy (%, higher is better).

Methodr=0.8r=0.8r=0.6r=0.6r=0.4r=0.4
SVD-LLM24.940.000.00
ZS-SVD13.190.000.00
LLift64.6717.130.38

We appreciate the reviewer's constructive feedback and hope our additional analyses and experiments address the concerns. We would be happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

Dear Area Chair and Reviewers,

We sincerely thank the reviewers for their constructive feedback and engagement throughout the discussion. We summarize their recognition of our contributions and our responses to the main concerns below.

QKdc confirmed that their concerns had been addressed and maintained their score of 3, while pAue acknowledged that their evaluation concerns had been substantially addressed and expressed an inclination to raise their score (currently 2). h86c stated that their main concerns had largely been addressed, increased their score to 3, and recommended acceptance.

Strengths highlighted by the reviewers:

  • Reviewer QKdc appreciated the simple and well-motivated method and the evaluation across multiple model families, scales, and deployment budgets.
  • Reviewer pAue highlighted that preparing a single checkpoint for multiple compression budgets is practical and reduces repeated recovery costs.
  • Reviewer h86c recognized that the workflow is straightforward to integrate with existing SVD-based compression methods.

Main concerns addressed:

  1. Stronger baselines and broader evaluation (pAue, QKdc). We added comparisons with Decomposable-Net, Swift-SVD, and KFAC-SVD with LEMS rank allocation, expanded the evaluation on Qwen3-8B, and reported PTB perplexity. We also clarified comparisons with FlexRank and ACIP. The results further demonstrate that a single LLift-prepared checkpoint improves compression across different budgets and SVD operators, particularly under stronger compression.

  2. Fair comparisons with fine-tuned baselines and computational efficiency (h86c, pAue). We introduced budget-specific SVD-LLM recovery experiments under matched training data, KL objectives, optimization settings, and deployment calibration. LLift achieves comparable mean accuracy at moderate compression levels while requiring only one preparation run. For three LLaMA-7B deployment budgets, its total pipeline cost is 3.13 versus 5.10 A100 GPU-hours. We also expanded resource reporting across five models and clarified the cost accounting for elastic baselines. The revised Section 5.2 also discusses the trade-off between LLift's shared preparation cost across multiple deployment budgets and the higher per-model quality achievable through separate recovery, particularly under stronger compression.

  3. Mechanistic justification and robustness (h86c, QKdc). We strengthened the analysis of residual restoration through additional ablations and reconstruction-error diagnostics. We also expanded anchor-selection experiments to Qwen3-8B, showing consistent accuracy improvements across all tested anchors and deployment budgets. Additional calibration-data and parameter-freezing studies further support the robustness of LLift's compression gains.

  4. Generative capabilities and generalization (h86c, pAue, QKdc). We added GSM8K generation experiments on Qwen3-8B, where LLift substantially outperforms SVD-LLM and ZS-SVD at 80% and 60% retention, providing additional evidence of preserved multi-step mathematical reasoning capabilities after compression. We also added PTB evaluation and further C4 analyses to assess generalization beyond WikiText-2.

These additions and clarifications have been incorporated into the revised manuscript. We sincerely thank the reviewers for helping us strengthen the evaluation and clarify LLift’s practical contributions and trade-offs.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.