acceptodds
Under review as a conference paper at ICLR 2027

SIFT-LoRA: Choose What a Layer-Sparse Adapter Inherits

Abstract

A trained LoRA adapter should be reusable across deployment storage budgets chosen after task adaptation. We introduce SIFT-LoRA, which compiles a same-task Full-LoRA parent into a layer-sparse adapter under an exact serialized-byte cap without further training or changing the retained weights. A layer’s value can depend on which other updates are retained, so fixed importance scores may miss its contribution to the sparse child. Our selector, Conditional SIFT-KL, evaluates how much each candidate layer reduces parent–child predictive KL when added to the current child. It chooses the feasible addition with the largest positive reduction per added serialized byte, then recomputes these reductions for the remaining candidates. Across three models and three tasks at 12.5% and 25% of the full adapter’s serialized size, Conditional SIFT-KL lowers predictive KL against each of the fixed-score and contiguous-layer baselines in 16 of 18 conditions. It also achieves the highest task utility among the three selectors in 14 of 18 conditions, including ties. On Qwen3.5-2B/HellaSwag at the 12.5% cap, it improves mean accuracy over fixed scoring by 3.0 percentage points across three independently trained parents at nearly identical adapter sizes. In a separate seven-method BoolQ comparison at the 12.5% cap under unmerged bf16 H100 inference, SIFT-LoRA lies on the observed utility–latency Pareto frontier for all three models. Broader deployment experiments cover five tasks and both byte budgets. These results support choosing layers by what they add to the retained set and separating task learning from deployment capacity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Low-Rank Adaptation (LoRA), Parameter-Efficient Fine-Tuning, Sparse Adapters, Post-Training Compression, Structured Sparsity, Efficient Inference

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m7axOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper states that a trained LoRA adapter should be reusable across deployment storage budgets. Furthermore, the paper introduces SIFT-LoRA, which compiles Full-LoRA parent into a layer-sparse adapter without additional training or changing model weights. The proposed selector, Conditional SIFT-KL evaluates how much each layer reduces predictive KL when added to the current child. The method lowers predictive KL against baselines in 16 out of 18 settings.

Critical strengths
  • The core technique is sound and supported with results
  • The method is practical as it does not need to be retrained, and SIFT-LoRA outperforms the baselines evaluated against
Critical weaknesses

Unclear practical motivation and benefit over retraining: The paper does not clearly explain when the budget would become known only after LoRA training, or why one could not wait until the budget is known. The method should be compared with training LoRA using a rank that fits the budget. This comparison should include downstream performance and computational cost. Without it, the practical benefit of the method remains unclear.

Unclear impact on downstream performance: The evaluation focuses on KL divergence and agreement with the parent model. However, it does not clearly show whether better results on these metrics lead to better downstream performance. It is also unclear how performance drops as the budget decreases and when it becomes too poor to be useful. This makes it difficult to judge how well the method works under tight budgets.

Questions for authors
  1. Could you provide a practical example where the budget is known only after LoRA adapters have been trained? Why would waiting until the budget is known be impractical?
  2. At what budget p% does performance become poor, and what is the trade-off between budget and performance?
  3. How does your method compare to retraining LoRA with a rank that satisfies the budget, in terms of performance, computational cost, and wall-clock time?
  4. Could you report variability or uncertainty in Table 2 to help assess whether the differences are statistically significant, even though no training is involved?
1 comment

Thank you for your rebuttal and additional experiments.

However, I remain unconvinced of the practical benefit over directly training LoRA to fit the budget. At 12.5%, SIFT-LoRA loses 3.33 percentage points in accuracy, while at 25% it matches accuracy but requires substantially more computation. The latency results are interesting, but their limited scope does not sufficiently resolve my concerns about the overall trade-off. I therefore maintain my original score.

Reviewer_rSNxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SIFT-LoRA, a post-training method that compresses a trained Full-LoRA adapter into a layer-sparse adapter under a specified storage budget without additional training. Its Conditional SIFT-KL algorithm greedily selects layers based on their marginal reduction in parent–child predictive KL per serialized byte, dynamically updating layer importance after each selection. Experiments across three models and multiple tasks show that SIFT-LoRA generally outperforms static and contiguous-layer selection baselines in predictive fidelity and task performance while achieving competitive utility–latency trade-offs.

Critical strengths

The paper addresses a practical and meaningful problem in LoRA compression. The proposed method is simple and intuitive, and its ability to adapt a trained LoRA adapter to different deployment budgets without retraining is appealing. The experiments are relatively comprehensive and demonstrate promising improvements in predictive fidelity and deployment efficiency.

Critical weaknesses
  1. Mismatch Between KL Minimization and Task Performance: Minimizing parent–child predictive KL does not necessarily maximize downstream task performance. The paper's own results show that lower KL can lead to worse accuracy, particularly on GSM8K. This raises concerns about whether predictive KL is an appropriate objective for selecting important LoRA layers.

  2. Limited Evaluation on Recent Models:The evaluation is limited to relatively small models (2B–8B), which may not adequately represent the behavior of modern large language models. More experiments on larger, state-of-the-art architectures are needed, as layer importance and inter-layer dependencies may exhibit different patterns at scale.

Questions for authors

none

Reviewer_VGF9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets post-training adapter compilation: given a frozen same-task parent adapter, a calibration set, and an exact byte budget, choose a subset of whole layers whose LoRA factors are exported unchanged — no retraining or recompression. The claim is that which layers are worth keeping depends on which others are kept, so fixed importance rankings mis-rank. The proposed Conditional SIFT-KL is greedy forward selection scoring each candidate by marginal teacher-forced parent–child KL reduction per added serialized byte, under a "compiler contract" in which unselected tensors are physically absent (54/54 structural and reload-equivalence audits pass). An 18-condition grid shows Conditional beating two harmonized controls (Static-KL, Boundary-KL) on KL in 16/18 conditions; deployment campaigns compare against published post-hoc compression methods.

Critical strengths
  1. The byte-exact framing is a genuine contribution to how such work should be done. Budgeting serialized bytes of the shipped artifact (Eq. 2) rather than "number of layers," plus the physical-omission contract (Eq. 5), means reported size cannot be bookkeeping fiction; even a reader who rejects the method should adopt the measurement convention.
  2. Unusually careful internal validity where it is careful. The C.1 protocol fixes everything except frozen-vs-recomputed marginal scores, so Table 1 measures exactly the conditional-rescoring effect; the byte-matched subset and the random-support floor address the obvious confounds.
  3. Algorithm and §3.3 are executable (tie-breaking, infeasibility reports, evaluation counts, failure modes), and the candour is exceptional: §6 names the KL–utility divergence with a mechanism, disclaims the greedy's lack of guarantees, and Appendix I states precisely what is not released.
Critical weaknesses
  1. The headline result is measured with the method's own objective; the independent evidence is weaker and never tested for significance. Table 1's 16/18 is on KL — the quantity Conditional minimizes by construction, on an evaluation set drawn from the same pool as calibration. The construct-validity evidence (task utility) is 14/1/3 on n = 300 per condition with no per-condition CI, no paired test, and no aggregation test — in a paper that demonstrates in its own Chen replay that lower KL can coexist with worse exact match in all six cells.
  2. There is no external baseline in the controlled comparison, and both controls are constructed to favour the proposal. Static-KL (singleton scores frozen against the empty projection) is a deliberately naive variant no one proposed; Boundary-KL reaches only 75–90% of the byte cap, so part of the KL advantage over it is simply more capacity, with no byte-matched subset to correct it. Published methods appear only in the deployment campaign on different cohorts.
  3. The deployment framing overstates Table 17: SIFT-LoRA's frontier position is bought by being the fastest and the least faithful. It sits near the bottom of the utility column (ahead only of Chen et al.), with statistically significant deficits against four baselines each on Llama and Gemma; the abstract's "highest utility and lowest median ITL" for the fresh BoolQ comparison rests on +0.37 pp and 0.17 ms with paired p-values straddling zero that appear only in the appendix.
  4. The "reuse learned state" contribution reduces to a step-0 offset, and the "13.57×" cost claim omits the dominant term. Table 3 shows the entire AULC gain is the initialization G₀, with endpoints indistinguishable or worse for inheritance after continuation; and the retrofit comparison charges compilation (which performs no optimization) against training while omitting the 1,500-step continuation that Table 3 shows is still necessary — an honest accounting plausibly lands well below 13.57×, possibly below 1×. The 9B "consistent larger-model check" shows a ~19× shrinkage of the effect, which is evidence of it disappearing with scale.
Questions for authors
  1. Q1 the headline is measured with the method's own objective; the independent evidence is untested. Table 1's headline row of 16/18 is on KL — the quantity Conditional minimizes by construction, evaluated on a 300-example set drawn from the same validation pool as the 128-example calibration set. Please report per-condition paired uncertainty on task utility (bootstrap CI and exact McNemar, as Table 25 does for BoolQ) for all 18 conditions, plus an across-condition aggregation test; without it, the 14/1/3 utility count cannot be told from chance. Your own Chen replay already shows lower KL coexisting with worse GSM8K exact match in all six cells — that is a constructive counterexample to the objective's premise, not a scope note.
  2. Q2 no external baseline in the controlled comparison, and both controls are constructed to favour the proposal. Static-KL (singleton scores frozen against the empty projection) is a deliberately naive variant no one proposed; Boundary-KL reaches only 75–90% of the byte cap, so the 16/18 KL advantage over it is partly just more capacity, with no byte-matched subset to correct it. Please run one published post-hoc layer-selection method under the C.1 protocol (Chen et al.'s validation-utility lower-prefix rule is the cheapest, and would not need the "Boundary-KL is not a reproduction" caveat), and position the work explicitly against AdapterDrop and LayerDrop — both already omit adapter/layer computation at inference to trade latency for accuracy, and a one-line table entry does not establish what is new.
  3. Q3 the deployment framing overstates: the frontier position is bought by being the fastest and the least faithful. In Table 17, SIFT-LoRA's utility is near the bottom (ahead only of Chen et al.), with statistically significant deficits against four baselines each on Llama and Gemma (Table 21). Please state plainly in the main text that the operating point is the low-latency/low-fidelity corner, with the accuracy cost per unit of ITL saved next to the latency savings; and please substantiate the abstract's BoolQ "highest utility and lowest median ITL" claim — +0.37 pp and 0.17 ms with paired p-values straddling zero, reported only in the appendix.
  4. Q4 "reuse learned state" reduces to a step-0 offset, and "13.57×" omits the dominant term. Table 3 shows the entire AULC gain comes from the initialization G0G_0, with endpoints indistinguishable or worse for inheritance after continuation (25%: .9205 vs .9247 ± .0014). Please recompute the App. H comparison as compile + the 1,500-step continuation to endpoint versus train-from-scratch to endpoint — the 13.57× ratio currently charges compilation for an optimization it does not attempt while omitting the continuation Table 3 shows is still required; my expectation is that the honest number is well below 13.57×, possibly below 1×, and the paper should report it either way. Please also revise the "consistent larger-model check" framing for 9B: a ~19× shrinkage in ΔAULC (+1.32/+0.69 vs +25.17/+26.83 pp) is evidence of the effect disappearing with scale, not of consistency.
1 comment
Reviewer_VGF9OpenReview ↗

Response to author

I thank the authors for the thorough and candid response, which delivers the requested statistical protocol, an external published baseline, and an endpoint-matched cost comparison. However, I think the new evidence is still insufficient to support the paper's original claims: the aggregate utility advantage over the controls is no longer significant after multiplicity correction, the authors' own external comparison shows the selection objective to be task-dependent, and the inheritance benefit reduces to an initialization offset with the cost claim withdrawn. Since what remains defensible is the byte-exact compilation contract and a task-dependent utility–latency trade-off rather than the claimed mechanism-level advantage, I maintain my rating.

Comments on the paper

1 comment

As only a few days remain in the author-reviewer discussion period, I encourage reviewers to review the authors' rebuttal and actively participate in the discussion. In particular, please clarify which concerns have been addressed and identify any remaining concerns or questions that may require further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.