acceptodds
Under review as a conference paper at ICLR 2027

WAS-LoRA: Output-Sensitive Subspace Selection for Post-Training LoRA Compression

Abstract

We study post-training LoRA compression to reuse trained adapters under tighter rank budgets at deployment, without requiring an additional fine-tuning stage. Small reconstruction errors in adapter weights or the local LoRA branch outputs, however, do not necessarily ensure that the compressed model's final logits remain close to those of the original adapted model. We propose WAS-LoRA (Whitened Active Subspace Low-Rank Adaptation), which guides active-subspace selection using the sensitivity of a selected teacher output downstream of the adapter's bottleneck. It constructs a Jacobian Gram matrix that measures the output sensitivity in the adapter's whitened bottleneck coordinates. The leading eigenvectors of this Jacobian Gram matrix span the retained subspace, from which we directly construct standard lower-rank LoRA factors. We prove that, for a fixed retained rank, this subspace minimizes a Jacobian-based measure of output sensitivity along discarded bottleneck directions. Experiments on GLUE classification tasks across five different rank budgets demonstrate that WAS-LoRA reduces geometric-mean centered relative logit error by 30.1% and 15.8% compared with the activation-aware and fixed-basis baselines, respectively. Task-accuracy differences among methods are concentrated at the tightest rank budgets.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Low-Rank Adaptation (LoRA), Post-Training Compression, Active Subspaces, Sensitivity Analysis, Parameter-Efficient Fine-Tuning (PEFT)

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bnoQOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses the problem of compressing LoRA updates to maximize the output fidelity using only a few unlabeled calibration data. It does so by first whitening the bottleneck's activations using their uncentered second moment, and then computes the Gram matrix of the Jacobian of the final logits with respect to the whitened activations, and keeping the leading eigenvectors of this matrix as the retained components. At low rank budgets WAS-LoRA (the proposed method) keeps more accuracy with respect to other baselines and methods, while being robust to the calibration set size.

Critical strengths

The paper is very easy to follow, the standard notation helps the understanding as well as the figures, the algorithm and tables. It's generally well written and the narrative is self-contained. The idea is clear and correct and the experimental protocol is sound.

Critical weaknesses

In my opinion, while the experimental protocol is correct, it should be applied to models with a realistic practical gain. As shown in Table 8, data free SVD already matches WAS-LoRA at r=8r=8 on these tasks and models.

A graph showing how the cost of the method scales with the number of parameters or the output dimension is not reported.

Bigger models on more significant tasks should be tested and the required time and memory should be highlighted. When does it reach the break even point with training a new adapter?

From my understading theorem one is a reformulation of the Eckart-Young theorem. Can the authors clarify the novelty of this result?

Questions for authors

See weaknesses.

11 comments
AuthorsOpenReview ↗

Response: Practical gains ,the data-free SVD comparison,Theorem 1, Eckart–Young, and the methodological contribution

All manuscript references in this reply are to the revised manuscript.

Response to Weakness (Practical gains and the data-free SVD comparison)

Thank you for these questions about practical value, computational cost, and theoretical novelty. Following your suggestions, we have expanded the evaluations in the revised manuscript.

Our practical objective is to preserve the trained teacher's behavior under a constrained adapter budget. Similar task accuracy does not imply equivalent preservation of the teacher's logits or predictive probabilities. We therefore evaluate centered relative logit error and teacher-to-compressed KL alongside accuracy. At matched adapter budgets, WAS-LoRA reduces overall geometric-mean centered relative logit error by 47.7% relative to data-free Uniform-SVD (Section 4.1, Table 1). This demonstrates a measurable improvement in teacher-output preservation beyond what task accuracy alone captures.

Response to Question (Theorem 1 and Eckart–Young)

Our methodological contribution is to optimize the retained subspace itself in each trained LoRA module, rather than merely selecting or rescoring components of a prescribed local basis. Downstream sensitivity can therefore select combinations of local SVD directions. We agree that the optimization step in Theorem 1 follows the classical Ky Fan variational principle, closely related to Eckart–Young (Appendix A.1); we do not claim a new spectral theorem. From an active-subspace perspective, a conditional Poincaré bound for the conditional-expectation predictor motivates minimizing discarded downstream Jacobian energy in the whitened LoRA bottleneck. Theorem 1 solves this surrogate, and local least squares reconstructs the selected coordinates as standard lower-rank LoRA factors without recovery training (Sections 2–3, Proposition 2). The ridge-free analysis further establishes the local Act-SVD specialization, refactorization invariance on the supported subspace under matched spectral selections, and the IO-SVD correspondence under matched weighting (Corollary 4 and Propositions 5 and 3).

The practical value of selecting the subspace is supported by a controlled ablation. With per-module ranks, whitening maps, and downstream Grams fixed, WAS-LoRA reduces geometric-mean centered relative logit error by 3.5%–28.8% across all 30 task–budget–allocation settings against fixed-basis rescoring (Section 4.3, Table 2, and Appendix C.2.1). This isolates the benefit of adapting the retained directions.

AuthorsOpenReview ↗

Response: Compression cost versus output dimension and backbone size

Response to Weakness (Compression cost vs. output dimension and backbone size)

All manuscript references in this reply are to the revised manuscript.

Following your suggestion, we added time and GPU-memory curves for output dimension and backbone size. For output dimension, we profile Qwen2.5-1.5B-Instruct using four calibration batches, eight probes, and the full output head resident in memory. The timed interval covers sensitivity-statistics collection and CPU whitening/spectral processing. The output-width endpoints are:

Computed output dimension dd (logits per token)Median wall time (s)Incremental peak allocated GPU memory (MiB)
2562.649462.66
151,9362.700650.35

At full output width, increasing the probe count from 1 to 16 raises median time from 0.886 to 5.033 seconds (Appendix E.5.1, Figures 7–8).

For backbone size, we added measurements across four backbones using the same four calibration batches, 151,936 logits per token, and eight probes on one A800 per run in FP32. B4 denotes the four-batch pipeline comprising statistics collection, CPU spectral processing, rank allocation, and factor construction at a parameter budget corresponding to uniform rank four.

Backbone parametersB4 time (s), median [min, max]Peak allocated GPU memory (GiB)Incremental peak GPU memory (MiB)
1.54B3.726 [3.722, 3.742]6.47650
3.09B5.796 [5.705, 5.886]12.53931
7.62B5.556 [5.526, 5.575]29.831,177
14.77B12.499 [12.468, 12.523]56.971,717

Each configuration uses five timed repeats. Median statistics-collection time increases from 2.693 to 9.192 seconds across these endpoints. Total B4 time is not strictly monotonic: rank allocation takes 2.215 seconds at 3B versus 0.549 seconds at 7B. Adapter tasks and drivers differ across configurations, so these measurements characterize the tested workloads (Appendix E.5.2, Figure 9).

AuthorsOpenReview ↗

Response: Larger-model evidence, runtime, and memory

Response to Weakness (Larger-model evidence, runtime, and memory)

All manuscript references in this reply are to the revised manuscript.

Following your suggestion, we extended the task evaluation to Qwen2.5-7B-Instruct on GSM8K and ARC-Challenge. WAS-LoRA reduces both centered relative logit error and teacher-to-compressed KL in all 18 matched teacher–budget comparisons against Output-scored Act-SVD. The new experiment covers three independently trained rank-16 teachers per task, three common physical adapter-parameter budgets, and four compression methods, with all 72 compressed configurations evaluated on the full test sets (Appendix F.8, Table 33).

Across the 18 within-teacher pairs, the geometric-mean centered-error and KL ratios are 0.8323 and 0.7843, corresponding to reductions of 16.8% and 21.6%. The centered-error reduction is 3.0% on GSM8K and 28.6% on ARC-Challenge. The table below reports the per-budget fidelity ratios alongside greedy exact match; the task-score differences do not show uniform accuracy gains (Table 34).

TaskBudgetCentered-error ratio (W/O), geometric meanKL ratio (W/O), geometric meanWAS mean exact match (%)Output-scored mean exact match (%)
GSM8KB20.979330.9526274.5373.97
GSM8KB40.967860.9349373.7474.07
GSM8KB80.963790.9331574.1574.15
ARC-ChallengeB20.706090.6718989.4289.25
ARC-ChallengeB40.691170.6252489.1689.08
ARC-ChallengeB80.745660.6667689.0889.14

W/O denotes WAS-LoRA / Output-scored Act-SVD. B2/B4/B8 are physical parameter budgets equivalent to uniform ranks 2/4/8; WAS-LoRA and Output-scored Act-SVD have identical actual parameter counts within each pair. Each ratio is the geometric mean of three within-teacher ratios, while exact match is an arithmetic mean. Budgets are repeated within six distinct teachers.

We also measured physically compact, exported 7B adapters. Adapter files decrease from 161.53 MB for the rank-16 teachers to 20.26, 40.45, and 80.82 MB, corresponding to 87.5%, 75%, and 50% reductions in adapter parameters. For the fixed 128-input/32-output-token workload on one A800 in FP32, the compressed WAS configurations have per-configuration median batch-one generation times of 1.274–1.378 s (ten timed runs after three warm-ups) and absolute peak allocated GPU memory of approximately 28.59–28.65 GiB across the tested tasks, teachers, and budgets; Table 35 reports the teacher references and both batch sizes. Table 36 separately reports calibration-trajectory generation, sensitivity-statistics collection, and CPU construction. These task-level results complement the fixed-workload cost measurements up to 14.77B parameters in Appendix E.5.2: the larger-model fidelity benefit is now supported by full task evaluation, as well as measurements of deployable adapters.

Reviewer_HAjUOpenReview ↗
Rating
2(+1)
Confidence
4(+1)
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces WAS-LoRA, a post-training LoRA compression method using unlabeled calibration data without recovery fine-tuning. It selects subspaces in whitened adapter bottlenecks based on downstream output sensitivity and reconstructs lower-rank LoRA factors. The paper provides theoretical guarantees for its sensitivity surrogate and evaluates output fidelity and task performance on GLUE and two decoder-model benchmarks, with additional calibration and computational-cost analyses.

Critical strengths

A key strength is the consistent improvement in teacher-output fidelity without recovery fine-tuning. Across five GLUE tasks, WAS-LoRA reduces geometric-mean centered logit error by 30.1% relative to Act-SVD and 15.8% relative to Output-scored Act-SVD, with supporting results on a decoder model. This provides empirical evidence that its compression procedure preserves the trained model’s behavior beyond local reconstruction and fixed-basis sensitivity scoring.

Critical weaknesses

1. Limited methodological novelty. WAS-LoRA uses downstream Jacobian information to guide compression-subspace selection, which is closely related to the output-aware low-rank approximation underlying IO-SVD [1]. Their optimization formulations and resulting spectral solutions also exhibit a direct correspondence under matched settings.

For a LoRA branch, let s=Ahs=Ah, F=γBF=\gamma B, and Σ=E[ss⊤]\Sigma=\mathbb{E}[ss^{\top}]. Let DHD_H denote the Jacobian of the selected downstream output with respect to the branch output FsFs. By the chain rule, J~H=DHFΣ1/2\widetilde{J}_H=D_HF\Sigma^{1/2}, and therefore

PWAS⋆=Πk(Σ1/2F⊤E[DH⊤DH]FΣ1/2),P_{\mathrm{WAS}}^{\star}=\Pi_k\left(\Sigma^{1/2}F^{\top}\mathbb{E}[D_H^{\top}D_H]F\Sigma^{1/2}\right),

where Πk(M)\Pi_k(M) denotes a projector onto a leading kk-dimensional eigenspace of MM. The corresponding IO-SVD-style solution is

PIO⋆=Πk(R1/2W⊤CWR1/2).P_{\mathrm{IO}}^{\star}=\Pi_k\left(R^{1/2}W^{\top}CWR^{1/2}\right).

Consequently, when the IO-SVD-style objective is applied to the same bottleneck mapping with W=FW=F, R=ΣR=\Sigma, and C=E[DH⊤DH]C=\mathbb{E}[D_H^{\top}D_H], the two Gram matrices coincide. They then yield the same optimal projector set and, for matched selections, the same compressed update FΣ1/2P⋆Σ−1/2AF\Sigma^{1/2}P^{\star}\Sigma^{-1/2}A, consistent with WAS-LoRA’s Eq. (7).

These similarities in both formulation and solution suggest that the methodological contribution is primarily a compact specialization of output-aware weighted low-rank approximation to trained LoRA adapters.

Scope of this correspondence. The equivalence above concerns fixed per-module ranks, matched downstream endpoints and statistics, and ridge-free, invertible whitening maps; tied eigenspaces require matched selections. Original IO-SVD uses the KL metric C=E[DH⊤QDH]C=\mathbb{E}[D_H^{\top}QD_H], whereas the WAS formulation above uses a Euclidean metric, so the metrics must be aligned.

Although this correspondence requires specific assumptions, I still consider the underlying designs closely related, and this similarity remains central to my concern about methodological novelty.

2. Presentation and artifact-reference clarity. Appendices D and E refer to relative artifact paths, including data/per_task_metrics.csv and scripts/build_encoder_tables.py. Several pages, notably pages 24, 29, and 42, also contain substantial unused space. More compact pagination would improve readability and navigation.

[1] Ali Abbasi, Chayne Thrash, Haoran Qin, Hamed Pirsiavash, and Soheil Kolouri. “IO-SVD: Input-Output Whitened SVD for Adaptive-Rank LLM Compression.” arXiv:2605.15626, 2026.

Questions for authors

Could you clarify the theoretical relationship between WAS-LoRA and IO-SVD?

4 comments
AuthorsOpenReview ↗

Response to Weakness 1 and the Question: Methodological novelty and IO-SVD

Response to Weakness 1 and the Question (Methodological novelty and IO-SVD)

Thank you for the precise derivation. We formalized the fixed-rank correspondence in Proposition 3 and Appendix A.4 of the revised manuscript, and clarified the methodological relationship in revised Section 5.

  1. Connection to IO-SVD. IO-SVD derives its weighted low-rank objective from a local approximation to predictive KL. Applied to the LoRA output factor F=γBF=\gamma B, it yields the same compressed update as WAS-LoRA with ridge-free whitening and matched downstream endpoints, output metrics, calibration moments, per-module ranks, spectral selections, and reconstruction. The update also solves the matched two-sided weighted low-rank approximation of the full adapter γBA\gamma BA.

  2. LoRA-specific formulation and analysis. From an active-subspace perspective, a conditional Poincaré bound for the conditional-expectation predictor motivates minimizing downstream Jacobian energy along discarded directions in the whitened LoRA bottleneck (Section 2.2 of the original submission). We then reconstruct the local branch output from the retained coordinates by least squares, yielding standard lower-rank LoRA factors without recovery training (revised Proposition 2). We prove ridge-free refactorization invariance on the supported subspace with matched spectral selections (revised Proposition 5) and exact rank allocation for the additive discarded-sensitivity objective at equal direction costs (revised Proposition 6). IO-SVD scores eligible spectral tail deletions by their estimated first-order effect on calibration loss.

  3. Fidelity at matched compression budgets. With downstream Grams, whitening maps, and per-module ranks fixed, WAS-LoRA reduces geometric-mean centered relative logit error by 3.5%–28.8% across all 30 task–budget–allocation settings compared with fixed-basis rescoring, isolating the effect of direction selection (revised Section 4.3, Table 2). In a separate FP32 decoder evaluation with native compact adapters and identical actual parameter counts, WAS-LoRA reduces geometric-mean centered error by 12.2% compared with Output-scored Act-SVD, with lower centered error and KL divergence in all 18 paired configurations (revised Appendix F.6, paragraph introducing Table 29).

AuthorsOpenReview ↗

Response to Weakness 2: Presentation and artifact-reference clarity

Response to Weakness 2 (Presentation and artifact-reference clarity)

Following your feedback, we have consolidated related tables in Appendix D of the revised manuscript and grouped the decoder comparisons in revised Appendices F.4–F.5 to improve navigation. We have also clarified the verification and table-regeneration instructions in revised Appendix E.6. It now explicitly identifies the archive WAS_LoRA_code.zip and specifies the extracted project root, was_lora_iclr2027_code_light/, as the starting directory for relative paths. From this directory, running python run.py regenerate rebuilds the seven consolidated encoder tables from the saved results.

Thank you for clarifying the fixed-rank correspondence between WAS-LoRA and IO-SVD and for explaining how the two methods allocate rank across modules. I appreciate the LoRA-specific contribution and will raise my score to 2. Given the close relationship between their core subspace selection methods, I still find the methodological novelty limited and consider the paper slightly below the acceptance threshold.

Thank you for revisiting your assessment and for recognizing the LoRA-specific contribution. Prior work has shown the value of spectral methods for compressing trained LoRA adapters (Vulić et al., 2026 and Kumaravelu et al., 2026). WAS-LoRA formulates output-sensitive compression directly in the whitened LoRA bottleneck. It can retain linear combinations of local singular directions, reconstruct standard lower-rank LoRA factors without recovery training, and use downstream sensitivity to allocate ranks across modules under a shared budget. With per-module ranks, whitening, and the sensitivity matrix held fixed, selecting these combinations reduced centered logit error relative to rescoring the fixed local basis (Section 4.3, Table 2). The agreement with IO-SVD under matched ridge-free, fixed-rank conditions gives this LoRA construction a weighted low-rank interpretation (Proposition 3 and Appendix A.4).

The revised experiments quantify two practical gains:

  • Consistency with the teacher. At the 90% target coverage of the fixed GLUE confidence policy, WAS-LoRA reduced action disagreement with the teacher from 5.44% to 2.98% relative to Uniform-SVD, with similar accepted-answer error (Appendix G.3, Table 40).
  • Task accuracy. In the new GSM8K experiment at half the teacher’s adapter-parameter budget, WAS-LoRA answered 41 more of the 1,319 questions correctly than Uniform-SVD (+3.11 percentage points). Of those net gains, 39 were on questions the teacher originally answered correctly (Section 4.2 and Appendix I).

We would be grateful if you would reconsider your rating in light of these results.

Reviewer_QP94OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

WAS-LoRA compresses a trained adapter without recovery training by selecting leading eigendirections of a downstream Jacobian Gram matrix in whitened bottleneck coordinates. The main result is improved teacher-output fidelity on GLUE and two decoder tasks.

Critical strengths

Against Output-scored Act-SVD, Table 1 reports 15.8% lower geometric-mean centered relative logit error and 28.4% lower KL divergence. Aggregate fidelity also improves under rank and physical-parameter matching in the decoder study. These results support better preservation of the teacher's outputs.

Critical weaknesses
  1. The contribution of learning new directions is not isolated. Per-module ranks and whitening regularization can differ from Output-scored Act-SVD. The comparison therefore does not separate the benefit of new directions from allocation and regularization.

  2. Practical gains remain limited. Against Output-scored Act-SVD, the largest mean accuracy gain across budgets is 0.114 percentage points; four of five descriptive intervals include zero, and rank 8 favors the control. Decoder task gains are mixed, while the parameter-matched study reports only teacher-forced fidelity. Actual serving savings are not measured.

  3. The guarantees do not cover joint nonlinear compression. The results concern surrogate/local models, a different predictor, or ridge-free whitening. They do not bound the implemented joint compression error. The single-component deletion test also does not validate the surrogate when multiple directions are removed together.

Questions for authors
  1. How much of the gain over Output-scored Act-SVD remains with identical per-module ranks and regularization?
  2. On a representative task, does the surrogate predict joint-compression error at the tightest budget, including interventions that remove multiple directions?
5 comments
AuthorsOpenReview ↗

Response to Weakness 1 and Question 1 (Isolating direction selection)

Response to Weakness 1 and Question 1 (Isolating direction selection)

Thank you for suggesting that we isolate the effect of direction selection. We added a controlled experiment comparing WAS-LoRA directions with directions selected by rescoring a fixed local basis using the same downstream Gram (revised Section 4.3, Table 2, and Appendix C.2.1). Within each pair, we hold per-module ranks, ridge-regularized whitening maps, teachers, calibration inputs, downstream Gram matrices, and LoRA scaling fixed. We repeat the comparison under three frozen rank allocations.

Matched direction ablation (Table 2 of the revised manuscript)

Budget KKFrozen rank allocationSST-2QNLIMRPCRTEMNLI
48Uniform0.93040.95840.93290.83650.9313
48WAS-derived0.93930.90110.81650.79780.9422
48Output-scored-derived0.95200.90890.86940.84440.9510
96Uniform0.92920.94090.79110.76350.9290
96WAS-derived0.88440.89760.71170.78650.9504
96Output-scored-derived0.90370.93820.77190.83830.9652

Entries are geometric-mean centered relative logit-error ratios (WAS-LoRA / matched fixed basis) over eight teachers per task. Values below one favor WAS-LoRA.

WAS-LoRA directions reduce geometric-mean centered relative logit error by 3.5%–28.8% across all 30 task–budget–allocation settings at K=48K=48 and 9696. Across five tasks and eight independently trained teachers per task, WAS-LoRA gives lower centered logit error in 226/240 paired configurations and lower KL divergence in 214/240 (revised Appendix C.2.1, final paragraph).

AuthorsOpenReview ↗

Response to Weakness 2: Accuracy, decoder evaluation, and practical gains

Response to Weakness 2 (Accuracy, decoder evaluation, and practical gains)

We agree that the GLUE task-accuracy gains over Output-scored Act-SVD in the original submission were modest (original submission: Section 4.2, Figure 3, and Appendix D.3, Table 11). Output-scored Act-SVD is itself a downstream-aware compression variant constructed in this work : it applies downstream-sensitivity scoring to a fixed activation-aware SVD basis and serves as a strong fixed-basis control.Following your suggestions, we have expanded the evaluation in the revised manuscript in three ways:

We isolate the fidelity benefit of direction selection. The newly added matched-direction experiment shows that choosing the directions improves preservation of the fixed teacher's outputs at the same per-module ranks and whitening (revised Section 4.3, Table 2).

We evaluate generation accuracy and fidelity at matched physical parameter budgets. The original submission reported teacher-forced fidelity at matched physical parameter counts (original submission: Appendix F.5, Table 20). In the revised FP32 evaluation on Qwen2.5-1.5B-Instruct, we use three teachers per task on ARC-Challenge and GSM8K at budgets corresponding to uniform ranks 2, 4, and 8. WAS-LoRA and Output-scored Act-SVD use identical actual adapter parameter counts in every pair. WAS-LoRA achieves lower teacher-forced centered relative logit error and KL divergence in all 18 paired configurations, with a 12.2% geometric-mean centered-error reduction (revised Appendix F.6, paragraph introducing Table 29). Greedy-accuracy differences are mixed across tasks and budgets (revised Tables 29–30).

We measure adapter storage and fixed-workload inference costs. We report adapter-file size, peak allocated GPU memory, and generation latency for native unmerged adapters (revised Appendix F.7, Tables 31–32). Compared with uncompressed rank-16 teachers, WAS-LoRA uses 50%–87.5% fewer adapter parameters. On ARC-Challenge, mean adapter-file size falls from 73.91 MB to 9.31–37.01 MB. Across the tested A800/FP32 workloads (128 input tokens, 32 generated tokens, and batch sizes 1 and 4), mean generation latency is 0.9%–2.4% lower than for the uncompressed teachers.

AuthorsOpenReview ↗

Response to Weakness 3 and Question 2: Joint nonlinear compression

Response to Weakness 3 and Question 2 (Joint nonlinear compression)

The original submission tested single-direction deletions (original submission: Section 4.3 and Appendix C.1, Figure 5). Following your suggestion, we added a joint-compression experiment on QNLI (revised Section 4.3 and Appendix C.1.2, Table 3).

We distinguish the theoretical guarantees from the empirical validation of joint compression. Our optimality results concern the sensitivity surrogate and the stated local or ridge-free reconstruction objectives. The conditional-expectation bound applies to the reduced predictor. We evaluate finite error for the implemented, jointly compressed nonlinear network empirically.

The new experiment directly evaluates simultaneous compression of all 24 modules. Each candidate folds projected updates into LoRA factors using the frozen ridge-regularized maps. We then evaluate the full nonlinear network. We use three teachers and 128 prespecified held-out examples per teacher, disjoint from calibration. The primary budget retains K=48K=48 of the original 384 bottleneck dimensions, with K=96K=96 as a second check. The core candidates combine three frozen rank allocations with two direction rules, yielding 6, 4, and 6 distinct candidates for seeds 0, 1, and 2 at K=48K=48 (revised Appendix C.1.2, Protocol and candidate sets, and Table 3).

Downstream sensitivity yields lower selection regret across the reported core and neighboring groups. We compare calibration-side scores against the actual held-out mean squared raw-logit change. Selection regret is the percentage excess error of the selected candidate relative to the best candidate in its set. The table below shows the core results at K=48K=48 (first three rows of revised Table 3).

Proxy at K=48K=48Spearman correlation, seeds 0 / 1 / 2Selection regret, seeds 0 / 1 / 2
Downstream sensitivity0.943 / 0.800 / 0.6570% / 0% / 7.40%
Local activation reconstruction-0.886 / -0.800 / -0.65776.81% / 62.60% / 56.70%
Weight reconstruction-0.886 / -1.000 / -0.77176.81% / 62.60% / 56.70%

Across both budgets, downstream sensitivity gives higher Spearman correlation and lower selection regret than both reconstruction proxies in all six core teacher–budget groups. With prespecified small and medium perturbations to the WAS-LoRA subspaces, it also gives lower regret than both proxies in all twelve neighboring groups and higher correlation in eleven. For the remaining medium-perturbation group (seed 2, K=48K=48), weight reconstruction ranks candidates better, and downstream sensitivity selects a lower-error candidate (revised Appendix C.1.2, Core and neighboring comparisons). These results support the surrogate's use for ranking and selecting candidates within these joint-compression sets.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.