acceptodds
Under review as a conference paper at ICLR 2027

SVDoRA: Spectral Verification-Aware Direction-Oriented Rank Adaptation via Outcome-Guided Spectral Annealing

Abstract

SVD-based parameter-efficient fine-tuning selects spectral directions by correctness-agnostic scoring criteria and treats every selected direction as a generation update. We propose SVDoRA, a verification-aware adapter that uses gold-outcome labels to allocate capacity across a residual subspace and two spectral roles: task-specific generation directions (TSGD) and task-specific verification directions (TSVD). After a short residual-adapter warmup, a one-time calibration pass regenerates answers, labels them against gold outcomes, and selects the two roles by outcome-guided spectral annealing; the chosen directions then stay fixed for the rest of training. Compared with standard rank- LoRA, SVDoRA trains residual singular values and direction coefficients in addition to the two residual factors—24 extra scalars per target matrix at the canonical , setting. SVDoRA reaches 88.1% average accuracy on Qwen2.5-7B commonsense reasoning at rank 16 (+1.4 pp over LoRA-DASH) and 79.0% at 1.5B rank 16; it leads the Llama-3 GSM8KMATH average (48.4%), with competitive transfer to factual QA and code generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
parameter-efficient fine-tuning, LoRA, singular value decomposition, self-verification, large language models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sUz3OpenReview ↗
Rating
2
Confidence
3(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SVDoRA, an SVD-based PEFT method that uses outcome supervision to select spectral directions. After a warmup stage, generated calibration answers receive task-specific verification labels and are used to select task-specific generation directions and task-specific verification directions through simulated annealing. A later stage also reweights the training loss using a signal derived from these two direction sets.

The method is evaluated on commonsense reasoning, mathematical reasoning, code generation, and factual QA, and is compared with several SVD-based PEFT baselines. The paper also includes selector ablations and linear probes intended to test whether the selected verification directions contain correctness-related information.

Critical strengths
  1. Using outcome supervision to guide spectral direction selection is an interesting extension of existing SVD-based PEFT methods.

  2. The paper evaluates the idea across several task families and attempts to validate the proposed verification interpretation with probing experiments.

Critical weaknesses
  1. The method builds closely on existing SVD-based task-specific direction approaches such as LoRA-DASH and DisLoRA. Its main novelty is the use of outcome-supervised annealing to jointly select spectral directions with separate generation and verification roles. This is a meaningful extension, but the overall methodological novelty remains relatively incremental.

  2. The ablations do not clearly justify the complexity of the full pipeline. On Qwen2.5-1.5B commonsense reasoning, SASS reaches 79.32% compared with 78.88% for the reported correctness-agnostic selector. Moreover, replacing the Phase-3 GVSCR weighting with simpler alternatives changes the eight-task mean by at most 0.3 percentage points. These results suggest that the incremental benefit of some components of the full pipeline is modest.

  3. The evidence for a distinct verification role of TSVD remains limited. At the deployed probe dimension, TSVD achieves 0.635 accuracy compared with a 0.624 majority baseline, while its AUROC only ties the strongest null-space control. This supports the presence of correctness-related information but does not clearly establish a verification-specific mechanism. In addition, code-generation calibration uses normalized text similarity rather than execution outcomes, so the outcome supervision is not defined consistently across task families.

Questions for authors
  1. How much of the performance gain remains in a simplified variant that retains outcome-guided spectral direction selection but removes the separate TSGD/TSVD role design and uses uniform cross-entropy instead of GVSCR? This would help isolate which parts of the full pipeline are necessary.

  2. Can the authors provide stronger evidence that TSVD captures verification-specific information rather than features that are merely correlated with answer correctness? For example, would selectively removing or replacing TSVD directions affect correctness discrimination more than matched control directions?

5 comments
AuthorsOpenReview ↗

Response to Reviewer sUz3 (Part 1/3): TSVD Directional Interventions, Answer-Span Probes, and Entropy Fusion (W3 & Q2)

1. Evidence on TSVD: Directional Interventions (E2a), Answer-Span Probes (E2b), and Entropy Fusion (E2c) (W3 & Q2)
We thank Reviewer sUz3 for the constructive evaluation. SVDoRA does not claim an inference-time verifier, nor claims that the LoRA module possesses a ready-made self-verification capability, nor claims that the SVDoRA training paradigm achieves full self-improvement (§1, §3.6, Limitations); However, we do hypothesize that the TSVD directions contain certain signals that can differentiate the model answer from correct or wrong. In paper's Table 6 (N=1319N{=}1319, positive rate 0.6240.624), raw accuracy is compressed by class imbalance (weighted F1 is 0.6390.639 vs. 0.4790.479 majority), though single-token L21 AUROC (0.6720.672) tied NullSpace (0.6740.674). Following Q2, we ran inference-time directional interventions (E2a) on the aligned (E1-A) Qwen2.5-1.5B-Instruct checkpoints (r=16, k_g=k_v=4, uniform CE, seed 42) on MedQA (1,273 test / 1,272 dev) and StrategyQA (229 dev). Note E2 intervenes on E1-A (not E1-B/E1-C), testing reliance on the selected TSVD directions in E1-A. We also aligned the probe (E2b) with Phase-2 calibration via pre-hooks on q/k/v/o_proj across layers {7,14,21,27}\{7,14,21,27\}, mean-pooling module inputs over the answer span and weighting by active channel gain (StandardScaler + L2-LogisticRegression, C=0.05C{=}0.05, 5-fold CV).

Table R2a. Inference-time directional interventions (E2a) on the unmodified E1-A checkpoint (Qwen2.5-1.5B-Instruct, r=16r{=}16, kg=kv=4k_g{=}k_v{=}4, greedy decoding).

Intervention ConditionMedQA Test Acc (%)MedQA Δ\Delta AccMedQA TSVD Probe Drop (Multi-Layer / 4-Layer)StrategyQA Acc (%)StrategyQA Δ\Delta AccStrategyQA TSVD Probe Drop (Multi-Layer / 4-Layer)
Baseline (unmodified E1-A)49.02 (624/1273)—— (base 0.6470 / 0.5825)66.81 (153/229)—— (base 0.5626 / 0.5206)
Remove TSVD (βj→0\beta_j \to 0)46.43 (591/1273)-2.59,pp$-0.1455∗∗/∗∗0.1455** / **-$0.082564.63 (148/229)-2.18,pp−-0.0682 / -\0.0220
Reset TSVD (βj→σj\beta_j \to \sigma_j)48.94 (623/1273)-0.08,pp$-0.0736∗∗/∗∗0.0736** / **-$0.024366.38 (152/229)-0.44,pp−-0.0077 / -\0.0199
Remove TSGD (αi→0\alpha_i \to 0)47.05 (599/1273)-1.96,pp$-$0.0373 / $-$0.015663.32 (145/229)-3.49,pp$+$0.0690 / $+$0.0251
Reset TSGD (αi→σi\alpha_i \to \sigma_i)49.18 (626/1273)+0.16,pp$-$0.0014 / $-$0.001066.81 (153/229)+0.00,pp$+$0.0017 / $+$0.0003
Remove σ\sigma-matched frozen (mean of 3 draws)48.05 (611.7/1273 (603,614,618))-0.97,pp$-$0.0254 / $-$0.011266.23 (151.7/229 (153,155,147))-0.58,pp$-$0.0092 / $-$0.0061

Table R2b. Baseline correctness probing (E2b) and incremental AUROC over output token entropy (E2c) across layers {7,14,21,27}\{7,14,21,27\}.

SubspaceMedQA Multi-Layer AUROC (BAcc)MedQA Pure Norm AUROCMedQA 4-Layer Mean AUROCMedQA E2c Δ\DeltaAUROC over Entropy (QDA / 2D-Logit / GNB)StrategyQA Multi-Layer AUROC (BAcc)StrategyQA 4-Layer Mean AUROCStrategyQA E2c Δ\DeltaAUROC (2D-Logit)
TSVD0.6470 (0.6118)0.61720.5825+0.0226 / +0.0048 / +0.06850.5626 (0.5443)0.5206+0.0047
TSGD0.6089 (0.5808)0.55530.5648-0.0158 / -0.0095 / -0.07290.5054 (0.4976)0.4862-0.0108
NullSpace0.5955 (0.5723)0.58100.5528-0.0256 / -0.0159 / -0.01430.5776 (0.5368)0.5195+0.0146
Random0.6210 (0.5829)0.57960.5698-0.0133 / -0.0073 / -0.04350.5115 (0.5066)0.4995-0.0078

Key Takeaways (Q2): (i) MedQA: TSVD attains the highest multi-layer AUROC (0.64700.6470, BAcc 0.61180.6118), pure norm AUROC (0.61720.6172, +3.62 pp+3.62\text{ pp} over NullSpace), and 4-layer mean AUROC (0.58250.5825). Zeroing (βj→0\beta_j{\to}0) or resetting (βj→σj\beta_j{\to}\sigma_j) TSVD drops multi-layer AUROC by −0.1455-0.1455 and −0.0736-0.0736 (vs. ≤0.0373\le 0.0373 for controls). In E2c, fusing TSVD with token entropy yields positive AUROC increments across all readouts (+2.26 pp+2.26\text{ pp} QDA, +0.48 pp+0.48\text{ pp} 2D-logit, +6.85 pp+6.85\text{ pp} GNB), whereas TSGD, NullSpace, and Random are negative. Zeroing TSVD also causes the largest MedQA test accuracy drop (−2.59 pp-2.59\text{ pp}, vs. −1.96 pp-1.96\text{ pp} for TSGD zeroing and −0.97 pp-0.97\text{ pp} for frozen removal; βj→σj\beta_j{\to}\sigma_j is −0.08 pp-0.08\text{ pp}). (ii) StrategyQA (229 items): Results are mixed—TSVD leads on 4-layer mean AUROC (0.52060.5206), multi-layer BAcc (0.54430.5443), and βj→0\beta_j{\to}0 probe drop (−0.0682-0.0682), and beats Random/TSGD on E2c 2D-logit (+0.47 pp+0.47\text{ pp} vs. −0.78 pp-0.78\text{ pp} / −1.08 pp-1.08\text{ pp}), whereas NullSpace has higher multi-layer AUROC (0.57760.5776) and E2c 2D-logit (+1.46 pp+1.46\text{ pp}), and TSGD zeroing drops accuracy more (−3.49 pp-3.49\text{ pp} vs. −2.18 pp-2.18\text{ pp}).

AuthorsOpenReview ↗

Response to Reviewer sUz3 (Part 2/3): Selector Ablation and Simplified Variant Without Separate TSGD/TSVD Roles (W2 & Q1)

2. Selector Ablation: SASS vs. VUS and Controlled Outcome-Label Ablation (E1) (W2)
In Table A3 (tab:b10-selection), Random selection (76.97%76.97\%) is the information floor under the same hard-partitioned parameterization (Prop. 5(iii)), over which SASS-v2 (79.32±0.13%79.32\pm0.13\%) gains +2.35 pp+2.35\text{ pp}, while VUS (78.88%78.88\%, single seed) is our own second-order Fisher selector (Eq. A4) that already beats random by +1.91 pp+1.91\text{ pp}. Because the VUS and SASS-v1 (78.83%78.83\%) rows in Table A3 are single-seed and SASS-v1 also differs in annealing energy form and singular direction shortlist construction, we ran a controlled label ablation (E1) within the identical SASS-v2 pipeline, which this version of SASS is same to the main table in the paper (Qwen2.5-1.5B-Instruct, r=16,kg=kv=4r{=}16, k_g{=}k_v{=}4, hard partition, uniform CE), comparing aligned labels (E1-A), shuffled labels (E1-B), and removed labels (E1-C, w1=w1b=0w_1{=}w_{1b}{=}0):

Table R1. Controlled correctness-label ablation (E1) within the identical SASS-v2 selection pipeline (Qwen2.5-1.5B-Instruct, r=16r{=}16, kg=kv=4k_g{=}k_v{=}4, hard partition, uniform CE).

Selection ArmMedQA Acc (%, 2 seeds)MedQA Cal. dverd_{\text{ver}}StrategyQA Acc (%)StrategyQA Cal. dverd_{\text{ver}}GSM8K Acc (%)GSM8K Cal. dverd_{\text{ver}}
E1-A: Aligned correctness labels (w1=1.0,w1b=0.5w_1{=}1.0, w_{1b}{=}0.5)48.47$\pm$0.55 (49.02/47.92)0.55866.81 (153/229)2.21144.35 (585/1319)0.641
E1-B: Shuffled correctness labels (w1=1.0,w1b=0.5w_1{=}1.0, w_{1b}{=}0.5)48.74$\pm$0.35 (49.10/48.39)0.33366.81 (153/229)0.35246.70 (616/1319)0.419
E1-C: Labels removed (w1=0,w1b=0w_1{=}0, w_{1b}{=}0)47.80$\pm$0.04 (47.76/47.84)0.20167.25 (154/229)0.57943.37 (572/1319)0.142

What Table R1 Shows: (1) Comparing E1-A against E1-C (labels removed), outcome-label supervision is a clear source of performance gain: it drives much higher TSVD calibration separation dverd_{\text{ver}} across all three tasks (0.5580.558 vs. 0.2010.201 on MedQA; 2.2112.211 vs. 0.5790.579 on StrategyQA; 0.6410.641 vs. 0.1420.142 on GSM8K) and improves test accuracy by +0.67 pp+0.67\text{ pp} on MedQA (48.47±0.55%48.47\pm0.55\% vs. 47.80±0.04%47.80\pm0.04\%, consistent across both seeds: +1.26 pp+1.26\text{ pp} on s42, +0.08 pp+0.08\text{ pp} on s123) and +0.98 pp+0.98\text{ pp} on GSM8K (44.35%44.35\% vs. 43.37%43.37\%). (2) Comparing E1-A against E1-B (shuffled labels), aligned labels are essential to achieve high correctness separation dverd_{\text{ver}} (0.5580.558 vs. 0.3330.333; 2.2112.211 vs. 0.3520.352; 0.6410.641 vs. 0.4190.419), whereas on downstream test accuracy under uniform CE the difference between aligned and shuffled order is less pronounced (48.47%48.47\% vs. 48.74%48.74\% on MedQA, 66.81%66.81\% vs. 66.81%66.81\% on StrategyQA, 44.35%44.35\% vs. 46.70%46.70\% on GSM8K), as keeping the non-zero separation terms (w1,w1b>0w_1, w_{1b}{>}0) active in the annealing energy already regularizes direction selection relative to dropping them in E1-C.

3. Simplified Variant Without Separate TSGD/TSVD Roles and With Uniform Cross-Entropy (E3) (Q1 & W2)
In Q1, you asked how much gain remains in "a simplified variant that retains outcome-guided spectral direction selection but removes the separate TSGD/TSVD role design and uses uniform cross-entropy instead of GVSCR." On 8-task commonsense, disabling GVSCR (Off, Table A4) achieves 79.39±0.21%79.39\pm0.21\% (+2.42 pp+2.42\text{ pp} over the 76.97%76.97\% random floor), and setting w2=0w_2{=}0 (Table A5) changes the 8-task mean by −0.13 pp-0.13\text{ pp} (79.06%79.06\%, single seed). On MedQA (Qwen2.5-1.5B-Instruct, seed 42, uniform CE), we trained this exact simplified variant (E3) with kg=0,kv=8,w2=0k_g{=}0, k_v{=}8, w_2{=}0 so all 8 directions are selected purely by the outcome-verified objective without a separate TSGD role:

Table R3. Simplified variant without separate TSGD/TSVD roles and with uniform CE (E3) vs. two-role anchor (E1-A) on MedQA (Qwen2.5-1.5B-Instruct, r=16r{=}16, hard partition, seed 42, greedy decoding).

VariantDirection BudgetEnergy Weights (w1,w1b,w2)(w_1, w_{1b}, w_2)Phase-3 LossCal. dverd_{\text{ver}}MedQA Test Acc (%)Δ\Delta vs. Two-Role
Two-role SVDoRA with uniform CE (E1-A anchor)kg=4,  kv=4k_g{=}4,\;k_v{=}4(1.0,  0.5,  0.3)(1.0,\;0.5,\;0.3)Off (wi=1w_i{=}1)0.532649.02 (624/1273)—
Simplified variant without separate TSGD/TSVD roles (E3)kg=0,  kv=8k_g{=}0,\;k_v{=}8(1.0,  0.5,  0.0)(1.0,\;0.5,\;0.0)Off (wi=1w_i{=}1)0.579448.47 (617/1273)-0.55,pp (-7 items)

In this single-seed comparison, E3 reaches 48.47%48.47\% (617/1273617/1273) vs. 49.02%49.02\% (624/1273624/1273, a −0.55 pp-0.55\text{ pp} point-estimate difference) for the two-role anchor (E1-A), showing that outcome-guided selection with uniform CE and a single role retains nearly all MedQA accuracy while the two-role split adds +0.55 pp+0.55\text{ pp}.

AuthorsOpenReview ↗

Response to Reviewer sUz3 (Part 3/3): GVSCR Design Motivation, Hyperparameter Sweep, and Literature Comparison (W2)

(Note: For a quick summary of all experiment codes E1-A/B/C, E2a/b/c, E3, and E7/E8, see Table R0 in General Comment Part 1.)

4. GVSCR Design Motivation, Hyperparameter Sweep (E7), and Literature Comparison (E8) (W2)
We agree that on commonsense (Table A4), GVSCR does not improve accuracy over Off. Its architectural motivation was not to act as a standalone accuracy booster, but to close the loop between Phase-2 selection and Phase-3 optimization: without GVSCR, TSVD directions extracted at TwT_w are updated only passively under uniform CE, never invoking their correctness-correlated readout during training. GVSCR was designed as a step-to-step training loop that consumes the TSVD readout signal at step tt to modulate updates so step t+1t{+}1 builds on weights that absorbed prior feedback (without claiming inference-time self-verification). To evaluate Phase-3 weighting on MedQA and zero-shot MMLU-Pro (Qwen2.5-1.5B-Instruct, seed 42), we swept 21 GVSCR configs on MedQA dev (E7) and compared against 4 literature methods (E8: Focal, GCE, SPL [Jiang et al., ACM MM 2014], and JTT, each tuned over 3 dev configs under matched warmup/normalization):

Table R6. Phase-3 loss reweighting comparison on MedQA and zero-shot MMLU-Pro (Qwen2.5-1.5B-Instruct, r=16r{=}16, kg=kv=4k_g{=}k_v{=}4; test columns report mean$\pm$std across 3 nucleus evaluations at T=0.2,p=0.95T{=}0.2, p{=}0.95, decoding RNG seeds 42/123/7 on a single trained checkpoint).

Phase-3 Weighting MethodDev ConfigsSelected Hyperparameters (on MedQA dev)MedQA Dev (Greedy)MedQA Test (%)MMLU-Pro Overall (%)MMLU-Pro Health (%)MMLU-Pro Biology (%)
Uniform CE (Off)1wi=1w_i = 1 (E1-A)46.54 (592/1272)48.94$\pm$0.3326.53$\pm$0.1233.54$\pm$0.1548.77$\pm$0.35
Legacy (TSVD-norm)1λconf=0.3\lambda_{\text{conf}} = 0.347.09 (599/1272)47.71$\pm$0.2125.16$\pm$0.0929.71$\pm$0.6547.05$\pm$0.29
Focal Loss3γ=0.5\gamma = 0.5 (from {0.5,1,2}\{0.5, 1, 2\})47.09 (599/1272)49.12$\pm$0.1326.22$\pm$0.0329.46$\pm$0.6251.28$\pm$0.47
Generalized CE (GCE)3q=0.5q = 0.5 (from {0.3,0.5,0.7}\{0.3, 0.5, 0.7\})47.88 (609/1272)46.24$\pm$0.2625.77$\pm$0.0531.30$\pm$0.7047.88$\pm$0.29
Self-Paced Learning (SPL)3qstart=0.5q_{\text{start}} = 0.5 (from {0.5,0.7,0.9}\{0.5, 0.7, 0.9\})48.66 (619/1272)47.58$\pm$0.2727.12$\pm$0.0132.36$\pm$0.1553.09$\pm$0.13
Just Train Twice (JTT)3λup=5.0\lambda_{\text{up}} = 5.0 (from {2,5,20}\{2, 5, 20\})46.70 (594/1272)46.19$\pm$0.6125.52$\pm$0.1130.52$\pm$0.3048.58$\pm$0.07
GVSCR (Paper Default)1λconf=0.5,γ=2.0,m=0.9\lambda_{\text{conf}}{=}0.5, \gamma{=}2.0, m{=}0.9, mid_last47.01 (598/1272)48.65$\pm$0.2626.40$\pm$0.1131.70$\pm$0.3050.49$\pm$0.41
GVSCR (Dev-Tuned)21Tλ=200T_{\lambda}{=}200 (gvscr_lambda_warmup_steps=200)47.56 (605/1272)49.86$\pm$0.3227.22$\pm$0.1232.11$\pm$0.1248.68$\pm$0.39

Findings: On MedQA test (E8), GVSCR (Dev-Tuned) (Tλ=200T_{\lambda}{=}200, dev 47.56%47.56\%) reaches 49.86±0.32%49.86\pm0.32\%, beating Uniform CE (48.94±0.33%48.94\pm0.33\%, +0.92 pp+0.92\text{ pp} vs. 0.65 pp0.65\text{ pp} std sum), GVSCR Default (48.65±0.26%48.65\pm0.26\%, +1.21 pp+1.21\text{ pp}), and all four literature methods (closest is Focal at 49.12±0.13%49.12\pm0.13\%, +0.74 pp+0.74\text{ pp} vs. 0.45 pp0.45\text{ pp} std sum; note the 21-vs-3 dev config asymmetry). On MMLU-Pro, it reaches 27.22±0.12%27.22\pm0.12\% (SPL 27.12±0.01%27.12\pm0.01\%, Uniform CE 26.53±0.12%26.53\pm0.12\%). On 1.5B math (E4), λconf=0\lambda_{\text{conf}}{=}0 lowered GSM8K$+MATHby∗∗MATH by **-1.45\text{ pp}$** (38.96%38.96\% vs. 40.42%40.42\%).

Reviewer_gsJnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SVDoRA uses answer-correctness labels to select spectral directions for parameter-efficient fine-tuning. After a residual-adapter warmup, a one-time calibration pass selects separate generation-aligned and correctness-discriminative directions, whose coefficients are then trained alongside the residual adapter. The paper evaluates the method on commonsense reasoning, mathematics, code generation, and factual QA, and probes whether the selected verification directions contain information about answer correctness.

Critical strengths
  1. Using correctness labels to choose adaptation directions is a useful extension of spectral PEFT. The calibration step ties direction selection to observed task outcomes, and the selected directions remain fixed afterward. This provides a concrete way to use outcome supervision without adding an inference-time verifier.

  2. The commonsense results are encouraging. On Qwen2.5-7B at rank 16, SVDoRA reaches 88.1 ± 0.4%, compared with 86.7 ± 0.1% for the authors’ LoRA-DASH runs; it also has higher averages at ranks 8 and 32 (Table 1).

Critical weaknesses
  1. The added value of correctness supervision is not yet isolated convincingly. Section 3.7 reports 79.32% for outcome-guided selection versus 78.88% for the correctness-agnostic VUS selector, but this changes the selection procedure as well as its signal. The probe results are also modest: at the deployed dimension, accuracy is 0.635 against a majority baseline of 0.624, and AUROC ties the best control (Section 3.6). These results suggest that correctness information is present, but do not establish how much it contributes to the generation gains. A control that removes or shuffles correctness labels within the same selection pipeline would be more decisive.

  2. The advantage over the closest spectral baseline is uneven. The Qwen2.5-7B commonsense comparison with DisLoRA uses results from another paper, while the authors’ math experiments favor DisLoRA on the three-task average for both backbones: 54.2 versus 52.1 on Qwen and 39.9 versus 38.4 on Llama (Table 2).

Questions for authors
  1. With the residual initialization, direction budget, annealing procedure, and training loss fixed, how much performance is lost when correctness labels are removed or shuffled? Does that difference persist across seeds?

  2. Does the Qwen2.5-7B commonsense advantage over DisLoRA remain when both methods are rerun with the same training and evaluation recipe?

3 comments
AuthorsOpenReview ↗

Response to Reviewer gsJn (Part 1/3): Controlled Correctness-Label Shuffling and Removal Ablation (W1 & Q1)

We thank Reviewer gsJn for the insightful review and for highlighting that tying direction selection to observed task outcomes without adding an inference-time verifier is a useful extension, with encouraging Qwen2.5-7B commonsense results. Below we address your concerns across three comments: isolating correctness labels within the same selection pipeline (E1, Part 1), directional interventions and answer-span probing on TSVD (E2, Part 2), and rerunning DisLoRA in our Qwen2.5-7B pipeline plus the math comparison (E5, Part 3).

1. Controlled Correctness-Label Ablation: Aligned (E1-A) vs. Shuffled (E1-B) vs. Removed Labels (E1-C) (W1 & Q1)
We agree with W1: comparing SASS-v2 (79.32%79.32\%) against deterministic top-kk VUS (78.88%78.88\%, single seed; random floor 76.97%76.97\%) in §3.7 changes both the search procedure and the scoring signal (Appendix E), and SASS-v1 (78.83%78.83\%, single seed) in Table A3 (tab:b10-selection) also differs from SASS-v2 in energy terms and shortlist construction.

To provide the direct control requested in W1 and Q1 ("With the residual initialization, direction budget, annealing procedure, and training loss fixed, how much performance is lost when correctness labels are removed or shuffled? Does that difference persist across seeds?"), we ran E1 on Qwen2.5-1.5B-Instruct (r=16,α=32r{=}16, \alpha{=}32, q/k/v/o_proj, kg=kv=4k_g{=}k_v{=}4, hard partition, uniform CE wi=1w_i{=}1, 3 epochs, batch 18, lr 3×10−43{\times}10^{-4}). Holding warmup, shortlist, annealing (4 restarts ×\times 2000 steps), and Phase-3 loss fixed, we compared three arms across three benchmarks (MedQA with 2 independent training seeds 42 and 123; StrategyQA and GSM8K with seed 42 and 3 deterministic greedy evaluation passes):

  • E1-A (Aligned labels): Default SASS-v2 (w1=1.0,w1b=0.5,w2=0.3w_1{=}1.0, w_{1b}{=}0.5, w_2{=}0.3) pairing each calibration answer's gold correctness label with its own projection.
  • E1-B (Shuffled labels): Identical weights (w1=1.0,w1b=0.5,w2=0.3w_1{=}1.0, w_{1b}{=}0.5, w_2{=}0.3), with correctness labels randomly permuted relative to calibration projections.
  • E1-C (Labels removed): Identical pipeline with outcome-label weights zeroed (w1=0,w1b=0,w2=0.3w_1{=}0, w_{1b}{=}0, w_2{=}0.3).

Table R1. Controlled correctness-label ablation (E1) within the identical SASS-v2 selection pipeline (Qwen2.5-1.5B-Instruct, r=16r{=}16, kg=kv=4k_g{=}k_v{=}4, hard partition, uniform CE).

Selection ArmMedQA Acc (%, 2 seeds)MedQA Cal. dverd_{\text{ver}}StrategyQA Acc (%)StrategyQA Cal. dverd_{\text{ver}}GSM8K Acc (%)GSM8K Cal. dverd_{\text{ver}}
E1-A: Aligned correctness labels (w1=1.0,w1b=0.5w_1{=}1.0, w_{1b}{=}0.5)48.47$\pm$0.55 (49.02/47.92)0.55866.81 (153/229)2.21144.35 (585/1319)0.641
E1-B: Shuffled correctness labels (w1=1.0,w1b=0.5w_1{=}1.0, w_{1b}{=}0.5)48.74$\pm$0.35 (49.10/48.39)0.33366.81 (153/229)0.35246.70 (616/1319)0.419
E1-C: Labels removed (w1=0,w1b=0w_1{=}0, w_{1b}{=}0)47.80$\pm$0.04 (47.76/47.84)0.20167.25 (154/229)0.57943.37 (572/1319)0.142

What Table R1 Shows (Q1):

  1. Outcome-label supervision is a source of performance gain (E1-A vs. E1-C): Across all three benchmarks and both MedQA training seeds, aligned labels (E1-A) drive substantially higher correct/incorrect separation dverd_{\text{ver}} in TSVD (0.5580.558 on MedQA [s42: 0.5330.533, s123: 0.5830.583], 2.2112.211 on StrategyQA, 0.6410.641 on GSM8K) than removed labels (E1-C: 0.201,0.579,0.1420.201, 0.579, 0.142). On downstream test accuracy, removing correctness labels (E1-C) loses −0.67 pp-0.67\text{ pp} on MedQA (48.47±0.55%48.47\pm0.55\% vs. 47.80±0.04%47.80\pm0.04\%, consistent in direction across both seeds: +1.26 pp+1.26\text{ pp} on s42 [624624 vs. 608/1273608/1273], +0.08 pp+0.08\text{ pp} on s123 [610610 vs. 609/1273609/1273]) and −0.98 pp-0.98\text{ pp} on GSM8K (44.35%44.35\% vs. 43.37%43.37\%), while remaining within 11 item on the 229-item StrategyQA set (153153 vs. 154/229154/229).
  2. Aligned vs. shuffled label order (E1-A vs. E1-B): Aligned labels (E1-A) achieve much higher calibration separation dverd_{\text{ver}} than shuffled labels (E1-B) on every task (0.5580.558 vs. 0.3330.333 on MedQA, 2.2112.211 vs. 0.3520.352 on StrategyQA, 0.6410.641 vs. 0.4190.419 on GSM8K), confirming that aligned outcome labels are required to isolate correctness-discriminative TSVD directions. On downstream generation accuracy under uniform CE, the difference between aligned and shuffled order is less pronounced (48.47%48.47\% vs. 48.74%48.74\% on MedQA, 66.81%66.81\% vs. 66.81%66.81\% on StrategyQA, 44.35%44.35\% vs. 46.70%46.70\% on GSM8K), because keeping the non-zero separation terms (w1,w1b>0w_1, w_{1b}{>}0) active in the annealing energy already regularizes direction selection relative to dropping them in E1-C.
AuthorsOpenReview ↗

Response to Reviewer gsJn (Part 2/3): TSVD Directional Interventions, Answer-Span Probes, and Entropy Fusion (W1)

2. Directional Interventions (E2a), Answer-Span Probes (E2b), and Entropy Fusion (E2c) on E1-A (W1)
In W1, you noted that in §3.6 (Table 6, N=1319N{=}1319, positive rate 0.6240.624), single-token L21 probe accuracy (0.6350.635 vs. 0.6240.624 majority; weighted F1 0.6390.639 vs. 0.4790.479) and AUROC (0.6720.672 vs. 0.6740.674 NullSpace) are modest. To examine the TSVD directions in the aligned (E1-A) models more closely, we ran inference-time directional interventions (E2a), answer-span probes (E2b), and entropy-fusion tests (E2c) on the unmodified MedQA (A-medqa-s42) and StrategyQA (A-strategyqa-s42) checkpoints. Note that E2 intervenes on the aligned E1-A checkpoints (not E1-B/E1-C), evaluating how much the trained model and probe rely on the selected TSVD directions in E1-A. We attached forward pre-hooks on q/k/v/o_proj across layers {7,14,21,27}\{7,14,21,27\}, mean-pooling module inputs over the generated answer span (matching Phase-2 calibration) and weighting by active channel gain (StandardScaler + L2-LogisticRegression, C=0.05C{=}0.05, 5-fold stratified CV):

Table R2a. Inference-time directional interventions (E2a) on the unmodified E1-A checkpoint (Qwen2.5-1.5B-Instruct, r=16r{=}16, kg=kv=4k_g{=}k_v{=}4, greedy decoding).

Intervention ConditionMedQA Test Acc (%)MedQA Δ\Delta AccMedQA TSVD Probe Drop (Multi-Layer / 4-Layer)StrategyQA Acc (%)StrategyQA Δ\Delta AccStrategyQA TSVD Probe Drop (Multi-Layer / 4-Layer)
Baseline (unmodified E1-A)49.02 (624/1273)—— (base 0.6470 / 0.5825)66.81 (153/229)—— (base 0.5626 / 0.5206)
Remove TSVD (βj→0\beta_j \to 0)46.43 (591/1273)-2.59,pp$-0.1455∗∗/∗∗0.1455** / **-$0.082564.63 (148/229)-2.18,pp−-0.0682 / -\0.0220
Reset TSVD (βj→σj\beta_j \to \sigma_j)48.94 (623/1273)-0.08,pp$-0.0736∗∗/∗∗0.0736** / **-$0.024366.38 (152/229)-0.44,pp−-0.0077 / -\0.0199
Remove TSGD (αi→0\alpha_i \to 0)47.05 (599/1273)-1.96,pp$-$0.0373 / $-$0.015663.32 (145/229)-3.49,pp$+$0.0690 / $+$0.0251
Reset TSGD (αi→σi\alpha_i \to \sigma_i)49.18 (626/1273)+0.16,pp$-$0.0014 / $-$0.001066.81 (153/229)+0.00,pp$+$0.0017 / $+$0.0003
Remove σ\sigma-matched frozen (mean of 3 draws)48.05 (611.7/1273 (603,614,618))-0.97,pp$-$0.0254 / $-$0.011266.23 (151.7/229 (153,155,147))-0.58,pp$-$0.0092 / $-$0.0061

Table R2b. Baseline correctness probing (E2b) and incremental AUROC over output token entropy (E2c) across layers {7,14,21,27}\{7,14,21,27\}.

SubspaceMedQA Multi-Layer AUROC (BAcc)MedQA Pure Norm AUROCMedQA 4-Layer Mean AUROCMedQA E2c Δ\DeltaAUROC over Entropy (QDA / 2D-Logit / GNB)StrategyQA Multi-Layer AUROC (BAcc)StrategyQA 4-Layer Mean AUROCStrategyQA E2c Δ\DeltaAUROC (2D-Logit)
TSVD0.6470 (0.6118)0.61720.5825+0.0226 / +0.0048 / +0.06850.5626 (0.5443)0.5206+0.0047
TSGD0.6089 (0.5808)0.55530.5648-0.0158 / -0.0095 / -0.07290.5054 (0.4976)0.4862-0.0108
NullSpace0.5955 (0.5723)0.58100.5528-0.0256 / -0.0159 / -0.01430.5776 (0.5368)0.5195+0.0146
Random0.6210 (0.5829)0.57960.5698-0.0133 / -0.0073 / -0.04350.5115 (0.5066)0.4995-0.0078

Key Takeaways (W1): (i) MedQA: Zeroing the 4 TSVD scalars (βj→0\beta_j{\to}0) drops MedQA test accuracy by −2.59 pp-2.59\text{ pp} (−33-33 items), exceeding TSGD zeroing (−1.96 pp-1.96\text{ pp}, −25-25 items) and σ\sigma-matched frozen removal (−0.97 pp-0.97\text{ pp} mean; resetting βj→σj\beta_j{\to}\sigma_j is −0.08 pp-0.08\text{ pp}). With answer-span module-input pooling (E2b), TSVD attains the highest MedQA multi-layer AUROC (0.64700.6470, BAcc 0.61180.6118) and pure norm AUROC (0.61720.6172, +3.62 pp+3.62\text{ pp} over NullSpace 0.58100.5810), and zeroing TSVD drops multi-layer AUROC by −0.1455-0.1455 (vs. ≤0.0373\le 0.0373 for controls). Fusing TSVD with output token entropy (E2c) yields positive AUROC increments across all three readouts (+2.26 pp+2.26\text{ pp} QDA, +0.48 pp+0.48\text{ pp} 2D-logit, +6.85 pp+6.85\text{ pp} GNB), whereas TSGD, NullSpace, and Random are all negative. (ii) StrategyQA (229 items): TSVD leads on 4-layer mean AUROC (0.52060.5206), multi-layer BAcc (0.54430.5443), and βj→0\beta_j{\to}0 probe drop (−0.0682-0.0682), and gains +0.47 pp+0.47\text{ pp} on E2c 2D-logit (vs. Random −0.78 pp-0.78\text{ pp}, NullSpace +1.46 pp+1.46\text{ pp}).

AuthorsOpenReview ↗

Response to Reviewer gsJn (Part 3/3): Qwen2.5-7B DisLoRA Rerun in Our Pipeline and Math Comparison (W2 & Q2)

(Note: For a quick reference of all supplementary experiment and sub-experiment codes such as E1-A/E1-B/E1-C, E2a/E2b/E2c, and E5, please see Table R0 in our General Comment, Part 1.)

3. Rerunning DisLoRA on Qwen2.5-7B Commonsense in Our Pipeline (E5) (W2 & Q2)
In Table 1 (tab:main-commonsense), we cited the published Qwen2.5-7B numbers from the DisLoRA paper (Table 2) as a public reference. As noted in our hyperparameter appendix (tab:main-hyperparams), the published DisLoRA recipe differs from ours at r=8r{=}8 (learning rate 5×10−45{\times}10^{-4} vs. 3×10−43{\times}10^{-4}) and r=32r{=}32 (α=48\alpha{=}48 vs. 6464). To answer Q2 directly, we reran DisLoRA on Qwen2.5-7B-Instruct across all three ranks r∈{8,16,32}r{\in}\{8, 16, 32\} and all three seeds {42,123,7}\{42, 123, 7\} (9 full runs, E5) inside our own training and evaluation pipeline.

Shared settings and remaining difference between our DisLoRA reruns and SVDoRA (run_B1.sh & training_config.json):

  • Shared: Base model (Qwen2.5-7B-Instruct), training set (commonsense_170k), target modules (q/k/v/o_proj), rank r∈{8,16,32}r{\in}\{8,16,32\} with α=2r\alpha{=}2r (16,32,6416, 32, 64), AdamW learning rate 3×10−43{\times}10^{-4}, linear schedule, weight decay 0.010.01, LoRA dropout 0.050.05, effective batch size 1818, 33 epochs, Phase-1 warmup of 1/31/3 total steps (9,4679,467 steps), training seeds {42,123,7}\{42, 123, 7\}, and the identical 8-task evaluation script (commonsense_evaluate.py, beam size 44, max new tokens 3232).

Table R5. Qwen2.5-7B-Instruct commonsense reasoning (8-task average %, mean$\pm$std over 3 seeds: 42, 123, 7) with DisLoRA rerun in our training and evaluation pipeline.

RankDisLoRA (Published Table 2)DisLoRA (Our Pipeline, 3 seeds)LoRA-DASH (Our Pipeline, 3 seeds)SVDoRA (Ours, 3 seeds)SVDoRA −- Rerun DisLoRA
885.5$\pm$1.387.0$\pm$0.186.8$\pm$0.288.4$\pm$0.0+1.4,pp
1686.1$\pm$1.087.4$\pm$0.186.7$\pm$0.188.1$\pm$0.4+0.7,pp
3286.2$\pm$1.487.1$\pm$0.186.4$\pm$0.187.5$\pm$0.4+0.4,pp

Table R5b. Per-task breakdown at r=16r{=}16 on Qwen2.5-7B-Instruct (3 seeds).

MethodBoolQPIQASIQAHellaS.WinoG.ARC-cARC-eOBQA8-Task Avg.
DisLoRA (Our Pipeline)72.4$\pm$0.289.5$\pm$0.280.4$\pm$0.394.9$\pm$0.186.0$\pm$0.388.8$\pm$0.295.7$\pm$0.291.4$\pm$0.687.4$\pm$0.1
SVDoRA (Ours)74.0$\pm$0.390.0$\pm$0.481.1$\pm$0.194.7$\pm$0.088.6$\pm$0.688.5$\pm$0.396.1$\pm$0.091.7$\pm$1.688.1$\pm$0.4

Key Takeaways (Q2):

  1. Rerun DisLoRA improves over published numbers, and SVDoRA remains highest at all three ranks: In our pipeline, DisLoRA rises from 85.5/86.1/86.2%85.5 / 86.1 / 86.2\% to 87.0±0.1%/87.4±0.1%/87.1±0.1%87.0\pm0.1\% / 87.4\pm0.1\% / 87.1\pm0.1\% (exact 2-decimal averages: 86.99±0.08%86.99\pm0.08\%, 87.38±0.14%87.38\pm0.14\%, 87.14±0.05%87.14\pm0.05\%). Against this stronger baseline, SVDoRA leads at all three ranks: +1.4 pp+1.4\text{ pp} at r=8r{=}8 (88.4±0.0%88.4\pm0.0\%), +0.7 pp+0.7\text{ pp} at r=16r{=}16 (88.1±0.4%88.1\pm0.4\%, higher on 6 of 8 tasks in Table R5b), and +0.4 pp+0.4\text{ pp} at r=32r{=}32 (87.5±0.4%87.5\pm0.4\%).
  2. Statistical resolution across ranks: The margin is clearest at r=8r{=}8 (+1.4 pp+1.4\text{ pp}), moderate at r=16r{=}16 (+0.7 pp+0.7\text{ pp}), and within SVDoRA's seed standard deviation (0.4 pp0.4\text{ pp}) at r=32r{=}32. On Qwen2.5-1.5B-Instruct at r=16r{=}16 (Table A1), our in-pipeline DisLoRA run (78.1%78.1\%, single seed 42) also trails SVDoRA (79.0±0.4%79.0\pm0.4\%).

4. Mathematical Reasoning Comparison (W2)
Your observation on Table 2 (tab:main-math) is completely accurate: DisLoRA has a higher three-task average on both Qwen2.5-7B-Instruct (54.2%54.2\% vs. 52.1%52.1\%) and Llama-3-8B-Instruct (39.9%39.9\% vs. 38.4%38.4\%). We acknowledge this openly. The gap is concentrated on AQuA-RAT, a 254-item out-of-distribution multiple-choice test (trained only on metamath_40k, which rewrites GSM8K and MATH): on Qwen2.5-7B, AQuA is 34.6±2.8%34.6\pm2.8\% vs. 39.2±2.6%39.2\pm2.6\% (overlapping intervals), while on Llama-3-8B it is 18.3±0.2%18.3\pm0.2\% vs. 23.4±0.2%23.4\pm0.2\% (a non-overlapping −5.1 pp-5.1\text{ pp} deficit for SVDoRA). On the in-distribution GSM8K$+$MATH average, SVDoRA ranks first on Llama-3-8B (48.4±0.1%48.4\pm0.1\% vs. DisLoRA 48.1±0.1%48.1\pm0.1\%) and second on Qwen2.5-7B (60.8±0.3%60.8\pm0.3\% vs. 61.7±0.2%61.7\pm0.2\%, ahead of LoRA/MiLoRA at 60.6%60.6\%).

Reviewer_6c57OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a new PEFT method called SVDoRA that trains task specific generation and verification spectral directions chosen using answer correctness for verification in addition to SVD initialized residual adapter. The method achieves strong commonsense reasoning results on Qwen2.5 while showing mixed results on math, code and factual QA.

Critical strengths
  • The SVDoRA parameter efficient technique proposed in the paper is well motivated. This is an important research direction and SVDoRA to my knowledge is one of the first to use outcome labels to select weight-space directions.

  • The work provides detailed mathematical definitions and formulations to explain the methodology, while referring to prior works for inherited components.

  • The control and ablation experiments are extensive.

Critical weaknesses
  • The results outside of commonsense reasoning are mixed and don't show clear accuracy gains. This calls into question the generalizability of the technique considering it adds significant complexity.

  • The task specific verification directions only marginally outperform majority-class baseline showing limited evidence of encoding meaningful correctness information.

  • The method specific hyper-parameters are tuned on commonsense reasoning and reused for other tasks. An analysis to confirm that the same parameters generalize across tasks is missing.

Questions for authors
  • Do the same hyper-parameters remain optimal across tasks? Is there any connection between this choice and the observed results where commonsense reasoning shows gains while the results are mixed for other tasks? A check on another task would help clarify this.

  • Many of the Qwen-2.5 7B baselines are picked from the DisLoRA paper. Were their training setting matched to yours?

4 comments
AuthorsOpenReview ↗

Response to Reviewer 6c57 (Part 1/3): Hyperparameter Transfer on Math, Complexity Breakdown, and Simplified Pipeline (W1, W3, Q1)

1. Hyperparameter Transfer Check on Mathematical Reasoning (E4) (W3 & Q1)
We thank Reviewer 6c57 for recognizing that SVDoRA is well motivated, that using outcome labels to select weight-space directions is an important direction. We reused the single commonsense-selected configuration across all families without per-task tuning (§3.1, Limitations) to avoid cherry-picking. On commonsense (Table A5 / tab:b11-sass, single seed, anchor σ=0.17 pp\sigma{=}0.17\text{ pp}), selection hyperparameters (w2∈{0,0.15,0.6}w_2{\in}\{0, 0.15, 0.6\}, w1b∈{0,1.0}w_{1b}{\in}\{0, 1.0\}, SA steps ∈{250,1000}\in\{250, 1000\}) stay within ±0.23 pp\pm 0.23\text{ pp}, with only Ncal=128N_{\text{cal}}{=}128 costing −0.43 pp-0.43\text{ pp}. Following Q1 ("A check on another task would help clarify this"), we ran a one-factor, single-seed check (E4) on mathematical reasoning (Qwen2.5-1.5B-Instruct on metamath_40k, seed 42, greedy decoding on GSM8K [1,319 items] and MATH [3,669 items]):

Table R4. One-factor hyperparameter transfer check on mathematical reasoning (Qwen2.5-1.5B-Instruct trained on metamath_40k, seed 42, greedy decoding).

ConfigurationGSM8K Acc (%)MATH Acc (%)GSM8K$+$MATH Avg (%)Δ\Delta vs. Anchor
Locked default anchor (Ncal=512,w2=0.3,λconf=0.5N_{\text{cal}}{=}512, w_2{=}0.3, \lambda_{\text{conf}}{=}0.5)61.87 (816/1319)18.97 (696/3669)40.42—
Larger calibration budget (Ncal=1024N_{\text{cal}}{=}1024)63.08 (832/1319)19.08 (700/3669)41.08+0.66,pp
Higher generation alignment weight (w2=0.6w_2{=}0.6)60.96 (804/1319)19.16 (703/3669)40.06-0.36,pp
Disable GVSCR reweighting (λconf=0\lambda_{\text{conf}}{=}0)59.67 (787/1319)18.26 (670/3669)38.96-1.45,pp

What Table R4 Shows (Q1):

  1. Sensitivity across factors on this single-seed math run: Doubling w2w_2 to 0.60.6 moves the GSM8K$+$MATH average by −0.36 pp-0.36\text{ pp} (40.06%40.06\%), whereas doubling NcalN_{\text{cal}} to 10241024 yields +0.66 pp+0.66\text{ pp} (41.08%41.08\%, +1.21 pp+1.21\text{ pp} on GSM8K; vs. +0.08 pp+0.08\text{ pp} on multiple-choice commonsense in Table A5). This suggests that per-task tuning (such as a larger calibration pass on math) may offer modest gains and have some connection to the mixed cross-task transfer when reusing a single fixed configuration; our main paper reports conservative, un-tuned numbers.
  2. Effect of λconf=0\lambda_{\text{conf}}{=}0 in this run: Setting λconf=0\lambda_{\text{conf}}{=}0 resulted in a −1.45 pp-1.45\text{ pp} lower GSM8K$+$MATH average (38.96%38.96\% vs. 40.42%40.42\%) in this single-seed 1.5B run.

2. Cross-Task Pattern, Training/Inference Complexity, and Simplified Pipeline Variants (E3, E7/E8) (W1)
We agree that results outside commonsense are mixed, and we clarify the cross-task pattern and overhead:

  • Cross-task pattern: Performance is strongest when the verifier is exact and the output space is discrete (commonsense); competitive on in-distribution open-ended tasks with exact verifiers (math: 1st on Llama-3-8B GSM8K$+$MATH at 48.4%48.4\% vs. 48.1%48.1\%, 2nd on Qwen2.5-7B at 60.8%60.8\% vs. 61.7%61.7\%, though lower on OOD AQuA; factual QA: 1st on Llama-3-8B TruthfulQA transfer at 25.6%25.6\%, tied on two-task average at 49.9%49.9\%); and least consistent when calibration uses a similarity heuristic rather than execution (code: 1st on Llama-3-8B at 60.3%60.3\% vs. 57.8%57.8\%, 4th on Qwen2.5-7B at 65.5%65.5\%).
  • Training & inference complexity: (i) Inference: Zero overhead—no inference-time verification or extra forward passes (§1). (ii) Training: Total steps are unchanged (Phase-1 warmup is the first 1/31/3 of steps, 9,467/28,4009,467 / 28,400 on 7B commonsense). The only added step is a one-time Phase-2 pass at step TwT_w: greedy generation (≤32\le 32 tokens) on Ncal=512N_{\text{cal}}{=}512 prompts plus 4×20004{\times}2000 annealing steps of cost O(Ncal)O(N_{\text{cal}}) on cached projections.
  • Simplified pipeline variants (E3 & E7/E8): While GVSCR was designed to close the loop between Phase-2 selection and Phase-3 updates (with GVSCR Dev-Tuned at gvscr_lambda_warmup_steps=200 reaching 49.86±0.32%49.86\pm0.32\% on MedQA test and 27.22±0.12%27.22\pm0.12\% on zero-shot MMLU-Pro in E7/E8, ahead of Uniform CE at 48.94±0.33%48.94\pm0.33\% / 26.53±0.12%26.53\pm0.12\% and Focal at 49.12±0.13%49.12\pm0.13\%), if a simpler pipeline is preferred: (a) disabling GVSCR (Off, uniform CE) reaches 79.39±0.21%79.39\pm0.21\% on 1.5B commonsense (Table A4) and 48.94±0.33%48.94\pm0.33\% on MedQA test (GVSCR Default is 48.65±0.26%48.65\pm0.26\%); and (b) in our simplified variant without separate TSGD/TSVD roles and with uniform CE (E3), setting kg=0,kv=8,w2=0k_g{=}0, k_v{=}8, w_2{=}0 reaches 48.47%48.47\% (617/1273617/1273) on MedQA test vs. 49.02%49.02\% (624/1273624/1273, −0.55 pp-0.55\text{ pp}) for the two-role anchor (E1-A).
AuthorsOpenReview ↗

Response to Reviewer 6c57 (Part 2/3): TSVD Probing, Directional Interventions, and Entropy Fusion (W2)

3. Evidence on Whether TSVD Encodes Meaningful Correctness Information (W2)
In Table 6 (tab:probing-main, GSM8K, N=1319N{=}1319), raw accuracy is compressed because the positive rate is 0.6240.624 (weighted F1 is 0.6390.639 vs. 0.4790.479 majority, AUROC 0.6720.672 vs. 0.6740.674 NullSpace). To evaluate TSVD more thoroughly on Qwen2.5-1.5B-Instruct (r=16,kg=kv=4r{=}16, k_g{=}k_v{=}4, uniform CE), we ran E1 and E2:

  1. Controlled Label Ablation (E1): Holding warmup, shortlist, annealing, and uniform CE fixed, outcome-label supervision (E1-A vs. E1-C) is a source of performance gain: aligned labels (E1-A) achieve much higher TSVD calibration separation dverd_{\text{ver}} than shuffled (E1-B) and removed labels (E1-C, w1=w1b=0w_1{=}w_{1b}{=}0) across all three benchmarks—MedQA (0.5580.558 vs. 0.333/0.2010.333 / 0.201), StrategyQA (2.2112.211 vs. 0.352/0.5790.352 / 0.579), and GSM8K (0.6410.641 vs. 0.419/0.1420.419 / 0.142)—and improve test accuracy over E1-C by +0.67 pp+0.67\text{ pp} on MedQA (48.47±0.55%48.47\pm0.55\% vs. 47.80±0.04%47.80\pm0.04\%, 2 seeds) and +0.98 pp+0.98\text{ pp} on GSM8K (44.35%44.35\% vs. 43.37%43.37\%). Between aligned (E1-A) and shuffled (E1-B) order, E1-A is required to isolate high-dverd_{\text{ver}} TSVD directions, while the downstream accuracy difference under uniform CE is less pronounced (48.47%48.47\% vs. 48.74%48.74\% on MedQA, 66.81%66.81\% vs. 66.81%66.81\% on StrategyQA, 44.35%44.35\% vs. 46.70%46.70\% on GSM8K), as keeping the non-zero separation terms (w1,w1b>0w_1, w_{1b}{>}0) active in the annealing energy already regularizes direction selection relative to dropping them in E1-C.
  2. Directional Interventions (E2a), Answer-Span Probes (E2b), and Entropy Fusion (E2c) on E1-A: On the aligned MedQA (A-medqa-s42) and StrategyQA (A-strategyqa-s42) checkpoints, we attached module-input pre-hooks mean-pooled over the answer span across layers {7,14,21,27}\{7,14,21,27\}:

Table R2a. Inference-time directional interventions (E2a) on the unmodified E1-A checkpoint (Qwen2.5-1.5B-Instruct, r=16r{=}16, kg=kv=4k_g{=}k_v{=}4, greedy decoding).

Intervention ConditionMedQA Test Acc (%)MedQA Δ\Delta AccMedQA TSVD Probe Drop (Multi-Layer / 4-Layer)StrategyQA Acc (%)StrategyQA Δ\Delta AccStrategyQA TSVD Probe Drop (Multi-Layer / 4-Layer)
Baseline (unmodified E1-A)49.02 (624/1273)—— (base 0.6470 / 0.5825)66.81 (153/229)—— (base 0.5626 / 0.5206)
Remove TSVD (βj→0\beta_j \to 0)46.43 (591/1273)-2.59,pp$-0.1455∗∗/∗∗0.1455** / **-$0.082564.63 (148/229)-2.18,pp−-0.0682 / -\0.0220
Reset TSVD (βj→σj\beta_j \to \sigma_j)48.94 (623/1273)-0.08,pp$-0.0736∗∗/∗∗0.0736** / **-$0.024366.38 (152/229)-0.44,pp−-0.0077 / -\0.0199
Remove TSGD (αi→0\alpha_i \to 0)47.05 (599/1273)-1.96,pp$-$0.0373 / $-$0.015663.32 (145/229)-3.49,pp$+$0.0690 / $+$0.0251
Reset TSGD (αi→σi\alpha_i \to \sigma_i)49.18 (626/1273)+0.16,pp$-$0.0014 / $-$0.001066.81 (153/229)+0.00,pp$+$0.0017 / $+$0.0003
Remove σ\sigma-matched frozen (mean of 3 draws)48.05 (611.7/1273 (603,614,618))-0.97,pp$-$0.0254 / $-$0.011266.23 (151.7/229 (153,155,147))-0.58,pp$-$0.0092 / $-$0.0061

Table R2b. Baseline correctness probing (E2b) and incremental AUROC over output token entropy (E2c) across layers {7,14,21,27}\{7,14,21,27\}.

SubspaceMedQA Multi-Layer AUROC (BAcc)MedQA Pure Norm AUROCMedQA 4-Layer Mean AUROCMedQA E2c Δ\DeltaAUROC over Entropy (QDA / 2D-Logit / GNB)StrategyQA Multi-Layer AUROC (BAcc)StrategyQA 4-Layer Mean AUROCStrategyQA E2c Δ\DeltaAUROC (2D-Logit)
TSVD0.6470 (0.6118)0.61720.5825+0.0226 / +0.0048 / +0.06850.5626 (0.5443)0.5206+0.0047
TSGD0.6089 (0.5808)0.55530.5648-0.0158 / -0.0095 / -0.07290.5054 (0.4976)0.4862-0.0108
NullSpace0.5955 (0.5723)0.58100.5528-0.0256 / -0.0159 / -0.01430.5776 (0.5368)0.5195+0.0146
Random0.6210 (0.5829)0.57960.5698-0.0133 / -0.0073 / -0.04350.5115 (0.5066)0.4995-0.0078
  • Key Takeaways (W2): On MedQA, TSVD leads all controls on multi-layer AUROC (0.64700.6470, BAcc 0.61180.6118), pure norm AUROC (0.61720.6172 vs. NullSpace 0.58100.5810), βj→0\beta_j{\to}0 test accuracy drop (−2.59 pp-2.59\text{ pp} vs. TSGD −1.96 pp-1.96\text{ pp} and frozen −0.97 pp-0.97\text{ pp}), βj→0\beta_j{\to}0 probe drop (−0.1455-0.1455), and E2c entropy fusion (+2.26 pp+2.26\text{ pp} QDA, +0.48 pp+0.48\text{ pp} 2D-logit, +6.85 pp+6.85\text{ pp} GNB). On StrategyQA (229 items), TSVD leads on 4-layer mean AUROC (0.52060.5206), multi-layer BAcc (0.54430.5443), and βj→0\beta_j{\to}0 probe drop (−0.0682-0.0682), and gains +0.47 pp+0.47\text{ pp} on E2c 2D-logit (vs. Random −0.78 pp-0.78\text{ pp}, NullSpace +1.46 pp+1.46\text{ pp}).
AuthorsOpenReview ↗

Response to Reviewer 6c57 (Part 3/3): Qwen2.5-7B Baseline Training Settings and DisLoRA Rerun (Q2)

(Note: For a quick reference of all supplementary experiment and sub-experiment codes such as E1-A/E1-B/E1-C, E2a/E2b/E2c, E3, E4, and E5, please see Table R0 in our General Comment, Part 1.)

4. Training Settings of Qwen2.5-7B Baselines and DisLoRA Rerun (E5) (Q2)
Thank you for asking whether the Qwen2.5-7B baselines picked from the DisLoRA paper had matched training settings:

  1. Which 7B baselines were cited vs. run by us, and how settings compare: On Qwen2.5-7B (Table 1 / tab:main-commonsense), PiSSA, LoRA-DASH, and SVDoRA were trained in our pipeline, while the four daggered rows (LoRA, MiLoRA, SORSA, and DisLoRA) were cited from Table 2 of the DisLoRA paper (Yifei et al., 2025). Comparing Table 5 of the DisLoRA paper against our recipe (tab:main-hyperparams), at r=16r{=}16 both use α=32\alpha{=}32, learning rate 3×10−43{\times}10^{-4}, linear schedule, dropout 0.050.05, and 33 epochs on commonsense_170k; however, at r=8r{=}8 the DisLoRA paper uses learning rate 5×10−45{\times}10^{-4} (vs. our 3×10−43{\times}10^{-4}), and at r=32r{=}32 it uses α=48\alpha{=}48 (vs. our α=64\alpha{=}64). We did not rerun 7B LoRA, MiLoRA, or SORSA, so those three cited rows reflect the DisLoRA paper's settings.
  2. What our 1.5B runs and 7B DisLoRA rerun (E5) establish:
    • First, on Qwen2.5-1.5B-Instruct (Table A1 / tab:commonsense-1.5b), all six baselines (LoRA, PiSSA, MiLoRA, SORSA, LoRA-DASH, DisLoRA) were trained inside our own pipeline, where LoRA-DASH and DisLoRA are the strongest baselines and SVDoRA leads at r=8,16r{=}8, 16 (trailing LoRA-DASH by 0.2 pp0.2\text{ pp} at r=32r{=}32).
    • Second, on Qwen2.5-7B, we reran the strongest cited baseline, DisLoRA, across all three ranks r∈{8,16,32}r{\in}\{8, 16, 32\} and all three seeds {42,123,7}\{42, 123, 7\} (9 full runs, E5) inside our own training and evaluation pipeline (sharing base model Qwen2.5-7B-Instruct, commonsense_170k, q/k/v/o_proj, α=2r\alpha{=}2r, lr 3×10−43{\times}10^{-4}, linear decay, weight decay 0.010.01, dropout 0.050.05, effective batch 1818, 33 epochs, 1/31/3-step warmup, seeds {42,123,7}\{42,123,7\}, and the identical 8-task evaluation script). As documented in tab:main-hyperparams.

Table R5. Qwen2.5-7B-Instruct commonsense reasoning (8-task average %, mean$\pm$std over 3 seeds: 42, 123, 7) with DisLoRA rerun in our training and evaluation pipeline.

RankDisLoRA (Published Table 2)DisLoRA (Our Pipeline, 3 seeds)LoRA-DASH (Our Pipeline, 3 seeds)SVDoRA (Ours, 3 seeds)SVDoRA −- Rerun DisLoRA
885.5$\pm$1.387.0$\pm$0.186.8$\pm$0.288.4$\pm$0.0+1.4,pp
1686.1$\pm$1.087.4$\pm$0.186.7$\pm$0.188.1$\pm$0.4+0.7,pp
3286.2$\pm$1.487.1$\pm$0.186.4$\pm$0.187.5$\pm$0.4+0.4,pp

Table R5b. Per-task breakdown at r=16r{=}16 on Qwen2.5-7B-Instruct (3 seeds).

MethodBoolQPIQASIQAHellaS.WinoG.ARC-cARC-eOBQA8-Task Avg.
DisLoRA (Our Pipeline)72.4$\pm$0.289.5$\pm$0.280.4$\pm$0.394.9$\pm$0.186.0$\pm$0.388.8$\pm$0.295.7$\pm$0.291.4$\pm$0.687.4$\pm$0.1
SVDoRA (Ours)74.0$\pm$0.390.0$\pm$0.481.1$\pm$0.194.7$\pm$0.088.6$\pm$0.688.5$\pm$0.396.1$\pm$0.091.7$\pm$1.688.1$\pm$0.4

Both of the top spectral baselines on 7B (LoRA-DASH and DisLoRA) are thus evaluated in our pipeline across all three ranks and three seeds: SVDoRA leads by +1.4 pp+1.4\text{ pp} at r=8r{=}8, +0.7 pp+0.7\text{ pp} at r=16r{=}16 (higher on 6 of 8 tasks), and +0.4 pp+0.4\text{ pp} at r=32r{=}32 (within SVDoRA's 0.4 pp0.4\text{ pp} seed std).

Thank you for the detailed analysis.

The math sensitivity check experiments suggest minimal correlation between the hyperparameter configuration and observed mixed cross-task results.

The claim that gains appear when the verifier is exact, the output space is discrete and the task is in-domain lacks thorough experimental evidence beyond commonsense results. Math has an exact verifier, yet the margins there are small.

My question about matching training settings for the baselines is resolved.

I still maintain my assessment that generalization beyond commonsense reasoning has not been demonstrated, which is why I am keeping my score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of New Experiments and Clarifications Across All Reviews (Part 2/2): Topic1–Topic3

  1. T1 — Outcome-Label Supervision (E1) and TSVD Evaluation (E2):

    • Controlled Label Ablation (E1, Table R1): Outcome-label supervision is a clear source of our performance gain (E1-A vs. E1-C): E1-A achieves much higher calibration separation dverd_{\text{ver}} than E1-B and E1-C on MedQA (0.5580.558 vs. 0.333/0.2010.333 / 0.201), StrategyQA (2.2112.211 vs. 0.352/0.5790.352 / 0.579), and GSM8K (0.6410.641 vs. 0.419/0.1420.419 / 0.142), and improves test accuracy over E1-C by +0.67 pp+0.67\text{ pp} on MedQA (48.47±0.55%48.47\pm0.55\% vs. 47.80±0.04%47.80\pm0.04\%, 2 seeds: +1.26 pp+1.26\text{ pp} on s42, +0.08 pp+0.08\text{ pp} on s123) and +0.98 pp+0.98\text{ pp} on GSM8K (44.35%44.35\% vs. 43.37%43.37\%). Between aligned (E1-A) and shuffled (E1-B) order, E1-A is required to isolate high-dverd_{\text{ver}} TSVD directions, while on downstream accuracy under uniform CE the difference between aligned and shuffled order is less pronounced (48.47%48.47\% vs. 48.74%48.74\% on MedQA, 66.81%66.81\% vs. 66.81%66.81\% on StrategyQA, 44.35%44.35\% vs. 46.70%46.70\% on GSM8K), as keeping the non-zero separation terms (w1,w1b>0w_1, w_{1b}{>}0) active in the annealing energy already regularizes direction selection relative to dropping them in E1-C.
    • TSVD Evaluation (E2, Tables R2a & R2b): On MedQA (E1-A), zeroing TSVD (E2a, βj→0\beta_j{\to}0) causes the largest test accuracy drop (−2.59 pp-2.59\text{ pp}, vs. −1.96 pp-1.96\text{ pp} for TSGD zeroing and −0.97 pp-0.97\text{ pp} for frozen removal). Under E2b, TSVD attains the highest multi-layer AUROC (0.64700.6470, BAcc 0.61180.6118) and pure norm AUROC (0.61720.6172); zeroing/resetting TSVD drops multi-layer AUROC by −0.1455-0.1455 / −0.0736-0.0736 (vs. ≤0.0373\le 0.0373 for controls); and E2c yields positive AUROC gains across all readouts (+2.26 pp+2.26\text{ pp} QDA, +0.48 pp+0.48\text{ pp} 2D-logit, +6.85 pp+6.85\text{ pp} GNB; controls are negative). On StrategyQA, TSVD leads on 4-layer AUROC (0.52060.5206), BAcc (0.54430.5443), and βj→0\beta_j{\to}0 probe drop (−0.0682-0.0682), and gains +0.47 pp+0.47\text{ pp} on E2c 2D-logit (vs. Random −0.78 pp-0.78\text{ pp}).
  2. T2 — Simplified Variant Without Separate TSGD/TSVD Roles (E3) and Phase-3 Reweighting (E7/E8):

    • Simplified Variant Without Separate TSGD/TSVD Roles + Uniform CE (E3, Table R3): Directly addressing sUz3 Q1, removing the separate TSGD role (kg=0,kv=8,w2=0k_g{=}0, k_v{=}8, w_2{=}0) under uniform CE reaches 48.47%48.47\% (617/1273617/1273) vs. 49.02%49.02\% (624/1273624/1273, −0.55 pp-0.55\text{ pp}) for two-role E1-A.
    • GVSCR (E7/E8, Table R6): Coupling Phase-2 TSVD readouts into Phase-3 updates. GVSCR was not purely added to increase the accuracy point: without it, the TSVD directions extracted in Phase-2 only follow the main loss passively, and the role assigned to them for distinguishing correct from incorrect LLM outputs is never explicitly invoked and exploited in the training objective. The design is instead a step-to-step training loop in which the correctness-correlated TSVD readout at step tt modulates that step's update, so that step t+1t{+}1 continues from weights that have already absorbed this last step's TSVD feedback signal, closing the loop between Phase-2 selection and Phase-3 optimization without claiming a self-verification or self-improvement capability. On MedQA test (E8), GVSCR Dev-Tuned (gvscr_lambda_warmup_steps=200, dev 47.56%47.56\%) reaches 49.86±0.32%49.86\pm0.32\%, outperforming Uniform CE (48.94±0.33%48.94\pm0.33\%, +0.92 pp+0.92\text{ pp}), GVSCR Default (48.65±0.26%48.65\pm0.26\%, +1.21 pp+1.21\text{ pp}), and all four literature methods (Focal 49.12±0.13%49.12\pm0.13\%, SPL 47.58±0.27%47.58\pm0.27\%, GCE 46.24±0.26%46.24\pm0.26\%, JTT 46.19±0.61%46.19\pm0.61\%), and reaches 27.22±0.12%27.22\pm0.12\% on zero-shot MMLU-Pro (SPL 27.12±0.01%27.12\pm0.01\%, Uniform CE 26.53±0.12%26.53\pm0.12\%).
  3. T3 — 7B DisLoRA Rerun (E5, Table R5) and T4 — 1.5B Math Check (E4, Table R4):

    • 7B DisLoRA (E5): Rerunning DisLoRA across r∈{8,16,32}r{\in}\{8,16,32\} in our pipeline yields 87.0±0.1%87.0\pm0.1\%, 87.4±0.1%87.4\pm0.1\%, and 87.1±0.1%87.1\pm0.1\%, while SVDoRA leads at all ranks (88.4±0.0%88.4\pm0.0\%, 88.1±0.4%88.1\pm0.4\%, 87.5±0.4%87.5\pm0.4\%, by +1.4,+0.7,+0.4 pp+1.4, +0.7, +0.4\text{ pp}).
    • 1.5B Math (E4): On GSM8K + MATH (anchor 40.42%40.42\%), w2=0.6w_2{=}0.6 is −0.36 pp-0.36\text{ pp} (40.06%40.06\%), Ncal=1024N_{\text{cal}}{=}1024 is +0.66 pp+0.66\text{ pp} (41.08%41.08\%), and λconf=0\lambda_{\text{conf}}{=}0 is −1.45 pp-1.45\text{ pp} (38.96%38.96\%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.