- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Existing spectral parameter-efficient fine-tuning methods typically rely on fixed-rank boundaries in the singular spectrum to protect pretrained capabilities. However, the authors show that reasoning-relevant information is distributed across the singular spectrum rather than concentrated at either end. To address this limitation, they propose PALoRA, a two-stage framework that first uses SVF to identify singular directions important for reasoning and then introduces a graded penalty to constrain LoRA updates along these directions during factual knowledge injection. This mechanism enables task-aware protection of reasoning capabilities while learning new knowledge. Extensive experiments demonstrate that PALoRA effectively preserves reasoning capabilities while maintaining competitive knowledge acquisition with minimal additional parameter overhead.
Critical strengths
-
The paper is well motivated and challenges the assumption that protecting a fixed-rank boundaries of the singular spectrum is necessary for preserving reasoning capabilities. The observation that skill-relevant directions are distributed across the spectrum naturally motivates relevance-based protection. The authors support this motivation with both theoretical analysis and empirical evidence.
-
PALoRA combines SVF-based skill relevance estimation with graded protection of singular directions during LoRA-based knowledge injection. The design is intuitive and well motivated. The theoretical analysis further explains the relationship between SVF scaling and skill relevance and provides bounds on skill degradation under the proposed penalty.
-
The experiments cover two model families, three reasoning domains, multiple knowledge-injection budgets, and five relevant baselines. The results generally support the main claims. The ablation studies are informative in demonstrating the contributions of the SVF expert and graded protection.
Critical weaknesses
-
Insufficient validation of SVF-derived importance weights. The paper assumes that the magnitude of SVF scaling reflects how strongly a reasoning skill relies on each singular direction. However, directions requiring larger adjustments during skill learning are not necessarily those most critical for skill preservation. For example, a direction may already support reasoning effectively and require little scaling, yet modifying it could still cause substantial degradation. Moreover, SVF-guided and raw top-k protection achieve identical performance despite selecting nearly disjoint directions.
-
The independent benefit of SVF-guided graded protection remains unclear. Although the ablations show that the proposed penalty improves skill retention, it remains unclear whether the gains arise from the accuracy of SVF-derived relevance weights or from the general regularization effect of graded protection.
-
Modest practical advantages over strong baselines. In Table 3, PALoRA achieves slightly lower average accuracy and factual recall than OPLoRA. Although PALoRA provides better skill retention under sequential injection, its factual recall is lower. Furthermore, its knowledge-injection stage incurs approximately 1.3–1.4 times the training cost of the baselines.
Questions for authors
-
Could the authors directly validate whether larger SVF scaling deviations correspond to greater sensitivity of reasoning performance to perturbations? For example, measuring skill degradation after controlled perturbations of high- and low-importance singular directions would help verify the proposed relevance measure.
-
Section 4.2 shows that SVF-guided and raw top-k protection achieve identical reasoning performance despite selecting nearly disjoint directions. How should this observation be interpreted? Could the authors further compare PALoRA with uniform or shuffled graded weights under matched conditions to distinguish task-specific protection from general regularization effects?
-
The proposed penalty constrains LoRA updates along the pretrained right singular directions. How are updates in directions outside the span of these singular vectors handled, particularly for rectangular MLP matrices? Could the authors quantify whether such unprotected updates affect skill retention?
-
In Figure 3(b), why does PALoRA's weighted penalty initially increase before decreasing? Could the authors clarify the mechanism behind this behavior and whether it is associated with changes in reasoning performance during training?
-
Given the additional SVF training and regularization costs, could the authors provide a more comprehensive comparison of skill retention, factual recall, and end-to-end computational cost against OPLoRA?
-
Figures 2(a) and 2(b) in the main paper are difficult to read. It would be helpful if the authors could improve their visual clarity to make the results easier to interpret.
5 comments
Part 2 of Rebuttal
Could the authors further compare PALoRA with uniform or shuffled graded weights under matched conditions to distinguish task-specific protection from general regularization effects?
We replaced only the protection weights and left the injection run otherwise identical.
| Protection weights | mean w | λ | Recall (%) | GSM8K (%) |
|---|---|---|---|---|
| none (SVF + LoRA) | — | 0 | 62.0 | 78.80 |
| uniform, w = 1 | 1.00 | 3e-5 | 61.6 | 81.50 |
| uniform, w = 1 | 1.00 | 1e-4 | 60.0 | 80.59 |
| uniform, w = 1 | 1.00 | 3e-4 | 55.6 | 80.59 |
| shuffled SVF w | 0.34 | 3e-4 | 59.8 | 81.03 |
| SVF w (PALoRA) | 0.34 | 3e-4 | 61.1 | 83.04 |
Two remarks on how the rows are matched. Setting w = 1 nearly triples the mean weight, so the same λ is not the same penalty strength, and we therefore report uniform weights at three values of λ rather than one. Permuting the weights leaves the mean unchanged, so the shuffled row is matched to PALoRA in λ, in penalty strength and hence in update size, and differs only in which direction carries which weight. It is pooled over two permutations and three seeds, at 81.03 ± 0.88 GSM8K and recall 59.8 ± 2.4.
The two effects separate as follows. Both controls lift skill above the 78.80 of unconstrained injection, so limiting the size of the update is worth about two points on its own, which is the general regularization effect the reviewer names. The SVF profile adds a further 1.5 to 2.0 points on top of it: 83.04 against 80.59 for uniform at PALoRA's own λ and 81.03 for shuffled, and against 81.50 for the best uniform row, which reaches it at recall 61.6 against PALoRA's 61.1. We note that tuning λ does not close this gap in either direction, which supports our reading that λ sets how large the update is while the weights set where it goes.
How are updates in directions outside the span of these singular vectors handled, particularly for rectangular MLP matrices? Could the authors quantify whether such unprotected updates affect skill retention?
Of the three adapted matrices, only one has input directions outside that span. gate_proj and up_proj map 4096 to 14336, so their 4096 right singular vectors span the input space and the penalty reaches every direction. down_proj maps 14336 to 4096, leaving 10240 directions outside the span. These are its null space: the pretrained matrix maps them to zero, and the expert rescales singular values, which exist only inside the span, so there is no skill importance to assign outside it.
Theorem 2 completes the basis with w_i = 0 there, so nothing in the complement is charged. Table 4(a) measures what this leaves open. The complement is unconstrained in both arms of that ablation: rows B and D differ only in the graded penalty, which acts on the span alone, and it recovers 4.24 GSM8K points, from 78.80 to 83.04 against the deployed expert's 83.85. Retention is therefore governed by the directions the penalty weights, and constraining those is enough to recover it with the complement left free throughout. We will state this in Appendix B.3.
In Figure 3(b), why does PALoRA's weighted penalty initially increase before decreasing? Could the authors clarify the mechanism behind this behavior and whether it is associated with changes in reasoning performance during training?
The penalty is quadratic in the update, L_ortho = ‖BAV diag(w)‖²_F, and B is zero at initialization, so both the penalty and its gradient start at zero while the SFT gradient does not. The first epochs are therefore governed by acquisition, and the penalty rises with ‖ΔW‖. Once the update is large enough for λ∇L_ortho to be comparable to the SFT gradient, and acquisition begins to saturate, the penalty term redistributes the update across the spectrum and pulls the penalty down, ending 6.6 times below the unconstrained run.
On the second question, what the descent does is redistribute rather than shrink. Figure 3(a) shows the end state: 1.6% of the update energy in the top importance decile against 24.4% in the bottom one, where unconstrained LoRA holds about 10% in each. Acquisition is not impeded, the SFT loss on the facts ending at 0.357 against 0.356, while skill accuracy ends 4.24 points higher.
Part 3 of Rebuttal
Given the additional SVF training and regularization costs, could the authors provide a more comprehensive comparison of skill retention, factual recall, and end-to-end computational cost against OPLoRA?
We bring the three axes together below. Each row averages the 24 configurations of Table 3 at one seed, so the two methods are compared on the same runs.
| Seed | PALoRA Acc. (%) | OPLoRA Acc. (%) | PALoRA Rec. (%) | OPLoRA Rec. (%) |
|---|---|---|---|---|
| 42 | 69.73 | 69.31 | 71.7 | 72.8 |
| 1234 | 69.61 | 69.33 | 71.4 | 73.5 |
| 2024 | 69.08 | 68.90 | 72.3 | 72.1 |
| mean | 69.47 | 69.18 | 71.8 | 72.8 |
PALoRA retains more skill at every seed, while OPLoRA recalls more at two of the three. The ACC column of Table 3 averages the two, so the gap there is one of recall rather than of retention, and PALoRA has the only positive backward transfer in the comparison, +0.3 against 0.0. End to end a run costs 1153 s against 984 s.
To compare retention at matched recall we swept OPLoRA's protected subspace in the cell of Table 4, with three seeds at each k.
| Method | Recall (%) | GSM8K (%) |
|---|---|---|
| OPLoRA, k = 64 | 63.4 ± 2.2 | 81.68 ± 0.61 |
| OPLoRA, k = 128 | 62.1 ± 3.0 | 81.65 ± 1.45 |
| OPLoRA, k = 256 | 61.1 ± 0.9 | 81.98 ± 1.26 |
| OPLoRA, k = 512 | 61.7 ± 2.7 | 81.68 ± 1.36 |
| PALoRA | 61.1 ± 0.8 | 83.04 ± 1.01 |
Skill accuracy moves within 0.4 points over the sweep and no k reaches PALoRA. At k = 256 the two recall equally, and PALoRA holds 1.06 points more skill. We wish to point out that this follows from the paper's analysis. A rank-fixed set leaves the skill loading outside it exposed, however strictly enforced (Corollary 1). Theorem 1 places that loading across the whole spectrum, so enlarging the set removes only a small part of it.
Under sequential injection (Table 5), where the updates accumulate, PALoRA ends round 4 at 79.7 GSM8K against 76.5 and stays above the base model at every round, while OPLoRA ends 2.1 points below it and recalls more over the rounds, by 4.5 points.
The cost difference is confined to the injection stage, since Phase 1 is common to both and PALoRA adds no preprocessing of its own. PALoRA therefore trades about one point of recall for retention that holds at matched recall and widens as updates accumulate, at 1.2 times the cost.
Figures 2(a) and 2(b) in the main paper are difficult to read.
We agree, and we will enlarge both figures for the revision.
We hope these results address the concerns raised; if they do, we would be grateful if you would consider revisiting your assessment, and we remain glad to run any further control you consider decisive during the discussion period.
Part 1 of Rebuttal
We thank the reviewer for the careful and detailed review. We are glad that the reviewer finds the paper well motivated, the distributed observation a natural motivation for relevance-based protection, and the ablations informative in demonstrating the contribution of each stage. The following are our responses to the comments and questions.
For this response we ran three further experiments, all in the configuration of Table 4, on Llama-3.1-8B-Instruct with the GSM8K expert and 1000 unknown facts: controlled perturbations of the deployed expert by importance decile, uniform and shuffled protection weights, and OPLoRA's protected-subspace size at k = 64, 128, 256 and 512. The paired-seed comparison re-analyses the existing runs behind Table 3. We will include all of them in the revision.
Could the authors directly validate whether larger SVF scaling deviations correspond to greater sensitivity of reasoning performance to perturbations?
We perturbed the deployed expert along directions drawn from a single decile of the SVF importance |z* − 1|, with no training. The perturbation matches a real PALoRA update in rank and size, so only the decile differs. Three random draws per decile:
The same perturbation costs 3.21 more GSM8K points in the top decile than in the bottom one. The directions Phase 1 rescaled most are therefore the ones the skill is most sensitive to, which is what the importance measure assumes.
Section 4.2 shows that SVF-guided and raw top-k protection achieve identical reasoning performance despite selecting nearly disjoint directions. How should this observation be interpreted?
The spectral baselines separate the spectrum at a fixed rank and protect the leading side, which carries the largest share of the matrix's spectral energy. OPLoRA protects the k directions of largest singular value absolutely. Table 2 tests whether that set preserves the skill best. The SVF set is chosen by measured skill relevance instead, and overlaps the leading set at a micro Jaccard index of 0.012. The two preserve the skill equally, at 81.12 GSM8K, while random-k reaches 79.61 at comparable recall. Protecting the directions of largest energy is therefore not necessarily the best protection of the skill. If no set of 64 directions is privileged, the protection should follow the importance profile over the whole spectrum rather than any cut of it, which is the graded rule of Eq. 3. We hope this clarifies how we read the observation, and we will rewrite the paragraph of Section 4.2 in these terms.