acceptodds
Under review as a conference paper at ICLR 2027

PALoRA: Projection-Adaptive LoRA for Preserving Reasoning in Large Language Models

Abstract

Efficiently updating Large Language Models (LLMs) with new or evolving factual knowledge can erode previously acquired reasoning abilities. Spectral parameter-efficient fine-tuning methods control this interference through a fixed-rank boundary in the singular value decomposition of pre-trained weights, treating singular-value rank as a proxy for skill relevance. We show, both theoretically and empirically, that this premise is incomplete: information essential for reasoning is distributed across the singular spectrum of multilayer perceptron weights rather than confined to either end of it. We therefore introduce PALoRA, a two-stage framework for relevance-based knowledge injection. PALoRA first trains a Singular Value Fine-Tuning (SVF) expert on a target reasoning skill and uses its learned singular scaling vector as a frozen probe of how strongly the skill relies on each singular component. It then injects factual knowledge with Low-Rank Adaptation (LoRA) while penalizing the projection of the update onto every singular direction in proportion to this reliance. We prove that this graded penalty bounds the first-order skill degradation of the update, with fixed-rank protection as its binary case. Across Llama 3.1 8B and Mistral 7B, and across mathematical, coding, and scientific reasoning benchmarks, PALoRA preserves on average 96% of the SVF expert's reasoning performance while maintaining competitive factual recall. It matches the strongest of five baselines without a fixed rank boundary, with the only positive backward transfer, retains more of the skill under sequential injection, and adds less than 0.006% parameter overhead.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
parameter-efficient fine-tuning, LoRA, catastrophic forgetting, plasticity-stability trade-off, continual learning, singular value decomposition, orthogonal subspace learning, reasoning preservation, large language models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_f6FiOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

Existing spectral parameter-efficient fine-tuning methods typically rely on fixed-rank boundaries in the singular spectrum to protect pretrained capabilities. However, the authors show that reasoning-relevant information is distributed across the singular spectrum rather than concentrated at either end. To address this limitation, they propose PALoRA, a two-stage framework that first uses SVF to identify singular directions important for reasoning and then introduces a graded penalty to constrain LoRA updates along these directions during factual knowledge injection. This mechanism enables task-aware protection of reasoning capabilities while learning new knowledge. Extensive experiments demonstrate that PALoRA effectively preserves reasoning capabilities while maintaining competitive knowledge acquisition with minimal additional parameter overhead.

Critical strengths
  1. The paper is well motivated and challenges the assumption that protecting a fixed-rank boundaries of the singular spectrum is necessary for preserving reasoning capabilities. The observation that skill-relevant directions are distributed across the spectrum naturally motivates relevance-based protection. The authors support this motivation with both theoretical analysis and empirical evidence.

  2. PALoRA combines SVF-based skill relevance estimation with graded protection of singular directions during LoRA-based knowledge injection. The design is intuitive and well motivated. The theoretical analysis further explains the relationship between SVF scaling and skill relevance and provides bounds on skill degradation under the proposed penalty.

  3. The experiments cover two model families, three reasoning domains, multiple knowledge-injection budgets, and five relevant baselines. The results generally support the main claims. The ablation studies are informative in demonstrating the contributions of the SVF expert and graded protection.

Critical weaknesses
  1. Insufficient validation of SVF-derived importance weights. The paper assumes that the magnitude of SVF scaling reflects how strongly a reasoning skill relies on each singular direction. However, directions requiring larger adjustments during skill learning are not necessarily those most critical for skill preservation. For example, a direction may already support reasoning effectively and require little scaling, yet modifying it could still cause substantial degradation. Moreover, SVF-guided and raw top-k protection achieve identical performance despite selecting nearly disjoint directions.

  2. The independent benefit of SVF-guided graded protection remains unclear. Although the ablations show that the proposed penalty improves skill retention, it remains unclear whether the gains arise from the accuracy of SVF-derived relevance weights or from the general regularization effect of graded protection.

  3. Modest practical advantages over strong baselines. In Table 3, PALoRA achieves slightly lower average accuracy and factual recall than OPLoRA. Although PALoRA provides better skill retention under sequential injection, its factual recall is lower. Furthermore, its knowledge-injection stage incurs approximately 1.3–1.4 times the training cost of the baselines.

Questions for authors
  1. Could the authors directly validate whether larger SVF scaling deviations correspond to greater sensitivity of reasoning performance to perturbations? For example, measuring skill degradation after controlled perturbations of high- and low-importance singular directions would help verify the proposed relevance measure.

  2. Section 4.2 shows that SVF-guided and raw top-k protection achieve identical reasoning performance despite selecting nearly disjoint directions. How should this observation be interpreted? Could the authors further compare PALoRA with uniform or shuffled graded weights under matched conditions to distinguish task-specific protection from general regularization effects?

  3. The proposed penalty constrains LoRA updates along the pretrained right singular directions. How are updates in directions outside the span of these singular vectors handled, particularly for rectangular MLP matrices? Could the authors quantify whether such unprotected updates affect skill retention?

  4. In Figure 3(b), why does PALoRA's weighted penalty initially increase before decreasing? Could the authors clarify the mechanism behind this behavior and whether it is associated with changes in reasoning performance during training?

  5. Given the additional SVF training and regularization costs, could the authors provide a more comprehensive comparison of skill retention, factual recall, and end-to-end computational cost against OPLoRA?

  6. Figures 2(a) and 2(b) in the main paper are difficult to read. It would be helpful if the authors could improve their visual clarity to make the results easier to interpret.

5 comments
AuthorsOpenReview ↗

Part 1 of Rebuttal

We thank the reviewer for the careful and detailed review. We are glad that the reviewer finds the paper well motivated, the distributed observation a natural motivation for relevance-based protection, and the ablations informative in demonstrating the contribution of each stage. The following are our responses to the comments and questions.

For this response we ran three further experiments, all in the configuration of Table 4, on Llama-3.1-8B-Instruct with the GSM8K expert and 1000 unknown facts: controlled perturbations of the deployed expert by importance decile, uniform and shuffled protection weights, and OPLoRA's protected-subspace size at k = 64, 128, 256 and 512. The paired-seed comparison re-analyses the existing runs behind Table 3. We will include all of them in the revision.

Could the authors directly validate whether larger SVF scaling deviations correspond to greater sensitivity of reasoning performance to perturbations?

We perturbed the deployed expert along directions drawn from a single decile of the SVF importance |z* − 1|, with no training. The perturbation matches a real PALoRA update in rank and size, so only the decile differs. Three random draws per decile:

Perturbed decileGSM8K (%)
1 (least important)82.76 ± 0.97
10 (most important)79.56 ± 0.59

The same perturbation costs 3.21 more GSM8K points in the top decile than in the bottom one. The directions Phase 1 rescaled most are therefore the ones the skill is most sensitive to, which is what the importance measure assumes.

Section 4.2 shows that SVF-guided and raw top-k protection achieve identical reasoning performance despite selecting nearly disjoint directions. How should this observation be interpreted?

The spectral baselines separate the spectrum at a fixed rank and protect the leading side, which carries the largest share of the matrix's spectral energy. OPLoRA protects the k directions of largest singular value absolutely. Table 2 tests whether that set preserves the skill best. The SVF set is chosen by measured skill relevance instead, and overlaps the leading set at a micro Jaccard index of 0.012. The two preserve the skill equally, at 81.12 GSM8K, while random-k reaches 79.61 at comparable recall. Protecting the directions of largest energy is therefore not necessarily the best protection of the skill. If no set of 64 directions is privileged, the protection should follow the importance profile over the whole spectrum rather than any cut of it, which is the graded rule of Eq. 3. We hope this clarifies how we read the observation, and we will rewrite the paragraph of Section 4.2 in these terms.

AuthorsOpenReview ↗

Part 2 of Rebuttal

Could the authors further compare PALoRA with uniform or shuffled graded weights under matched conditions to distinguish task-specific protection from general regularization effects?

We replaced only the protection weights and left the injection run otherwise identical.

Protection weightsmean wλRecall (%)GSM8K (%)
none (SVF + LoRA)—062.078.80
uniform, w = 11.003e-561.681.50
uniform, w = 11.001e-460.080.59
uniform, w = 11.003e-455.680.59
shuffled SVF w0.343e-459.881.03
SVF w (PALoRA)0.343e-461.183.04

Two remarks on how the rows are matched. Setting w = 1 nearly triples the mean weight, so the same λ is not the same penalty strength, and we therefore report uniform weights at three values of λ rather than one. Permuting the weights leaves the mean unchanged, so the shuffled row is matched to PALoRA in λ, in penalty strength and hence in update size, and differs only in which direction carries which weight. It is pooled over two permutations and three seeds, at 81.03 ± 0.88 GSM8K and recall 59.8 ± 2.4.

The two effects separate as follows. Both controls lift skill above the 78.80 of unconstrained injection, so limiting the size of the update is worth about two points on its own, which is the general regularization effect the reviewer names. The SVF profile adds a further 1.5 to 2.0 points on top of it: 83.04 against 80.59 for uniform at PALoRA's own λ and 81.03 for shuffled, and against 81.50 for the best uniform row, which reaches it at recall 61.6 against PALoRA's 61.1. We note that tuning λ does not close this gap in either direction, which supports our reading that λ sets how large the update is while the weights set where it goes.

How are updates in directions outside the span of these singular vectors handled, particularly for rectangular MLP matrices? Could the authors quantify whether such unprotected updates affect skill retention?

Of the three adapted matrices, only one has input directions outside that span. gate_proj and up_proj map 4096 to 14336, so their 4096 right singular vectors span the input space and the penalty reaches every direction. down_proj maps 14336 to 4096, leaving 10240 directions outside the span. These are its null space: the pretrained matrix maps them to zero, and the expert rescales singular values, which exist only inside the span, so there is no skill importance to assign outside it.

Theorem 2 completes the basis with w_i = 0 there, so nothing in the complement is charged. Table 4(a) measures what this leaves open. The complement is unconstrained in both arms of that ablation: rows B and D differ only in the graded penalty, which acts on the span alone, and it recovers 4.24 GSM8K points, from 78.80 to 83.04 against the deployed expert's 83.85. Retention is therefore governed by the directions the penalty weights, and constraining those is enough to recover it with the complement left free throughout. We will state this in Appendix B.3.

In Figure 3(b), why does PALoRA's weighted penalty initially increase before decreasing? Could the authors clarify the mechanism behind this behavior and whether it is associated with changes in reasoning performance during training?

The penalty is quadratic in the update, L_ortho = ‖BAV diag(w)‖²_F, and B is zero at initialization, so both the penalty and its gradient start at zero while the SFT gradient does not. The first epochs are therefore governed by acquisition, and the penalty rises with ‖ΔW‖. Once the update is large enough for λ∇L_ortho to be comparable to the SFT gradient, and acquisition begins to saturate, the penalty term redistributes the update across the spectrum and pulls the penalty down, ending 6.6 times below the unconstrained run.

On the second question, what the descent does is redistribute rather than shrink. Figure 3(a) shows the end state: 1.6% of the update energy in the top importance decile against 24.4% in the bottom one, where unconstrained LoRA holds about 10% in each. Acquisition is not impeded, the SFT loss on the facts ending at 0.357 against 0.356, while skill accuracy ends 4.24 points higher.

AuthorsOpenReview ↗

Part 3 of Rebuttal

Given the additional SVF training and regularization costs, could the authors provide a more comprehensive comparison of skill retention, factual recall, and end-to-end computational cost against OPLoRA?

We bring the three axes together below. Each row averages the 24 configurations of Table 3 at one seed, so the two methods are compared on the same runs.

SeedPALoRA Acc. (%)OPLoRA Acc. (%)PALoRA Rec. (%)OPLoRA Rec. (%)
4269.7369.3171.772.8
123469.6169.3371.473.5
202469.0868.9072.372.1
mean69.4769.1871.872.8

PALoRA retains more skill at every seed, while OPLoRA recalls more at two of the three. The ACC column of Table 3 averages the two, so the gap there is one of recall rather than of retention, and PALoRA has the only positive backward transfer in the comparison, +0.3 against 0.0. End to end a run costs 1153 s against 984 s.

To compare retention at matched recall we swept OPLoRA's protected subspace in the cell of Table 4, with three seeds at each k.

MethodRecall (%)GSM8K (%)
OPLoRA, k = 6463.4 ± 2.281.68 ± 0.61
OPLoRA, k = 12862.1 ± 3.081.65 ± 1.45
OPLoRA, k = 25661.1 ± 0.981.98 ± 1.26
OPLoRA, k = 51261.7 ± 2.781.68 ± 1.36
PALoRA61.1 ± 0.883.04 ± 1.01

Skill accuracy moves within 0.4 points over the sweep and no k reaches PALoRA. At k = 256 the two recall equally, and PALoRA holds 1.06 points more skill. We wish to point out that this follows from the paper's analysis. A rank-fixed set leaves the skill loading outside it exposed, however strictly enforced (Corollary 1). Theorem 1 places that loading across the whole spectrum, so enlarging the set removes only a small part of it.

Under sequential injection (Table 5), where the updates accumulate, PALoRA ends round 4 at 79.7 GSM8K against 76.5 and stays above the base model at every round, while OPLoRA ends 2.1 points below it and recalls more over the rounds, by 4.5 points.

The cost difference is confined to the injection stage, since Phase 1 is common to both and PALoRA adds no preprocessing of its own. PALoRA therefore trades about one point of recall for retention that holds at matched recall and widens as updates accumulate, at 1.2 times the cost.

Figures 2(a) and 2(b) in the main paper are difficult to read.

We agree, and we will enlarge both figures for the revision.

We hope these results address the concerns raised; if they do, we would be grateful if you would consider revisiting your assessment, and we remain glad to run any further control you consider decisive during the discussion period.

Reviewer_gwNAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes PALoRA, a two-stage adaptation method to update the factual information of LLM models while preserving the existing reasoning capabilities. The paper claims that the reasoning information spreads across the singular spectrum of the model weight metrics rather than only towards the dominant directions. The paper experments on mutiple becnhmakrs i.e LLama-3.1-8B and Mistral-7B using GSM8K,, MBPP, and AI2-ARC for reasoning evaluation.

Critical strengths

PALoRA proposes a solution for practical challenges of updating the factual knowledge for large models. PALoRA uses SVF experts to identify the relevant components, which reduces interference during knowledge injection. The paper claims ~5-10% additional overhead for subsequent knowledge injection, which is quite impressive.

Critical weaknesses

PALoRA mostly uses all the existing technology, i.e SFVT, LoRA or orthogonality-based regularisation. So overall algorithmic novelty is limited; PALoRA just positions these methods efficiently in this paper. The paper conducts experiments with a limited set of factual knowledge injection from 100 to 1k facts, but the real-world injection happens on much larger updates. So the paper may need more ablation on this.

Questions for authors
  1. PALoRA primarily considers individual reasoning skills. How would PALoRA perform when we combine multiple reasoning skills, i.e., mathematical reasoning, coding and scientific reasoning?

  2. Can the author report the overall training cost, including the SVF expert construction, SV decomposition, and constrained LoRA adaptation?

3 comments

We thank the reviewer for the review and for the two questions. We are glad that the reviewer finds the problem practically important and the overhead of subsequent knowledge injection low. The following are our responses to the comments and questions.

PALoRA mostly uses all the existing technology, i.e SFVT, LoRA or orthogonality-based regularisation. So overall algorithmic novelty is limited; PALoRA just positions these methods efficiently in this paper.

The novelty lies in the premise the paper tests and in the protection rule that follows from the outcome. The spectral methods rely on the leading directions carrying the largest share of the matrix's spectral energy. They use that property to keep a model's existing abilities while its weights are updated. Each of them separates the spectrum at a fixed rank, and either places the update in those components or keeps it out (Table 1). Whether that cut-off decides retention, or which side of it does, is assumed in this literature rather than tested. We test it in the setting where it is used, injecting facts while preserving a reasoning skill on GSM8K, MBPP and AI2-ARC. Theorem 1 places skill loading outside the leading rank-k block. Table 2 then shows that preserving the top k is not necessarily the best option. An SVF-guided set of the same size preserves the skill equally, at a micro Jaccard index of 0.012 against the leading one. We wish to point out that this concerns the boundary PiSSA, MiLoRA, OPLoRA and CorDA all draw, and holds whether or not PALoRA is adopted. Corollary 1 states the case where that boundary is enforced as protection, which leaves the skill loading outside it exposed however strictly enforced. That outcome is what motivates graded protection, since if no cut of the spectrum is privileged the protection should follow measured relevance across the whole of it (Eq. 3). The relevance is measured by a short probe rather than assumed from spectral position. That measured signal is what the rank-fixed rules do not have. PALoRA is also the only method in Table 3 with positive backward transfer, +0.3 over 24 configurations.

The paper conducts experiments with a limited set of factual knowledge injection from 100 to 1k facts, but the real-world injection happens on much larger updates. So the paper may need more ablation on this.

We agree that larger injection scales are interesting. At much larger loads, however, adapter capacity and the optimization schedule govern the outcome as much as the protection rule does. This is therefore a separate problem rather than a larger instance of ours, and we will name it in the limitations of the revision.

PALoRA primarily considers individual reasoning skills. How would PALoRA perform when we combine multiple reasoning skills, i.e., mathematical reasoning, coding and scientific reasoning?

Combining several reasoning skills is named as future work in the conclusion, and Appendix D takes a first step in it. It gives the M-skill extension, taking the elementwise maximum of the per-skill protection weights over a mixture of the corresponding experts, and measures what the protection costs without any further training. The protected subspace grows sublinearly, since the skills share a large part of the directions they rely on: all three of GSM8K, MBPP and AI2-ARC occupy 3.71% of the spectrum, against the 4.69% a disjoint union would need. The plasticity budget is therefore eroded slowly as skills are added, which is what makes the direction worth pursuing.

Can the author report the overall training cost, including the SVF expert construction, SV decomposition, and constrained LoRA adaptation?

Appendix I.1 reports measured wall-clock time per stage on an A100-80GB for this cell. Phase 1 costs 353 s, covering the policy-gradient training of the expert and the decomposition it is written through. We wish to point out that it is common to every row, since all methods are deployed on the same expert, and that it is paid once per skill and amortized over every later injection run. PALoRA adds no preprocessing of its own, reusing the Phase-1 decomposition, against 208 s for PiSSA, 212 s for MiLoRA, 21 s for OPLoRA and 389 s for CorDA. Its injection stage runs at 800 s, against 631 s for LoRA, 582 s for PiSSA, 609 s for MiLoRA, 610 s for OPLoRA and 594 s for CorDA, so 1.3 to 1.4 times the baselines, and the difference comes from evaluating the regularizer rather than from a larger trainable parameterization. End to end a run is 1153 s against 984 s for LoRA, 1.2 times its cost.

We hope these results address the concerns raised; if they do, we would be grateful if you would consider revisiting your assessment, and we remain glad to run any further control you consider decisive during the discussion period.

I really appreciate the authors' responses. Given the limited scope of PALoRA, with limited knowledge injection up to 1k facts and applying only individual skill reasoning, I believe the paper will benefit from substantial revision. Specifically, it needs more assessment of how PALoRA performs when combining multiple reasoning skills, i.e math, coding and scientific reasoning, as mentioned in the review comments.

Overall, I would like to keep my current rating.

We thank the reviewer for the follow-up. We would like to note that the scope of the paper is set by the question it asks: whether singular-value rank identifies the components to protect during parameter-efficient knowledge injection, the premise shared by the spectral methods of Table 1. We test this premise extensively. Table 3 compares six methods over three skills, two model families and four loads from 100 to 1000 facts, at three seeds each. The paper adds sequential injection, where the updates accumulate, ablations of each stage, of the protection weights, of λ and of OPLoRA's protected subspace, and the theory that predicts these results. Combining several skills is a separate question, which we name as future work and for which Appendix D takes a first step.

We thank the reviewer again for the engagement.

Reviewer_phx5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PALoRA, which protects reasoning skills during fact injection by penalizing LoRA updates along directions weighted by an SVF skill expert. The comparison is fair and the constraint helps. However, the specific value of the SVF weights and the advantage over OPLoRA at matched recall are not shown.

Critical strengths
  1. The problem matters, and the proposed fix can be tested. The method addresses the loss of reasoning ability when new facts are injected. It first trains an SVF skill expert. It then uses the change in singular-value scaling to estimate how important each direction is. Finally, it adds a weighted penalty on the LoRA update (Eqs. 1–3). The new idea to evaluate is using SVF changes as protection weights. SVF, LoRA, and importance-based regularization are all existing tools.
  2. The main comparison is fair with respect to the expert. All baselines in the main table start from the same SVF expert, so gains from the extra skill training are not credited only to PALoRA. The ablation in Table 4 separates the effect of the expert from the effect of the constraint. With the same expert, adding the constraint raises GSM8K from 78.80 to 83.04, while knowledge recall drops from 62.0 to 61.1. The net gain from the constraint is clear in this setting.
  3. The paper adds a useful check on fixed singular-value ranking. In Table 2, the directions chosen by SVF and by plain top-k hardly overlap, but both give the same GSM8K retention, and both beat random selection. This suggests that different sets of directions can each protect the skill, which motivates continuous weighting. It does not show exactly where the skill is stored.
  4. The theory and experiments are broad, and limitations are stated openly. The paper links weighted protection to a first-order bound on skill loss, and discusses its assumptions about curvature, training stage, and gradient structure. It tests two model families, three skill domains, different numbers of injected facts, and sequential injection. It also separates results relative to the base model from results relative to the SVF expert, and reports per-item standard deviations. It also says clearly that the multi-skill appendix only analyzes subspaces, and that full multi-skill retention is left for future work.
Critical weaknesses
  1. It is not shown that the SVF weights themselves are what matters. The shared-expert ablation shows that the constraint helps. But it does not separate three possible causes: the specific information in the SVF weights, simply limiting the size of the update, or the strength of the penalty. Simple controls, such as uniform weights or shuffled SVF weights, would show more directly whether the importance estimate is needed.
  2. It is unclear whether PALoRA keeps more skill at the same knowledge recall. In the main table, PALoRA averages 70.7 in ACC and 71.8 in knowledge recall, while OPLoRA gets 71.0 and 72.8. In the last round of sequential injection, PALoRA gets 79.7 in skill and 47.4 in recall, while OPLoRA gets 76.5 and 51.9. The authors report this trade-off openly, and describe PALoRA as close to OPLoRA, with better skill retention in some settings. What is missing is a comparison at matched recall. OPLoRA uses a fixed k = 128 as in the original paper. PALoRA has an ablation over protection strength. But the paper does not show trade-off curves for both methods that can be compared.
  3. There is no experiment linking the theory to the short probe used in practice. Appendix B.10 lists several limitations: curvature is not estimated, early and late training behave differently, the bound is first-order, and an assumption is made about gradient structure. One more specific gap remains. The weights come from ten SVF updates. The paper does not check whether these weights are stable across seeds and short training lengths, or how this affects retention.
Questions for authors
  1. At similar knowledge recall or update size, do SVF weights beat uniform and shuffled weights in the shared-expert setup?
  2. Can you sweep OPLoRA's k in one main setting, and explain how PALoRA's λ is chosen on the validation set, so that skill retention can be compared at similar recall?
  3. How stable are the protection weights and the final retention across a few probe seeds or short training lengths? For Table 3, can you also report paired-seed differences between PALoRA and OPLoRA?
  4. For MBPP, which exact splits, sample counts, or IDs are used for expert training, tuning, and final evaluation, and how does the test set relate to them?
3 comments
AuthorsOpenReview ↗

Part 1 of Rebuttal

We thank the reviewer for the careful and detailed review. We are glad that the reviewer finds our main comparison fair with respect to the expert, the check on fixed singular-value ranking useful, and our limitations openly stated. The following are our responses to the comments and questions.

For this response we ran three further experiments, all in the configuration of Table 4, on Llama-3.1-8B-Instruct with the GSM8K expert and 1000 unknown facts: uniform and shuffled protection weights, OPLoRA's protected-subspace size at k = 64, 128, 256 and 512, and controlled perturbations of the deployed expert by importance decile. The paired-seed comparison re-analyses the existing runs behind Table 3. We will include all of them in the revision.

At similar knowledge recall or update size, do SVF weights beat uniform and shuffled weights in the shared-expert setup?

We replaced only the protection weights and left the injection run otherwise identical.

Protection weightsmean wλRecall (%)GSM8K (%)
none (SVF + LoRA)—062.078.80
uniform, w = 11.003e-561.681.50
uniform, w = 11.001e-460.080.59
uniform, w = 11.003e-455.680.59
shuffled SVF w0.343e-459.881.03
SVF w (PALoRA)0.343e-461.183.04

Two remarks on how these rows are matched. We wish to point out that setting w = 1 nearly triples the mean weight, so the same λ is not the same penalty strength, and we therefore report uniform weights at three values of λ rather than one. Permuting the weights, by contrast, leaves the mean unchanged, so the shuffled row is matched to PALoRA in λ, in penalty strength and hence in update size, and differs only in which direction carries which weight. It is pooled over two permutations and three seeds, at 81.03 ± 0.88 GSM8K and recall 59.8 ± 2.4.

The SVF profile leads both controls. At PALoRA's own λ, uniform reaches 80.59 and shuffled 81.03, while PALoRA reaches 83.04. At matched recall, uniform reaches 81.50 at recall 61.6, while PALoRA reaches 83.04 at 61.1. We note that tuning λ does not close this gap in either direction, which supports our reading that λ sets how large the update is while the weights set where it goes.

We also agree that the penalty itself accounts for part of the gain. Both controls lift skill above the 78.80 of unconstrained injection, so limiting the size of the update helps on its own, and only the remaining 1.5 to 2.0 points are attributable to the importance estimate. We will add this table and this clarification to the revision.

Can you sweep OPLoRA's k in one main setting, and explain how PALoRA's λ is chosen on the validation set, so that skill retention can be compared at similar recall?

We have run the sweep, in the same cell as above and with three seeds at each k.

kRecall (%)GSM8K (%)
6463.4 ± 2.281.68 ± 0.61
12862.1 ± 3.081.65 ± 1.45
25661.1 ± 0.981.98 ± 1.26
51261.7 ± 2.781.68 ± 1.36

The curve is flat in k. Recall falls by 2.3 points between k = 64 and k = 256, while skill accuracy moves within 0.4 points, and no k reaches PALoRA's 83.04. At k = 256, OPLoRA's recall equals PALoRA's 61.1, and PALoRA holds 1.06 points more skill there. We wish to point out that this follows from the paper's analysis. A rank-fixed set leaves the skill loading outside it exposed, however strictly enforced (Corollary 1). Theorem 1 places that loading across the whole spectrum, so enlarging the set removes only a small part of it.

On λ, we would like to clarify that it is a per-configuration hyperparameter rather than one value tuned once, and that we read it off the trade-off curve reported in Table 4(b). In this cell, GSM8K rises from 78.80 at λ = 0 to 83.04 at 3e-4, then flattens, at 82.1 for 1e-3 and 82.0 for 3e-3, while recall falls monotonically across the same range, from 62.0 to 55.7. We use the value at the knee of that curve, 3e-4, the smallest λ at which retention saturates. We will state this selection rule explicitly in the revision.

AuthorsOpenReview ↗

Part 2 of Rebuttal

For Table 3, can you also report paired-seed differences between PALoRA and OPLoRA?

All 24 configurations of Table 3 have both methods at all three seeds, so the difference can be taken seed by seed. The table below averages over the 24 configurations.

SeedPALoRA Rec.OPLoRA Rec.Δ Rec.PALoRA Acc.OPLoRA Acc.Δ Acc.
4271.772.8−1.0969.7369.31+0.43
123471.473.5−2.1769.6169.33+0.28
202472.372.1+0.2269.0868.90+0.18
mean71.872.8−1.0169.4769.18+0.29

Over the 72 paired comparisons, the mean difference is +0.29 in skill accuracy and −1.01 in recall, with standard errors of 0.25 and 0.53. We agree with the reviewer's reading of the main table. Pairing separates the two methods by less than the seed spread on either axis, and this is the comparison the paper makes: PALoRA matches OPLoRA while weighting a nearly disjoint set of directions, which we offer as evidence for Theorem 1 rather than as a claim to dominate OPLoRA. We would like to add that the two differ in what each can be tuned to do. As the sweep above shows, OPLoRA's retention varies little with the size of its protected subspace, whereas λ moves PALoRA's retention by 4.24 points from the unconstrained run (Table 4(a)).

How stable are the protection weights and the final retention across a few probe seeds or short training lengths?

We did not run several probe seeds. We ran ablations instead, since isolating what the profile contributes tells us more about the probe's impact than repeating it would.

Uniform and shuffled weights remove the information in the profile, and both land near 81 against 83.04 for the SVF profile, so an uninformative probe would cost about two points and still retain more skill than unconstrained injection at 78.80. We also ran a new experiment for this response, checking the probe's ranking directly. We split the directions into deciles by protection weight, then perturbed the expert one decile at a time with an untrained update matched in rank and size to a real PALoRA update, so only the location differs. Three random draws per decile:

Perturbed decileGSM8K (%)
1 (least important)82.76 ± 0.97
10 (most important)79.56 ± 0.59

The same perturbation costs 3.21 more GSM8K points in the top decile than in the bottom one, so the ranking predicts what a direction costs the skill.

On short training lengths, Proposition 1 (Appendix B.6) bounds the error in the protection weights by the noise in the probe, and the number of rollouts needed to estimate them grows only as log r (Section 3.3). A short probe is therefore sufficient for graded weights, where a subset rule would need a margin between relevant and irrelevant components.

For MBPP, which exact splits, sample counts, or IDs are used for expert training, tuning, and final evaluation, and how does the test set relate to them?

We use the MBPP+ release of EvalPlus, 399 problems in the release's fixed order, and report pass@1 on its base test suites.

  • Phase 1 uses only indices 0–299. A fixed shuffle (seed 16) assigns 200 of them to training and the remaining 100 to checkpoint selection, and every Phase-1 iteration trains on all 200.
  • Final evaluation uses indices 300–398, 99 problems, disjoint from both Phase-1 partitions.
  • Phase 2 never trains on MBPP.

Every MBPP number in the paper is therefore k/99, so a single problem is 1.01 points.

We hope these results address the concerns raised; if they do, we would be grateful if you would consider revisiting your assessment, and we remain glad to run any further control you consider decisive during the discussion period.

I thank the authors for their detailed responses and additional experiments, which have largely addressed my main concerns. I have raised my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the Revised Paper

We thank all reviewers again for their careful reviews and for the questions raised in the discussion. We have uploaded a revised version of the paper. All new or modified text is marked in blue, and the main text uses the additional page allowed for the revision. The following summarizes what has been added and updated, together with the reviewers whose questions each change addresses.

New experiments

All three were run in the configuration of Table 4, on Llama-3.1-8B-Instruct with the GSM8K expert and 1000 unknown facts.

  • Uniform and shuffled protection weights (Reviewers phx5, f6Fi). Table 4(a) adds rows (E) and (F), which keep the penalty but discard the skill relevance encoded in its weights, and a new paragraph of Section 4.4 discusses them. Both controls retain about two points more than unconstrained injection, since any penalty limits the size of the update, yet neither exceeds 81.50 GSM8K, against 83.04 for PALoRA at matched recall.
  • Protected-subspace size of OPLoRA (Reviewers phx5, f6Fi). Table 4(c) reports k = 64, 128, 256 and 512, and Section 4.3 states the result next to Corollary 1. Skill accuracy stays within 0.4 points over the sweep, and at k = 256, where the two methods recall equally, OPLoRA reaches 81.98 against 83.04.
  • Perturbation by importance decile (Reviewers phx5, f6Fi). Section 4.4 reports, next to Figure 3(a), that an untrained perturbation of the expert, matched in rank and norm to a PALoRA update, leaves 79.56 GSM8K in the top decile and 82.76 in the bottom one. The protocol of all three experiments is given in Appendix F.6.

New analysis of existing runs

  • Paired-seed comparison with OPLoRA (Reviewers phx5, f6Fi). Section 4.3 gives the seed-paired differences, +0.29 in skill accuracy and −1.01 in recall, and Appendix H reports them per seed together with their standard errors.

Updated text

  • Section 4.2 (Reviewer f6Fi). The paragraph on Table 2 is rewritten in the terms of our response. Top-k protection is tested as the best set of its size, and a nearly disjoint SVF-guided set that preserves the skill equally shows that protecting the directions of largest singular value is not necessarily the best protection of the skill.
  • Choice of λ (Reviewer phx5). Section 4.4 states that retention saturates at the knee of the trade-off in Table 4(b), so a stronger weight mostly costs recall.
  • Directions outside the span (Reviewer f6Fi). Appendix B.3 explains why only down_proj has input directions outside the span of its singular vectors, and how rows (B) and (D) of Table 4(a) measure retention with these directions left free.
  • Figure 3(b) (Reviewer f6Fi). The caption explains why the weighted penalty first rises under PALoRA as well.
  • Injection scale (Reviewer gwNA). The conclusion names larger injection loads as a limitation, since adapter capacity and the optimization schedule then govern the outcome as much as the protection rule.
  • Readability (Reviewer f6Fi). Figure 2 is enlarged, as are Figures 1 and 3.

All other results are unchanged, and the numbering of tables, figures and appendices is preserved, so the references in our individual responses remain valid. We thank the reviewers again for their time, and we remain glad to answer any further question during the discussion period.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.