- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper addresses a practical problem: how can a model that has gained reasoning ability through reinforcement learning retain that ability while learning new tasks? The authors propose NB-LoRA. It first records the inputs to each layer while the model solves reasoning problems, then identifies directions with weaker activation signals. Subsequent fine-tuning restricts updates to these directions to reduce interference with the original reasoning process. The authors evaluate Qwen2.5-1.5B and 3B on eight commonsense tasks and tool use. NB-LoRA achieves new-task performance close to standard LoRA while largely maintaining reasoning accuracy at its pre-fine-tuning level. The paper also evaluates other reasoning benchmarks, different parameter settings, and two-stage sequential learning.
Critical strengths
- The method strikes a good balance between learning new tasks and retaining existing abilities. In Table 2, the 3B model achieves 84.6% average new-task accuracy, close to the 84.7% achieved by standard LoRA. Its reasoning accuracy remains at 63.2%, compared with 63.8% before fine-tuning, whereas standard LoRA drops to 13.9%. The 1.5B model shows similar results. This is the paper's most convincing contribution.
- The method is simple and offers efficiency benefits. NB-LoRA restricts update directions through its structure, avoiding full-gradient projection at every step. Figure 4 reports approximately 6.3× faster training than NSCL on the 3B model, with roughly half the peak GPU memory usage. The authors also examine cross-benchmark retention, anchor-set size, and sequential learning, providing a fairly broad evaluation.
Critical weaknesses
- The claim of exact preservation goes beyond what the implemented method guarantees. Section 3.3 uses an approximate null space, but Eq. (5) states an exact equality to zero. The selected directions still contain some activation signal, so updates generally have a nonzero effect on the original inputs. Large updates could amplify this effect. Updating several layers at once also changes the inputs to later layers. The authors should distinguish the ideal mathematical result from the approximate behavior of the actual algorithm. Appendix B addresses optimization convergence, which does not directly establish that reasoning ability will be preserved. Its gradient-descent conditions also differ from the Adam optimizer used in the experiments.
- Some experimental details and conclusions need further explanation. Appendix D mainly describes NB-LoRA's settings and gives limited information about baseline tuning, replay ratios, and SDFT generation settings. Table 3 does not identify the downstream adaptation task either. In Table 2, replay scores 0% on tool use for the 1.5B model, while SDFT achieves only 11.9% average new-task accuracy. These low scores need explanation before the fairness of the comparison can be assessed. The threshold results are also not stable across all tasks. In Table 6, the 3B model's reasoning accuracy after HellaSwag adaptation is 53.68% at threshold 0.90, about 10 percentage points below the original 63.8%. Looking only at averages overlooks this substantial drop.
Questions for authors
-
Approximate constraints: How should Eq. (5) be stated for the actual 99% energy threshold? Could an error analysis or direct measurements show how much each layer's output changes after training? This would help assess the method's protective effect.
-
Experimental comparisons: Which downstream adaptation task was used in Table 3? Please provide the training settings and evaluation rules for the main baselines, and explain replay's zero tool-use score and SDFT's low scores. In particular, could response formatting or answer extraction have affected these results?
-
Stability: How do the authors explain the substantial HellaSwag drop at threshold 0.90 in Table 6? Should the claim of threshold insensitivity be qualified? Variation across tasks in Table 2 also does not capture variation across repeated training runs. If results from different random seeds are available, please report them.
-
Costs: Does Figure 4 include the preparation costs of generating reasoning traces and computing the null-space bases? How much extra time and GPU memory does NB-LoRA require compared with standard LoRA under the same settings? This information would help readers assess its practical cost.
3 comments
review
Thank you for clarifying what the 99% spectral-energy threshold guarantees and for agreeing to revise Eq. (5). The comparison with vanilla LoRA remains encouraging: on the 3B model, NB-LoRA retains 63.2% reasoning accuracy versus 13.9% for LoRA, with nearly identical adaptation accuracy. My remaining concern is whether the comparisons with replay and SDFT are reliable. Could you report what you find when checking the raw outputs, especially how many of replay’s 62 tool calls were scored incorrectly because of formatting or parsing, and whether rescoring changes the comparison? It would also help to identify the adaptation task and checkpoint used in Table 3. I am keeping my rating at 2 for now and would reconsider it in light of those checks.
Thank you for the follow-up. We checked the saved tool-use outputs for both model sizes.
Replay. All 62 responses on 1.5B contain additional calls, but 44 begin with the exact correct tool and arguments. The parser extracts these correctly; they fail because the evaluator compares the entire sequence against one gold call. The same first-call diagnostic gives:
| Model | Method | Full-sequence exact | First-call exact |
|---|---|---|---|
| 1.5B | Replay | 0/62 (0.0%) | 44/62 (71.0%) |
| 1.5B | NB-LoRA | 39/62 (62.9%) | 39/62 (62.9%) |
| 1.5B | Vanilla LoRA | 41/62 (66.1%) | 41/62 (66.1%) |
| 1.5B | SDFT | 10/62 (16.1%) | 12/62 (19.4%) |
| 3B | Replay | 47/62 (75.8%) | 49/62 (79.0%) |
| 3B | NB-LoRA | 49/62 (79.0%) | 49/62 (79.0%) |
| 3B | Vanilla LoRA | 48/62 (77.4%) | 48/62 (77.4%) |
| 3B | SDFT | 40/62 (64.5%) | 44/62 (71.0%) |
First-call scoring places replay above NB-LoRA on 1.5B and ties NB-LoRA on 3B; NB-LoRA scores higher on 3B full-sequence compliance. We will report this distinction while retaining extra calls as a protocol failure.
SDFT. On 1.5B, 44 responses yield no extracted call, including some with correct tool names and arguments in prose. Formatting affects its score, although these tool-use files cannot explain the 11.9% average across all adaptation tasks.
Table 3. The downstream adaptation task is ARC-Challenge. We evaluate the resulting merged checkpoints on the five reasoning benchmarks to assess retention after adaptation. The reasoning anchor corpora—MATH-lighteval for 3B and GSM8K for 1.5B—are used to construct the preservation subspaces; they are distinct from the downstream adaptation task. We will clarify this protocol and specify the adaptation task and checkpoint identifiers in the revised caption and experimental details.
We thank the reviewer for their careful feedback.
W1 & Q1. Approximate preservation
Threshold-based approximate subspaces are commonly used in projection-based continual learning to balance preservation and adaptation, including GPM [R1] and Adam-NSCL [R2]. Following this common practice, we protect dominant activation directions using a 99% spectral-energy threshold and adapt in the remaining low-energy directions. This limits interference but does not guarantee unchanged outputs: large updates and activation changes propagated from earlier layers can still affect reasoning.
Our experiments support the strategy's practical effectiveness: across nine adaptation tasks on Qwen2.5-3B, NB-LoRA achieves 84.6% target accuracy and 63.2% retained reasoning accuracy, versus 84.7% and 13.9% for vanilla LoRA (post-RL reasoning reference: 63.8%). This demonstrates effective retention in the evaluated setting, without guaranteeing exact preservation. Appendix B concerns gradient-descent convergence, not Adam convergence or reasoning retention; subspace membership holds independently of the optimizer.
[R1] Saha et al. Gradient Projection Memory for Continual Learning. ICLR 2021.
[R2] Wang et al. Training Networks in Null Space of Feature Covariance for Continual Learning. CVPR 2021.
W2 & Q2. Table 3 and baseline reproducibility
We agree that Table 3 must identify its downstream adaptation task and checkpoint, which require verification from the original evaluation records. We will also report method-specific learning rates and tuning ranges, trainable modules, ranks, optimizers, training steps, replay buffer sizes/mixing ratios, and SDFT teacher prompts and generation settings.
Replay's 0% tool-use score on the 1.5B model corresponds to 0/62 correctly scored calls. This and SDFT's 11.9% average adaptation accuracy require checking raw outputs, parsing/extraction, truncation, and teacher-rollout quality. Aggregate scores alone cannot establish whether these reflect task errors or formatting failures.
W2 & Q3. Threshold sensitivity and run variation
We agree that threshold sensitivity is task-dependent. After HellaSwag adaptation on Qwen2.5-3B, reasoning accuracy falls from 63.80% to 53.68% at threshold 0.90, whereas 0.99 retains 63.54%. We will explicitly highlight this failure and remove the broad threshold-insensitivity claim. Accuracy need not vary monotonically with threshold because independently trained adapters can differ in update norms and optimization trajectories; the specific cause requires repeated runs and residual measurements.
Variation across tasks also does not measure run-to-run uncertainty or establish the significance of small method differences.
Q4. Preparation costs and LoRA overhead
Figure 4 compares training efficiency with full-weight NSCL, reporting 627.72 s/11.23 GiB for NB-LoRA versus 3,924.93 s/23.62 GiB for NSCL on 3B. The current implementation excludes rollout and basis preparation from its adaptation timer; the original benchmark boundaries still require verification. These results do not establish an end-to-end speedup or overhead relative to vanilla LoRA.
The full cost is Trollout+Tactivation/basis+Tadaptation, with stagewise peak memory reported separately. Bases can be reused for independent adaptations from the same checkpoint and protected corpus. Compared with vanilla LoRA, NB-LoRA reduces trainable parameters from mr+rn to mr+rk, but adds an n×k fixed basis and basis transformations, so a matched benchmark is needed.