acceptodds
Under review as a conference paper at ICLR 2027

OSD: Orthogonal Spectral Descent For Continual Learning In Large Language Models

Abstract

Continual fine-tuning enables large language models (LLMs) to adapt to evolving tasks and domains, but sequential adaptation often leads to catastrophic forgetting of previously acquired capabilities. Projection-based continual learning methods mitigate this interference by constraining updates to avoid directions associated with past tasks. However, these methods are typically built around Euclidean steepest-descent geometry, where update magnitude is controlled by the Frobenius norm. In parallel, recent matrix-aware optimizers such as Muon motivate a different view of neural-network optimization: for matrix-valued parameters, descent directions can instead be selected under a spectral-norm constraint. We introduce Orthogonal Spectral Descent (OSD), a practical continual-learning instantiation of spectral steepest descent that combines protected task subspaces with efficient Muon-compatible constrained updates. For selected matrix-valued parameters, OSD formulates each update as a spectral-norm-bounded descent problem subject to linear non-interference constraints derived from previously learned tasks. The resulting constrained problem admits a low-dimensional dual formulation that is approximated using a small number of warm-started dual updates together with Newton–Schulz matrix-sign iterations, avoiding an exact SVD at every optimizer step. We evaluate OSD across standard continual-learning benchmarks, TRACE, and domain-specific sequential fine-tuning settings using both encoder–decoder and decoder-only language models. Across the evaluated settings, OSD improves retention and overall continual-learning performance relative to sequential fine-tuning and competitive orthogonal-gradient baselines while maintaining strong adaptation to new tasks. These results suggest that combining protected task subspaces with spectral update geometry provides a practical way to improve the stability–plasticity tradeoff in continual LLM fine-tuning.

Primary area:
foundation or frontier models, including LLMs
Keywords:
continual learning, LLM post-training, orthogonal gradient descent, spectral norm, geometry-aware optimization

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.20 over 5 reviews, above 49% of the others.
Reviewer_623ROpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Orthogonal Spectral Descent (OSD), a continual learning method that combines subspace protection with spectral-norm-constrained optimization. The method derives a nuclear-norm dual formulation and uses Newton–Schulz iterations with dual corrections to approximate the constrained updates. Experiments on T5, LLaMA, and Qwen models demonstrate improvements over existing continual learning baselines.

Critical strengths
  1. The paper presents an interesting approach to continual learning by combining orthogonal subspace protection with spectral optimization. The motivation is clear, and the proposed formulation provides a principled way to control parameter updates while preserving previously learned knowledge.

  2. The theoretical derivation is generally sound, and the proposed optimization algorithm offers a practical way to approximate the spectral update without performing expensive SVD computations at every step.

  3. The experiments cover several continual learning benchmarks and different model architectures. OSD consistently outperforms OSFT in the reported settings, with noticeable improvements on the 4-task benchmark and Qwen2.5-3B. The ablation study also provides useful evidence for the effectiveness of combining spectral optimization with subspace protection.

Critical weaknesses
  1. The improvements over strong baselines are relatively modest on some benchmarks, particularly the 15-task benchmark and TRACE. Additional experiments on more recent language models would help strengthen the generality of the findings.

  2. The ablation study could be more comprehensive. In particular, comparing OSD with straightforward combinations of subspace projection and Muon would help clarify the specific contribution of the proposed dual optimization.

  3. The practical solver could be analyzed in more detail. Since the method uses approximate Newton–Schulz iterations and a small number of dual corrections, reporting the constraint residuals would help assess how closely the implemented updates satisfy the theoretical formulation.

Questions for authors
  1. How does OSD compare with simpler approaches that directly combine subspace projection and Muon optimization?

  2. Could the authors report the orthogonality constraint residuals of the practical solver and discuss the effect of the number of dual correction steps?

  3. Have the authors considered evaluating OSD on more recent LLMs to further demonstrate its generality?

1 comment
AuthorsOpenReview ↗

Response to Reviewer 623R

  1. How does OSD compare with simpler approaches that directly combine subspace projection and Muon optimization?

    Answer: The direct baseline applies subspace projection and Muon as separate operations. OSD uses dual corrections to incorporate subspace protection into the spectral update itself. In the updated Table 4, the combined baseline reaches 68.9±2.2%68.9 \pm 2.2\% average accuracy on Standard CL. OSD reaches 78.9±1.0%78.9 \pm 1.0\%, a gain of 10.010.0 percentage points, and takes 26.5426.54 minutes versus 27.8327.83 minutes for the combined baseline.

    Table 4. Component ablation on the Standard CL benchmark with T5-Large. AA is final average accuracy (mean ± standard deviation). Training time is the mean wall-clock time per task sequence.

    MethodAA (%)Training time (min)
    Muon-only50.72±5.3050.72 \pm 5.3025.12
    Frob-only76.44±1.0376.44 \pm 1.0330.59
    Frob-OGD74.84±2.4474.84 \pm 2.4430.98
    Subspace projection + Muon68.9±2.268.9 \pm 2.227.83
    OSD78.9±1.078.9 \pm 1.026.54
  2. Could the authors report the orthogonality constraint residuals of the practical solver and discuss the effect of the number of dual correction steps?

    Answer: We thank the reviewer. For each protected matrix update ΔW\Delta W, we measure the normalized constraint residual

    r(ΔW)=∥U⊤ΔWV∥Fmax⁡{∥ΔW∥F,ε},r(\Delta W) = \frac{\|U^\top \Delta W V\|_F}{\max\{\|\Delta W\|_F,\varepsilon\}},

    where ε>0\varepsilon > 0 prevents division by zero. This is the fraction of the update norm that lies in the protected subspace.

    Controlled example. On Reviewer zJqR's 2×22 \times 2 example (U=V=e1U = V = e_1), with five Newton–Schulz iterations, Frobenius normalization, and dual step size 0.10.1, a cold start (λ(0)=0\lambda^{(0)} = 0) with 0,1,2,10,20,0, 1, 2, 10, 20, and 5050 dual corrections gives residuals of 31.60%31.60\%, 30.43%30.43\%, 29.27%29.27\%, 19.43%19.43\%, 7.69%7.69\%, and 0.14%0.14\%. The iteration settles at λ≈0.63\lambda \approx 0.63 rather than at the exact dual optimum λ⋆=1\lambda^\star = 1: it converges to the multiplier at which the approximate matrix sign satisfies the constraint. Near-zero residual is therefore attainable even though, as discussed in our response to Reviewer zJqR, the exact polar factor at λ⋆\lambda^\star is infeasible.

    Effect of warm starting. The practical solver warm-starts the dual variable across optimizer steps, so corrections accumulate over training. With GG held fixed, one correction per step reaches 0.14%0.14\% residual after 50 optimizer steps, and two corrections per step after 25. With a drifting momentum matrix (β=0.9\beta = 0.9, i.i.d. Gaussian gradient noise of standard deviation 0.50.5 around GG), the steady-state median (95th percentile) residual is 34.0%34.0\% (41.2%41.2\%) with no corrections, 7.6%7.6\% (20.2%20.2\%) with one, and 3.6%3.6\% (11.1%11.1\%) with two. The effect of TinT_{\mathrm{in}} is thus governed by the dual correction rate relative to the drift of the momentum matrix. The finite solver provides no general guarantee of monotonic residual reduction or exact feasibility.

    Residuals during LLM training. On the Standard CL benchmark, we compare Tin∈{0,1,2,5}T_{\mathrm{in}} \in \{0,1,2,5\} while holding the Newton–Schulz iteration count and the training configuration fixed. With Tin=0T_{\mathrm{in}} = 0 and zero-initialized multipliers, the dual variable never moves, so this setting coincides with our Muon-only arm.

  3. Have the authors considered evaluating OSD on more recent LLMs to further demonstrate its generality?

    Answer: We thank the reviewer for this suggestion. To further evaluate OSD's generality, we conducted additional experiments on Qwen3.5-4B using the Standard CL benchmark. The table below reports the final accuracy on each task after completing the four-task continual learning stream, together with average accuracy (AA).

    Final task accuracies and average accuracy (AA) on the Standard CL benchmark using Qwen3.5-4B. All values are percentages.

    MethodDBpediaAmazonYahooAGNewsAA
    AdamW0.000.8025.4988.7428.76
    OSD73.1359.2858.7590.3270.37

    OSD improves AA from 28.76% to 70.37%, a gain of 41.61 percentage points, and achieves higher final accuracy on all four tasks. The substantial gains on DBpedia, Amazon, and Yahoo are consistent with stronger retention of earlier tasks. These additional results support OSD's applicability to a more recent LLM backbone, extending the evidence beyond the models evaluated in the original submission. We will include these results in the revised manuscript.

Reviewer_8stTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents Orthogonal Spectral Descent (OSD) for continual fine-tuning of language models. OSD combines linear constraints protecting previously learned subspaces with a spectral-norm bound on matrix updates, and the paper derives a nuclear-norm dual and approximately solves it using warm-started dual corrections and Newton–Schulz iterations. Empirical results on T5 continual-learning benchmarks, TRACE, and coding–math–medical sequences demonsrate better performance under qwen2.5 and Llama2 architectures.

Critical strengths
  1. This work is focusing on a important challenge via a meaningful optimization perspective. It is important to allow continual pre/mid/post-training for modern LLMs to incorporate new knowledge while reducing computational cost, and combing protection of previous subspaces with updating geometry provides a useful direction. The intervention operates in training process without introducing any inference-time module.

  2. The proposed update is principled in theory, and the evaluation spans several settings. The derived duality gives a clear mathematical basis for the update, and experiments include different architectures, task-stream lengths, and domains.

Critical weaknesses
  1. The explored architecture is outdated. Llama2 and qwen2.5 are quite a old in nowadays sense, leaving the performance of proposed optimization method on SoTA LLMs unknown.

  2. The evidence for an advantage on longer task streams is modest. For example, on 15-task CL benchmark, OSD achieves 72.0% comparing to OSFT’s 71.3%, rendering a rather marginal improvement. This observation, combined with old architecture further undermine the practical benefits of proposed optimization method.

  3. Connection between protected weight subspaces and retained behavior is also under-validated. The constraint U^{T} Δ WV = 0 still allows ΔWV != 0: representations can still change for inputs within the protected right-singular subspace. Moreover, the domain experiments constrain selected MLP matrices while updating other parameters with AdamW.

Questions for authors
  1. Could authors explain more clearly how the constraint U^{T} Δ WV = 0 helps preserve old-task behavior? Reviewer is also interested in the effect of updating the remaining parameters with AdamW. Do authors have measurements or ablations showing how much these unconstrained updates contribute to forgetting?

  2. How much retuning was needed when applying OSD to different backbones? In particular, did the protected rank and dual step size transfer across models? Any existing results on more recent model families would also help assess how broadly the benefits carry over.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 8stT

  1. Could authors explain more clearly how the constraint U⊤ΔWV=0U^\top \Delta W V = 0 helps preserve old-task behavior? The reviewer is also interested in the effect of updating the remaining parameters with AdamW. Do authors have measurements or ablations showing how much these unconstrained updates contribute to forgetting?

    Answer:

    How subspace protection helps preserve old-task behavior. Let VV and UU span the protected input and output directions of a weight matrix WW. For an input h=Vzh = Vz in the protected input subspace, the constraint implies

    = U^\top W h + U^\top \Delta W Vz = U^\top W h.$$ Thus, when the constraint is satisfied, the update preserves the part of the layer response that maps protected inputs to protected outputs. This reduces one source of interference with earlier tasks. Inputs can also have components outside $\operatorname{span}(V)$, and updates to other parameters can change the model's predictions. The practical solver enforces the constraint approximately. **Comparison with unconstrained updates.** Table 4 compares OSD with Muon-only on the Standard CL benchmark. Both use the same Newton–Schulz matrix-sign approximation, and Muon-only omits the protected-subspace constraint on the targeted matrices. OSD reaches $78.9 \pm 1.0\%$ final average accuracy, compared with $50.72 \pm 5.30\%$ for Muon-only, a difference of 28.2 percentage points. This comparison measures the benefit of protection in terms of final average accuracy, which reflects both learning new tasks and retaining earlier ones. **Effect of the remaining AdamW updates.** Table 13 varies the AdamW-branch learning rate while holding the Muon learning rate fixed. It shows that reducing the AdamW learning rate can impair learning on the current task. Our reported experiments do not isolate how much forgetting is attributable to the AdamW-updated parameters. A direct ablation would keep OSD active on the same targeted matrices and compare the standard AdamW branch with freezing those remaining parameters after the first task.
  2. How much retuning was needed when applying OSD to different backbones? In particular, did the protected rank and dual step size transfer across models? Any existing results on more recent model families would also help assess how broadly the benefits carry over.

    Answer: We thank the reviewer for this question. The protected rank kk and dual step size ηλ\eta_\lambda largely transferred across backbones in the Coding → Math → Medical curriculum. They were selected using Qwen2.5-1.5B-Instruct and reused without a separate hyperparameter search for each model. We used two configurations:

    • Qwen2.5 family. The stage-wise default used k=10k=10 for the first task to protect the pretrained weights' leading singular directions, followed by k=3k=3 for the remaining tasks. The dual step sizes were 10−510^{-5}, 10−410^{-4}, and 10−510^{-5} across the three stages. The exception was Qwen2.5-3B-Instruct, which used ηλ=10−4\eta_\lambda=10^{-4} in stage C.

    • LLaMA-3.2-1B and LLaMA-3.2-3B. Both models used k=3k=3 and ηλ=5×10−5\eta_\lambda=5\times10^{-5} at every stage, unchanged from the settings selected in the one-parameter sweeps on Qwen2.5-1.5B-Instruct in Figure 5.

    These sweeps indicate limited sensitivity within the tested ranges. On Qwen2.5-1.5B-Instruct, varying kk over {1,3,5,10,20}\{1,3,5,10,20\} changed the reported score by at most 1.51.5 points, while varying ηλ\eta_\lambda from 10−610^{-6} to 10−410^{-4} changed it by at most 1.61.6 points. This supports transferring the protection settings without an extensive model-specific search. Most backbone-specific calibration instead concerned the learning rates of the Muon and AdamW branches. The T5 and TRACE settings in Table 5 were tuned separately for their respective benchmarks.

    To assess applicability to a more recent model family, we also evaluated OSD on Qwen3.5-4B using the Standard CL benchmark. OSD achieved 70.37% final average accuracy, compared with 28.76% for AdamW, an improvement of 41.61 percentage points, with higher final accuracy on all four tasks. These results provide additional evidence that OSD's benefits extend to a more recent backbone.

    We will add a per-backbone hyperparameter table documenting the transferred settings and exceptions, and revise the “optimal peak” wording in Appendices F.2–F.3 to reflect the limited sensitivity observed over the tested ranges.

Reviewer_xEnHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OSD combines protected task subspaces with spectral-norm steepest descent using low-dimensional dual corrections and Newton–Schulz approximations. Evaluations cover T5 continual classification, TRACE, and coding→math→medical sequences, with Muon-only and Frobenius controls probing update geometry.

Critical strengths
  1. The joint constrained formulation separates protected directions from update geometry, and Proposition 1 specifies compatible nuclear-norm subgradients for primal recovery (pp. 4–5), providing a useful methodological framework.
  2. Across standard four-task, 15-task, and TRACE benchmarks, reported AA exceeds OSFT by 3.0, 0.7, and 1.0 percentage points, respectively (Tables 1–2). Table 4 adds useful Muon-only and Frobenius component comparisons.
Critical weaknesses
  1. Algorithm 1 uses one or two dual corrections without reporting constraint residuals, leaving comparable protection precision against Frob-OGD unverified. Layerwise energy matching (§5.2) and matrix learning-rate tuning (Table 12) lack a unified update formula, so Table 4 may conflate geometry with protection strength or scale; its caption also conflicts with Appendix F over the task stream.
  2. The medical protocol trains on medical-o1-reasoning-SFT and evaluates 1,000 medical-o1-verifiable-problem questions (§C.3; §E.1–E.2). The cited HuatuoGPT-o1 paper (§3.1) derives SFT data from this question pool; without question-level overlap accounting, medical scores and corresponding AA in Tables 3 and 7–9 face an unresolved independence risk, not demonstrated leakage.
Questions for authors
  1. Under the main configurations, what normalized residual ||UᵀΔV||F/||Δ||F does OSD attain relative to Frob-OGD?
  2. Which task stream produced Table 4, and what complete Frobenius update formula reconciles learning-rate tuning with energy matching?
  3. After aligning original questions, how many of the 1,000 medical evaluation items overlap training, and were overlaps excluded?
2 comments
AuthorsOpenReview ↗

Response to Reviewer xEnH

Q1. Under the main configurations, what normalized residual does OSD attain relative to Frob-OGD?

Answer: We thank the reviewer to ask this question to help us further strengthen our paper. We measure the protection residual on the matrix optimizer update:

ρ=∥U⊤ΔV∥F∥Δ∥F.\rho=\frac{\Vert U^\top\Delta V\Vert_F}{\Vert\Delta\Vert_F}.

For orthonormal protected bases, Frob-OGD applies the projector Π(X)=X−UU⊤XVV⊤\Pi(X)=X-UU^\top XVV^\top. Its nonzero projected updates have zero residual in exact arithmetic. OSD uses a finite number of warm-started dual corrections, so the resulting update can have a nonzero residual.

We measured it on controlled streams using the decoder-only update rule with a full rank-3 dual block, EMA momentum with β=0.95\beta=0.95, warm starts, and four Newton–Schulz iterations. Each instance consists of 300 momentum matrices with spectral norms approximately one. We report the median across six instances per family after averaging over the final 100 steps of each instance.

UpdateGenericLLM-likeRank-deficient
OSD, 1 correction, dual step 0.050.013 (94%)0.025 (97%)0.021 (95%)
OSD, 1 correction, dual step 0.010.024 (89%)0.062 (92%)0.042 (90%)
OSD, 2 corrections, dual step 0.050.010 (95%)0.014 (98%)0.015 (97%)
OSD, 5 corrections, dual step 0.050.007 (97%)0.004 (99%)0.009 (98%)
OSD + final projection000
Frob-OGD000
Muon-only0.210.740.44

Parentheses give the percentage reduction in normalized residual relative to Muon-only, calculated from the displayed values. A zero denotes a residual below 1e−14.

With one correction and a dual step of 0.05, OSD attains residuals of 0.013–0.025, a 94–97% reduction relative to Muon-only. With five corrections, as in the correction count listed in Table 6, the residuals fall to 0.004–0.009 on these controlled streams. Frob-OGD and OSD with a final projection both reach the numerical floor. We will add this diagnostic and the final-projection variant to the appendix.

Q2. Which task stream produced Table 4, and what complete Frobenius update formula reconciles learning-rate tuning with energy matching?

Answer: We thank the reviewer for asking us to clarify this point.. Table 4 reports the T5-Large component ablation on the four-task Standard CL benchmark: DBpedia, Amazon Reviews, Yahoo Answers, and AG News. We used the Standard CL task orders in Table 5. Each run used one H100 GPU, one epoch per task, a microbatch size of 8, and 8 gradient accumulation steps, for an effective batch size of 64.

Let Gt,ℓG_{t,\ell} be the matrix direction supplied to the optimizer at step tt in layer ℓ\ell. Frob-only uses that direction directly, while Frob-OGD projects it using the protected bases UℓU_\ell and VℓV_\ell:

\qquad D_{t,\ell}^{\mathrm{Frob\text{-}OGD}} =G_{t,\ell}-U_\ell(U_\ell^\top G_{t,\ell}V_\ell)V_\ell^\top.$$ For Table 4, the norm-matched update for a nonzero direction is $$\Delta W_{t,\ell}^{\mathrm{opt}} =-r_{t,\ell}\frac{D_{t,\ell}}{\Vert D_{t,\ell}\Vert_F}, \qquad r_{t,\ell}=\Vert\Delta W_{t,\ell}^{\mathrm{spectral}}\Vert_F.$$ Here $r_{t,\ell}$ is the update norm of the corresponding spectral control at the same step and layer. With a fixed reference norm, any positive scalar applied to the Frobenius direction before normalization cancels. Table 12 sweeps the matrix learning rate $\alpha$ using the unnormalized optimizer update $\Delta W_{t,\ell}^{\mathrm{opt}}=-\alpha D_{t,\ell}$. It characterizes learning-rate sensitivity for the unnormalized controls, while the spectral reference sets the update magnitude in Table 4. We will clarify this distinction in Appendix G.1 and correct the benchmark reference in Appendix F.
AuthorsOpenReview ↗

Response to Reviewer xEnH Q3

Q3. After aligning original questions, how many of the 1,000 medical evaluation items overlap training, and were overlaps excluded?

Answer:

We thank the reviewer for raising this point. We matched the original question texts of the 1,000 HuatuoGPT-o1 evaluation questions sampled with seed 42 against the English-language SFT training set. We found that 321 of the 1,000 questions (32.1%) also appeared in the training set.

To assess the effect of overlap removal, we rescored the saved predictions from a 500-question evaluation of Qwen2.5-1.5B-Instruct. We excluded 155 exact matches and 17 reworded versions of training questions, leaving 328 questions. We also updated the evaluation code to skip overlapping items by default.

Stage C medical accuracy (%) averaged over seeds 40, 41, and 42

MethodBefore exclusionAfter exclusion
MuonSFT20.219.8
SeqSFT22.019.9
OSD23.422.3

Excluding overlaps reduced Stage C accuracy by 0.4–2.1 percentage points across the three methods. OSD retained the highest accuracy on the filtered set.

Reviewer_dfdXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Orthogonal Spectral Descent (OSD), combining Muon-style spectral optimization with protected-subspace constraints to reduce catastrophic forgetting in continual learning. Experiments across several LLMs and benchmarks show improvements over existing methods.

Critical strengths

The combination of spectral optimization and protected-subspace learning is well motivated.

The paper evaluates multiple architectures, model sizes, and continual-learning benchmarks.

The ablation studies provide useful insights into the effects of different update geometries.

Critical weaknesses

Limited novelty. The method mainly combines existing ideas from Muon and OSFT. The protected-subspace construction and spectral optimization are both established, making the methodological contribution somewhat incremental.

Modest improvements over strong baselines. OSD improves over OSFT by only 0.7% on the 15-task benchmark and 1.0% on TRACE. It also performs worse than ProgPrompt on the 15-task benchmark.

Unclear source of improvement. OSD already improves accuracy on the first task, where catastrophic forgetting is not an issue. This raises the question of whether the gains come from better optimization rather than improved knowledge retention. The ablation results also do not clearly explain why combining the two components works better.

Limited efficiency analysis. The reported speedup is interesting, but broader runtime and memory comparisons across model scales are missing.

Questions for authors

How does OSD compare with a straightforward combination of Muon and OSFT under identical settings?

Can the authors separate improvements in new-task learning from reductions in catastrophic forgetting?

1 comment
AuthorsOpenReview ↗

Response to Reviewer dfdX

Question 1. How does OSD compare with a straightforward combination of Muon and OSFT under identical settings?

Answer

We thank the reviewer to propose this question. Let Πs\Pi_s denote the OSFT projection at stage ss and MK\mathcal{M}_K the KK-step Muon matrix-sign map. We evaluated both straightforward compositions:

  • Projection then Muon — MK(Πs(Mt))\mathcal{M}_K(\Pi_s(M_t)), reported as “Subspace projection + Muon” in updated Table 4.
  • Muon then projection — Πs(MK(Mt))\Pi_s(\mathcal{M}_K(M_t)), reported as “Muon + OSFT” in updated Table 4.

OSD incorporates the protection constraint into the spectral update by applying a dual correction to the momentum matrix before computing the final matrix-sign direction.

All methods used the same T5-Large initialization, three Standard CL task orders, seeds, minibatches, one epoch per task, effective batch size of 6464, protected bases and ranks, targeted layers, gradient clipping, weight decay, and training-step budget. The final update was normalized separately for each protected layer to match OSD's Frobenius norm.

Muon + OSFT obtained 72.18±1.02%72.18 \pm 1.02\% final average accuracy (AA), and OSD obtained 78.9±1.0%78.9 \pm 1.0\%, a gain of 6.726.72 percentage points. Mean end-to-end wall-clock time per four-task sequence was 29.5729.57 minutes for Muon + OSFT and 26.5426.54 minutes for OSD, a reduction of 3.033.03 minutes (10.2%10.2\%). OSD achieved higher final AA and lower measured runtime than both sequential compositions on this benchmark.

Table 4. Component ablation on the Standard CL benchmark with T5-Large. AA is final average accuracy (mean ± standard deviation). Training time is mean wall-clock time per four-task sequence.

MethodAA (%)Training time (min)
Muon-only50.72 ± 5.3025.12
Frob-only76.44 ± 1.0330.59
Frob-OGD74.84 ± 2.4430.98
Subspace projection + Muon68.9 ± 2.227.83
Muon + OSFT72.18 ± 1.0229.57
OSD78.9 ± 1.026.54

Question 2. Can the authors separate improvements in new-task learning from reductions in catastrophic forgetting?

Answer

We thank the reviewer to ask this question for us to have a better comparison. Let at,ia_{t,i} be the test score on task ii after training through task tt, and let TT be the number of tasks. We define immediate new-task average accuracy and backward transfer as

\qquad \mathrm{BT}=\frac{1}{T-1}\sum_{i=1}^{T-1}(a_{T,i}-a_{i,i}).$$ Final average accuracy then decomposes exactly as $$\mathrm{AA}_{\mathrm{final}} =\mathrm{AA}_{\mathrm{new}}+\frac{T-1}{T}\mathrm{BT}.$$ **TRACE (eight tasks, Table 2).** OSD improves final AA by **1.0 percentage point** over OSFT (49.4 vs. 48.4). Its BT is 2.4 points higher (−4.7 vs. −7.1), contributing 2.1 points (7/8 × 2.4) to the final-AA difference. OSD’s immediate new-task average is 53.5 (Table 17), and OSFT’s is approximately 54.6, inferred from its reported final AA and BT. The smaller net decline on earlier tasks offsets OSD’s 1.1-point lower immediate new-task average. **Qwen2.5-3B-Instruct (Coding → Math → Medical, Table 3).** The immediate new-task averages are $$\mathrm{AA}_{\mathrm{new}}^{\mathrm{OSD}} =\frac{32.9+64.6+28.8}{3}=42.1, \qquad \mathrm{AA}_{\mathrm{new}}^{\mathrm{OSFT}} =\frac{32.7+62.6+27.6}{3}\approx41.0.$$ The corresponding BT values are $$\mathrm{BT}^{\mathrm{OSD}} =\frac{(32.9-32.9)+(69.8-64.6)}{2}=2.6, \qquad \mathrm{BT}^{\mathrm{OSFT}} =\frac{(32.1-32.7)+(63.0-62.6)}{2}=-0.1.$$ Of OSD’s **2.9-point final-AA gain** (43.8 vs. 40.9), approximately **1.1 points** come from higher immediate new-task scores and **1.8 points** from better net performance on earlier tasks. The latter includes positive backward transfer to Math, where OSD’s score rises from 64.6 to 69.8 after Medical training. BT therefore captures both losses and gains on earlier tasks.
Reviewer_zJqROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Orthogonal Spectral Descent for continual fine-tuning of large language models. The method combines subspace constraints derived from previous tasks with spectral-norm-bounded updates. Instead of using the Euclidean geometry common in projection-based methods, it formulates the constrained update as a nuclear-norm dual problem. The practical solver uses warm-started dual corrections and Newton–Schulz iterations, while keeping the singular-vector bases fixed within each training stage. The experiments include T5 classification streams, TRACE with LLaMA-2-7B-Chat, and Coding–Math–Medical sequences using Qwen and LLaMA models. The reported average accuracy increases from 75.9 to 78.9 over OSFT on Standard CL, from 71.3 to 72.0 on the 15-task benchmark, and from 48.4 to 49.4 on TRACE. The authors also conduct a component study to examine the effect of spectral update geometry, although some inconsistencies in the experimental description make these results difficult to interpret.

Critical strengths
  1. An interesting approach to constrained updates in continual learning. The paper makes a useful distinction between estimating protected subspaces and choosing the geometry of subsequent updates. Exploring spectral steepest descent under linear protection constraints is a reasonable direction. I also appreciate that the dual formulation makes the compatible-subgradient requirement explicit, rather than assuming that a dual solution automatically yields a feasible primal update.
  2. Evaluation across several continual learning settings. The evaluation includes classification streams, TRACE, and coding, mathematics, and medical adaptation, with both encoder–decoder and decoder-only architectures. The improvement over OSFT on Standard CL is worth investigating. The per-task results are helpful for understanding retention, and the matched-energy comparisons are relevant to determining whether the improvement comes from the update geometry itself.
Critical weaknesses
  1. The practical solver does not guarantee a feasible update Proposition 1 requires a nuclear-norm subgradient that satisfies the equality constraints. However, Algorithm 1 uses a matrix-sign approximation computed through Newton–Schulz iterations after only one or two dual corrections (pp. 5–6). It is unclear how this procedure handles rank-deficient dual solutions, where the canonical polar factor may not satisfy the constraints. A simple 2 × 2 example illustrates the issue. Suppose the top-left entry of the update must be zero, and let G have rows (0, 1) and (1, 1). The dual optimum adds 1 to the top-left entry of G, giving an all-ones matrix. Its canonical polar factor has every entry equal to 1/2, so the resulting update violates the constraint. In contrast, a feasible optimal update has both diagonal entries equal to zero and both off-diagonal entries equal to the negative spectral-norm bound. Thus, even an exact dual solution does not necessarily recover a feasible primal update through the matrix-sign operation used in Algorithm 1. The paper does not explain how such cases are handled, nor does it report constraint residuals for the approximate solver. Since constraint satisfaction is central to the proposed method, I consider this a significant gap.
  2. The component experiments do not clearly isolate spectral geometry There is a discrepancy in the description of Table 4. Section 5.4 and the table caption refer to Standard CL, whereas Appendix F describes the same experiment as Coding–Math–Medical. This needs to be resolved before the reported 4.1-point accuracy improvement and 14.3% training-time reduction can be interpreted. I also have questions about the matched-energy comparison. If the update is multiplied by a positive scalar learning rate and then normalized at every step to a fixed Frobenius norm, the learning-rate factor should cancel. However, Table 12 reports substantial differences across learning rates. The paper should explain exactly how normalization is applied and whether the tuning experiments follow the same procedure. Finally, I would have liked to see comparisons with simpler combinations of projection and Muon. Without these controls, it is difficult to tell how much of the improvement requires the proposed joint solver, rather than a more direct combination of existing operations.
Questions for authors
  1. How does Algorithm 1 recover a feasible primal update when the dual optimum is rank deficient? In particular, how would the method handle the 2 × 2 example above? It would be helpful to report normalized protection-constraint residuals and objective errors against an exact solver on small instances, varying both the number of dual corrections and Newton–Schulz iterations.
  2. Could the authors clarify which benchmark was used for Table 4? Please provide the backbone, task orders, seeds, and training budget, and reconcile the descriptions in Section 5.4 and Appendix F. I would also like to know what is included in the reported training time.
  3. How exactly is update energy matched across methods? Is normalization applied at every step and for each layer? How is the target Frobenius norm chosen, and is normalization performed before or after clipping and weight decay? Does Table 12 use the same procedure? It would also be useful to compare projection followed by Muon and Muon followed by projection, with matched computational budgets and appropriate control of the final update norm.
  4. What statistics are used to construct the protected bases, and how is information from earlier tasks preserved when the bases are recomputed? For the Qwen2.5-7B experiments, could the authors clarify what “Gradient Replay” involves and whether the competing methods use the same replay data and budget? Please also report the final hyperparameter configurations and uncertainty estimates for the closest baselines, since several of the reported margins are small.
3 comments
AuthorsOpenReview ↗

Response to Reviewer zJqR Q1, Q2

Q1. How does Algorithm 1 recover a feasible primal update when the dual optimum is rank deficient?

Answer: We thank the reviewer for identifying the missing qualification in the primal recovery argument and for suggesting the numerical diagnostics.

Let A(Δ)=(⟨Ci,Δ⟩)i\mathcal A(\Delta)=(\langle C_i,\Delta\rangle)_i and H(λ)=G+A∗(λ)H(\lambda)=G+\mathcal A^*(\lambda). When the dual optimum H⋆H^\star has full rank, the nuclear norm is differentiable at H⋆H^\star, and Δ⋆=−ηmsgn⁡(H⋆)\Delta^\star=-\eta\operatorname{msgn}(H^\star) is a feasible primal optimum. If H⋆=UrΣrVr⊤H^\star=U_r\Sigma_rV_r^\top is rank deficient, its nuclear-norm subdifferential is

\left\\{U_rV_r^\top+U_0WV_0^\top:\Vert W\Vert_2\leq 1\right\\},

where U0U_0 and V0V_0 span the left and right null spaces. A primal optimum is obtained by multiplying a member of this set by −η-\eta and choosing WW so that A(Δ⋆)=0\mathcal A(\Delta^\star)=0. The canonical matrix sign selects W=0W=0, which can violate the protection constraint.

For the finite Newton–Schulz map NKN_K, the dual update gives

=\frac{\eta}{\eta_\lambda} \bigl(\lambda^{(m+1)}-\lambda^{(m)}\bigr), \qquad \Delta^{(m)}=-\eta N_K(H(\lambda^{(m)})).$$ A fixed point therefore satisfies the protection constraint regardless of rank. Exact zeros remain zero under the finite NS map. Small nonzero singular values can map to intermediate values as the dual iterate moves away from the rank-deficient optimum. The residual identity establishes protection feasibility at a fixed point. The spectral-norm bound and objective accuracy are assessed separately. For the reviewer’s example, $G=\operatorname{diag}(1,0)$, $C=\operatorname{diag}(1,2)$, and $\eta=1$. The dual optimum is $\lambda^\star=0$. The canonical sign gives the infeasible update $\operatorname{diag}(-1,0)$, while the primal optimum is $\operatorname{diag}(-1,1/2)$. Starting from zero with $0<\eta_\lambda<1$, the exact-sign updates alternate between $\operatorname{diag}(-1,0)$ and $\operatorname{diag}(-1,1)$. Averaging each pair recovers the primal optimum. More generally, the average of the first $M$ updates satisfies $$\left\Vert\mathcal A(\bar\Delta_M)\right\Vert =\frac{\eta}{\eta_\lambda M} \left\Vert\lambda^{(M)}-\lambda^{(0)}\right\Vert.$$ In the 2 × 2 example, finite NS reached numerical protection feasibility for $K\geq 3$. We also evaluated small rank-deficient instances against a reference SDP solved with CVXPY and Clarabel. We measured the normalized protection residual $\Vert\mathcal A(\Delta)\Vert_2/\Vert\Delta\Vert_F$ and the relative objective error $|\langle G,\Delta\rangle-p^\star|/|p^\star|$, where $p^\star$ is the reference objective value. The exact-sign last-iterate residual was approximately **0.31**, while averaging reduced it to **0.007** after 1,024 corrections. Finite NS reached a protection residual below **1e−12**. The corresponding relative objective errors were **0.16**, **0.006**, and **0.0003** for 3, 5, and 10 NS iterations, respectively. We will add the full sweep over both iteration counts and report spectral-norm feasibility alongside the residual and objective error. **Q2. Which benchmark was used for Table 4, and what does the reported training time include?** **Answer:** We thank the reviewer for flagging the inconsistent benchmark descriptions and asking us to clarify the timing protocol. Table 4 reports the **T5-Large component ablation on the four-task Standard CL benchmark**, using Orders 1–3 from Table 5: 1. DBpedia → Amazon Reviews → Yahoo Answers → AG News 2. DBpedia → Amazon Reviews → AG News → Yahoo Answers 3. Yahoo Answers → Amazon Reviews → AG News → DBpedia Each run used one NVIDIA H100 80 GB GPU, BF16 precision, gradient checkpointing, and one epoch per task. A microbatch size of 8 and 8 gradient accumulation steps gave an effective batch size of 64. Results were averaged over seeds 0, 1, and 2. We will correct the benchmark reference in Appendix F. Table 4 uses Standard CL. The separate BigCodeBench → GSM8K → HuatuoGPT-o1 results appear in Table 3 and Appendix E.4. The **Train Time** column reports mean wall-clock training time for one complete four-task sequence. The timer covers data loading, forward and backward passes, optimizer updates, and protected-basis construction at task boundaries. Evaluation and checkpoint writing are excluded. We will state the timing and averaging protocols in the Table 4 caption.
AuthorsOpenReview ↗

Response to Reviewer zJqR Q3

Q3. How is update energy matched, and how do the two projection orders compare?

Answer: We thank the reviewer for asking us to specify the energy-matching protocol and for suggesting the comparison of projection orders.

We compute the reference norm at each optimizer step and separately for each targeted matrix layer. Let MM be the momentum formed from globally clipped gradients, SS the corresponding spectral direction, and α\alpha the matrix learning rate. The Frobenius update is

=-\tau\frac{D}{\Vert D\Vert_F+\epsilon}, \qquad \tau=\Vert\Delta W_{\mathrm S}\Vert_F =\alpha\Vert S\Vert_F.$$ Here $D=M$ for Frob-only and $D=\Pi(M)$ for Frob-OGD, where $\Pi(M)=M-UU^\top MVV^\top$. The stabilizer $\epsilon$ makes the resulting norm $\tau\Vert D\Vert_F/(\Vert D\Vert_F+\epsilon)$, which approaches $\tau$ when $\Vert D\Vert_F$ is large relative to $\epsilon$. Global gradient clipping at 1.0 precedes momentum accumulation and normalization. Decoupled weight decay is applied separately and excluded from the matched norm. Non-targeted parameters use identical AdamW settings. **Table 10** uses the per-step, per-layer norm-matching protocol for the update-spectrum analysis. **Table 12** uses the same normalization while sweeping the matrix learning rate, which changes the target norm. **Table 4** uses the learning rates selected in Table 12 and compares methods at their tuned scales. We will revise Section 5.4 and Appendix F to distinguish these protocols. We compared projection followed by Muon, $-N_{10}(\Pi(M))$, with Muon followed by projection, $-\Pi(N_{10}(M))$. Each composition uses ten NS iterations. OSD uses the same total number across two five-iteration calls, one for the dual correction and one for the final update. We rescaled each final update to the same spectral norm $\eta$. Each entry below gives the **normalized protection residual / descent relative to the reference constrained optimum**. | Update | Generic | LLM-like | Rank-deficient | |:---|---:|---:|---:| | OSD | 0.021 / 0.95 | 0.036 / 0.88 | 0.033 / 0.99 | | OSD + final projection | 0 / 0.94 | 0 / 0.87 | 0 / 0.98 | | Projection → Muon | 0.12 / 1.01 | 0.19 / 1.07 | 0.25 / 1.00 | | Muon → projection | 0 / 0.87 | 0 / 0.81 | 0 / 0.87 | Projection followed by Muon reintroduces protected components, so its descent ratio can exceed one while violating the constraint. Muon followed by projection satisfies the protection constraint but yields lower descent than OSD with a final projection. Adding a final projection to OSD brings the residual to the numerical floor and lowers the displayed descent ratio by **0.01** in each family.
AuthorsOpenReview ↗

Response to Reviewer zJqR Q4

Q4. How are protected bases constructed, what does Gradient Replay use, and how reliable are the closest margins?

Answer: We thank the reviewer for asking us to clarify these questions.

Protected bases and replay. For the Qwen2.5-7B-Instruct run, the OSD obtained its protected directions from a truncated SVD of the weights at the start of each stage. The bases were held fixed within each stage. As receiving reviewer's question about Gradient Replay we clearly and carefully rechecked all our training logs and checkpoints. The recovered logs show weight-SVD basis construction and contain no record of a gradient replay pass. We will correct the caption to describe the procedure documented in the logs.

Training settings. The training logs and saved LoRA configurations provide the following settings:

  • Shared settings: Microbatch size 1, 8 gradient accumulation steps, and one epoch per stage. Coding ran for 90 steps with a peak learning rate of 5e−6. Math ran for 1,000 steps with a peak learning rate of 2e−5. The logged peak learning rate for Medical was 5e−6.
  • O-LoRA: Rank 16, LoRA alpha 32, and dropout 0.05, applied to all seven projection types.
  • Sculpting Subspaces (OSFT): SVD bases on the query, key, value, and output attention projections.
  • OSD: Protected rank 3 in both Math and Medical, with matrix step sizes of 1e−4 and 1e−5, respectively.

Training budget and uncertainty. The Medical stage used 2,412 optimizer steps for each method. The table gives the observed differences in percentage points with approximate 95% interval half-widths.

Stage C comparisonCodingMathMedicalAverage
OSD − O-LoRA+0.6 ± 1.3+1.2 ± 1.4+0.6 ± 1.9+0.8 ± 0.9
OSD − Sculpting Subspaces+0.2 ± 1.3+2.0 ± 1.4+0.4 ± 5.9+0.9 ± 2.1

We will report these scores as descriptive comparisons under the observed training budgets and revise the caption to disclose the step counts and basis construction.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.