- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Orthogonal Spectral Descent (OSD), a continual learning method that combines subspace protection with spectral-norm-constrained optimization. The method derives a nuclear-norm dual formulation and uses Newton–Schulz iterations with dual corrections to approximate the constrained updates. Experiments on T5, LLaMA, and Qwen models demonstrate improvements over existing continual learning baselines.
Critical strengths
-
The paper presents an interesting approach to continual learning by combining orthogonal subspace protection with spectral optimization. The motivation is clear, and the proposed formulation provides a principled way to control parameter updates while preserving previously learned knowledge.
-
The theoretical derivation is generally sound, and the proposed optimization algorithm offers a practical way to approximate the spectral update without performing expensive SVD computations at every step.
-
The experiments cover several continual learning benchmarks and different model architectures. OSD consistently outperforms OSFT in the reported settings, with noticeable improvements on the 4-task benchmark and Qwen2.5-3B. The ablation study also provides useful evidence for the effectiveness of combining spectral optimization with subspace protection.
Critical weaknesses
-
The improvements over strong baselines are relatively modest on some benchmarks, particularly the 15-task benchmark and TRACE. Additional experiments on more recent language models would help strengthen the generality of the findings.
-
The ablation study could be more comprehensive. In particular, comparing OSD with straightforward combinations of subspace projection and Muon would help clarify the specific contribution of the proposed dual optimization.
-
The practical solver could be analyzed in more detail. Since the method uses approximate Newton–Schulz iterations and a small number of dual corrections, reporting the constraint residuals would help assess how closely the implemented updates satisfy the theoretical formulation.
Questions for authors
-
How does OSD compare with simpler approaches that directly combine subspace projection and Muon optimization?
-
Could the authors report the orthogonality constraint residuals of the practical solver and discuss the effect of the number of dual correction steps?
-
Have the authors considered evaluating OSD on more recent LLMs to further demonstrate its generality?
Response to Reviewer 623R
How does OSD compare with simpler approaches that directly combine subspace projection and Muon optimization?
Answer: The direct baseline applies subspace projection and Muon as separate operations. OSD uses dual corrections to incorporate subspace protection into the spectral update itself. In the updated Table 4, the combined baseline reaches 68.9±2.2% average accuracy on Standard CL. OSD reaches 78.9±1.0%, a gain of 10.0 percentage points, and takes 26.54 minutes versus 27.83 minutes for the combined baseline.
Table 4. Component ablation on the Standard CL benchmark with T5-Large. AA is final average accuracy (mean ± standard deviation). Training time is the mean wall-clock time per task sequence.
Could the authors report the orthogonality constraint residuals of the practical solver and discuss the effect of the number of dual correction steps?
Answer: We thank the reviewer. For each protected matrix update ΔW, we measure the normalized constraint residual
r(ΔW)=max{∥ΔW∥F,ε}∥U⊤ΔWV∥F,
where ε>0 prevents division by zero. This is the fraction of the update norm that lies in the protected subspace.
Controlled example. On Reviewer zJqR's 2×2 example (U=V=e1), with five Newton–Schulz iterations, Frobenius normalization, and dual step size 0.1, a cold start (λ(0)=0) with 0,1,2,10,20, and 50 dual corrections gives residuals of 31.60%, 30.43%, 29.27%, 19.43%, 7.69%, and 0.14%. The iteration settles at λ≈0.63 rather than at the exact dual optimum λ⋆=1: it converges to the multiplier at which the approximate matrix sign satisfies the constraint. Near-zero residual is therefore attainable even though, as discussed in our response to Reviewer zJqR, the exact polar factor at λ⋆ is infeasible.
Effect of warm starting. The practical solver warm-starts the dual variable across optimizer steps, so corrections accumulate over training. With G held fixed, one correction per step reaches 0.14% residual after 50 optimizer steps, and two corrections per step after 25. With a drifting momentum matrix (β=0.9, i.i.d. Gaussian gradient noise of standard deviation 0.5 around G), the steady-state median (95th percentile) residual is 34.0% (41.2%) with no corrections, 7.6% (20.2%) with one, and 3.6% (11.1%) with two. The effect of Tin is thus governed by the dual correction rate relative to the drift of the momentum matrix. The finite solver provides no general guarantee of monotonic residual reduction or exact feasibility.
Residuals during LLM training. On the Standard CL benchmark, we compare Tin∈{0,1,2,5} while holding the Newton–Schulz iteration count and the training configuration fixed. With Tin=0 and zero-initialized multipliers, the dual variable never moves, so this setting coincides with our Muon-only arm.
Have the authors considered evaluating OSD on more recent LLMs to further demonstrate its generality?
Answer: We thank the reviewer for this suggestion. To further evaluate OSD's generality, we conducted additional experiments on Qwen3.5-4B using the Standard CL benchmark. The table below reports the final accuracy on each task after completing the four-task continual learning stream, together with average accuracy (AA).
Final task accuracies and average accuracy (AA) on the Standard CL benchmark using Qwen3.5-4B. All values are percentages.
OSD improves AA from 28.76% to 70.37%, a gain of 41.61 percentage points, and achieves higher final accuracy on all four tasks. The substantial gains on DBpedia, Amazon, and Yahoo are consistent with stronger retention of earlier tasks. These additional results support OSD's applicability to a more recent LLM backbone, extending the evidence beyond the models evaluated in the original submission. We will include these results in the revised manuscript.