acceptodds
Desk rejected at ICLR 2027

SCULPT-LoRA: Spectral Sculpting for Continual Low-Rank Adaptation

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

19 views

Abstract

Low-Rank Adaptation (LoRA) enables parameter-efficient continual learning, yet a fixed rank budget neither safeguards pretrained representations nor guarantees effective use of adaptation capacity. We identify two complementary spectral challenges: interference with dominant pretrained directions and concentration of update energy in a few singular modes. Addressing both requires controlling where updates act and how their energy is distributed. To alleviate these problem, We propose SCULPT-LoRA, a framework that couples Elastic Principal Subspace Protection (EPSP) with Polynomial Spectral Balancing (PSB). EPSP constructs a fixed orthonormal adaptation basis by attenuating its overlap with the pretrained principal subspace, allowing controlled modification rather than strict exclusion. PSB reshapes the trainable factor through a compact Gram-matrix polynomial during the forward pass, reducing spectral concentration. The two mechanisms are linked by row orthonormality, which transfers the nonzero singular spectrum of the conditioned factor exactly to the effective LoRA update. We derive a sharp worst-case bound on principal-subspace perturbation that decreases with protection strength under a fixed update-energy budget, and establish conditions for condition-number improvement under polynomial spectral shaping. Experiments across five continual learning benchmarks show improved final and average accuracy, with ablations supporting the complementary benefits of elastic protection and spectral balancing.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JSGLOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

SCULPT-LoRA couples elastic protection of an original pretrained principal input subspace with polynomial conditioning of the trainable LoRA factor. The input basis is fixed and row-orthonormal, so the conditioned factor and effective update share their nonzero singular spectrum. The paper proves a sharp protected-component perturbation bound and condition-number improvement for the specified polynomial. Across five image-classification datasets and eight incremental settings, it reports competitive or improved final and average accuracy, using three seeds. Component, protection-strength, and scale/gradient-path ablations support the combined design; updates are merged after each task.

Critical strengths
  1. The coupling of subspace orientation and factor-level spectral control is mathematically precise. The sharp EPSP bound, exact spectrum-transfer identity, and polynomial positivity/contraction certificates establish concrete properties of the stated parameterization. The appendix also distinguishes these properties from optimization or accuracy guarantees, making the theoretical contribution assessable.
  2. The empirical evaluation spans multiple datasets and sequence lengths, with three-seed reporting and meaningful component controls. Table 3 shows that both components improve over a fixed random orthogonal basis, while Table 6 includes learnable-scale-only and norm-matched PSB variants. This gives useful support for spectral shaping beyond simply amplifying the update norm.
Critical weaknesses
  1. The link between principal-weight protection and functional knowledge retention remains incompletely tested. The actual ViT and trained classifier do not receive a numerical retention certificate (Appendix C.5). Figure 7 reports overall and new-task trajectories, but these do not directly measure preservation of an independent pretrained capability or provide per-old-task forgetting statistics. Thus the geometric theorem is stronger evidence for controlled parameter perturbation than for the paper's functional preservation explanation.
  2. The empirical incremental value of EPSP over PSB is modest in several settings, and its robustness is not fully established. Table 3 reports additional final-accuracy gains of only 0.37-1.03 points, while the 20S-ImageNetA combined result has 1.71-point standard deviation. Sensitivity tables lack uncertainty and the development procedure for selecting alpha and K is not explained. Together with absent measured training-time/memory costs, this makes the practical benefit of the complete combination less certain than the headline best means suggest.
Questions for authors
  1. Beyond Figure 7's overall/new-task curves, do the existing runs provide per-task forgetting statistics or retention measurements on an independent pretrained capability? This would clarify how much of the gain supports the functional preservation explanation.

  2. How were alpha, K, rank, and baseline tuning budgets selected without using final evaluation outcomes? Please report any paired-seed evidence for the gain of EPSP over PSB and the measured training overhead of exact spectral normalization.

Reviewer_kBRsOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SCULPT-LoRA for replay-free class-incremental learning with a pretrained Vision Transformer. EPSP constructs a fixed, row-orthonormal LoRA down-projection by attenuating overlap with the pretrained principal subspace, while PSB reshapes the trainable up-projection’s singular-value spectrum via a low-dimensional Gram-matrix polynomial. Row orthonormality enables exact transfer of the up-projection’s nonzero spectrum to the effective LoRA update. The paper provides principal-subspace perturbation and condition-number analyses, and reports generally competitive final and average accuracy across five benchmarks under eight task splits.

Critical strengths
  1. The problem is clearly characterized. The difficulties of CL LoRA are decomposed into two complementary dimensions: where updates act and how the limited rank is effectively utilized. The motivation presented in Figures 1 and 6 is convincing.

  2. The method design is well-structured. EPSP replaces hard exclusion with elastic protection, while PSB uses a low-dimensional Gram polynomial to avoid full-matrix SVD, making it computationally reasonable. The two components exactly transfer the nonzero spectrum through the row-orthonormality condition, making the design self-consistent.

  3. The theoretical part is relatively complete. Theorem 3.1 provides a sharp worst-case protection bound, Lemma 3.2 proves exact spectral transfer, and Corollary 3.4 gives condition-number improvement.

Critical weaknesses
  1. Boundary of novelty relative to closely related work. The basic idea of protecting pretrained principal directions and adapting in the remaining directions has already appeared in MiLoRA and KeepLoRA. KeepLoRA further considers the protection of historical task feature directions. For PSB, the paper explicitly adopts the polynomial and coefficients published by Wang et al. (2026b); the polynomial itself is therefore not an original contribution of this work. In addition, EBLoRA [1] has already linked spectral imbalance in LoRA updates to forgetting and proposed a method to balance the update spectrum. This paper should discuss these highly related works and clearly delineate the boundary of its novelty.

  2. Lack of important and directly relevant recent baselines. The experiments do not compare with the recent methods LoDA [2] and E²-LoRA [3], nor do they explain why such comparisons are not possible. The absence of these comparisons weakens the current conclusions regarding the method’s competitiveness and novelty.

  3. A key gap remains between pretrained subspace protection and the preservation of historical task knowledge. EPSP constructs constraints based on the principal subspace of the original pretrained weights, rather than forming new protected spaces from historical tasks. The analysis in the paper is insufficient to explain why this design can continuously preserve knowledge acquired later.

  4. The link between improved spectral conditioning and continual learning benefits still requires more direct validation. The condition-number theorem compares the same original factor and its forward transformation; it cannot directly explain differences between models obtained from two independent training runs, nor does it guarantee that the cumulative update has a better spectral structure. Spectral concentration may also simply reflect that a task genuinely requires only a few effective directions, so it cannot be judged as a capacity failure solely on this basis.

[1] EBLoRA: Spectral Imbalance Causes Forgetting in Low-Rank Continual Adaptation. ICML, 2026.

[2] LoDA: Task-Driven Subspace Decomposition for Knowledge Sharing and Isolation in LoRA-based Continual Learning. ICML, 2026.

[3] E²-LoRA: Energy-Structured Low-Rank Adaptation for Continual Learning. ICML, 2026.

Questions for authors
  1. Please clarify the independent contributions of this work relative to KeepLoRA, MiLoRA, PC Layer, and EBLoRA. Can you compare soft protection, strict protection, and their corresponding variants with the same PSB added under a unified implementation, so as to disentangle component improvements from combined gains?

  2. Why are LoDA and E²-LoRA not compared? Please supplement results under a consistent backbone, task split, and reasonable tuning budget, or specify the concrete obstacles in terms of setup, implementation, or resources.

  3. How is knowledge learned by historical tasks outside the pretrained principal subspace protected? Can you report the proportion of energy of old-task features inside and outside the protected subspace, as well as the drift in old-task outputs caused by subsequent updates?

  4. What happens when historical task protection is added? Please compare the current method with a variant augmented with historical feature protection, in order to examine whether omitting this mechanism can improve knowledge sharing and whether its cost grows with the number of tasks.

  5. Why is the current fixed polynomial chosen? Can simpler spectral transformations achieve comparable gains? Please also report training time, peak memory, and the layers, checkpoints, and numerical rank threshold used for spectral statistics.

Reviewer_nSyUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies LoRA-based continual learning and proposes SCULPT-LoRA, which combines Elastic Principal Subspace Protection (EPSP) and Polynomial Spectral Balancing (PSB). EPSP uses an interpolation-based initialization to balance knowledge preservation and adaptation to new tasks. PSB applies a polynomial transformation to address spectral plasticity collapse. The method is evaluated on five image classification tasks.

Critical strengths
  1. The paper clearly identifies the key challenges and presents corresponding approaches to address them.
  2. The polynomial spectral balancing method appears interesting to me.
Critical weaknesses

The experiments are limited to image classification tasks. Given the widespread use of LoRA in NLP and other vision tasks, this narrow evaluation raises concerns about the generalizability of the proposed method. Additional experiments on more diverse tasks would help establish its broader effectiveness.

Questions for authors
  1. How is the parameter α\alpha in EPSP selected in the experiments? Could the authors further explain how this choice effectively affects the balance between preserving previous knowledge and adapting to new tasks?
  2. What motivates the use of a polynomial transformation to adjust the spectrum? What are its advantages and limitations compared with other approaches? Could the authors also explain the rationale behind the design of Equation (5)?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.