acceptodds
Under review as a conference paper at ICLR 2027

LANS: Tuning in Larger Null Spaces for Continual Learning

Abstract

Continual learning aims to enable models to sequentially acquire knowledge while maintaining a balance between stability and plasticity. To achieve this, null-space-based approaches enforce weight updates to lie in the null spaces of past input features, thereby preserving historical responses. Although this strict constraint offers a strong stability guarantee, it inevitably reduces plasticity, especially when model capacity is limited or the number of tasks grows. In this paper, we revisit the stability condition for the attention mechanism in modern Transformers. Unlike prior approaches that constrain each layer independently, we analyze how updates to the attention parameters affect both the attention affinities and the final output, and derive attention-specific stability conditions. These conditions define a feasible update subspace that is provably larger than the conventional null space, offering greater plasticity while preserving historical attention behavior to first order. Building on these results, we characterize the feasible subspace in closed form and develop LANS, a low-rank parameterization that efficiently learns weight increments within the derived constraints. Extensive experiments on multiple class-incremental learning benchmarks demonstrate consistent improvements over existing state-of-the-art approaches. Code is available at https://anonymous.4open.science/r/lans-207E.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual learning, null space, attention-specific stability conditions

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_49SpOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LANS for class-incremental learning with pretrained ViTs. It replaces layerwise input null-space constraints with attention-aware constraints: V updates preserve attention-weighted outputs, while Q/K updates preserve first-order attention scores. It uses fixed subspace bases and trainable coefficients to enable mergeable low-rank updates. The paper derives the feasible space and its dimensionality gains.

Critical strengths

1.The characterization of the Q/K update space provides a concrete technical contribution. Expressing the solution to the two-sided constraint LΔWR⊤=0L\Delta WR^\top=0 as the sum of two null spaces yields both verifiable dimensionality conditions and a direct training parameterization.

2.The experimental gains in more challenging settings are compelling. In the 50-task DeiT-S setting, LANS leads the strongest listed baselines in FAA by 7.56 and 5.10 percentage points on CIFAR-100 and ImageNet-R, respectively.

Critical weaknesses
  1. The comparison with the most relevant baseline is subject to a key confound. LANS uses classifier alignment (CA), but the main table’s ImageNet-R results for InfLoRA, 75.65/80.82, correspond to the version without CA in the original paper; the InfLoRA paper reports 79.78/83.38 with CA. Results across papers cannot be used to directly determine which method is better, but this means the claimed gains over InfLoRA cannot yet be attributed to null-space expansion. The standard LoRA ablation in Table 4 also cannot substitute for an input null-space comparison under the same calibration and parameter budget.

  2. The experimental performance is not competitive enough: on ImageNet-R at t=20, it is only around 77, and comparisons with recent state-of-the-art methods are lacking.

  3. The theoretical gains in the space have not yet been adequately linked to the space used in actual training. The dimensionality result concerns the exact null space and explicitly applies before rank truncation; the implementation, however, uses a spectrally thresholded approximation to the null space and then truncates it further. After these operations, the original space-inclusion relationship and dimensionality advantage do not automatically hold.

Questions for authors

The main issue is that I’m not impressed by how the paper’s method performs in the experiments.

16 comments
AuthorsOpenReview ↗

Response to Reviewer 49Sp (Part 1/2)

We thank the reviewer for recognizing LANS's contributions and for the constructive feedback. We try our best to address your questions as follows.

Q1.1: Calibration protocol

A1.1 Original InfLoRA results omitted CA, whereas LANS used it. We added InfLoRA+CA with the same CA strategy and clarified the protocol in Appendix C.1 (pp. 19–20). LANS retains these FAA/CAA gains (percentage points; Tables 1–2, p. 7):

BackboneCIFAR-100ImageNet-RCUB-200
ViT-B/16-IN21K+0.35 / +0.45+0.42 / +0.27+0.85 / +2.48
DeiT-S/16-IN611+6.38 / +6.79+10.19 / +11.00+10.12 / +12.13

Long-task results (Table 3, p. 8) also favor LANS. Entries give FAA/CAA (%); C100 and IN-R denote CIFAR-100 and ImageNet-R. Bold marks the better result within each setting.

BackboneTasksDatasetInfLoRA+CALANS
ViT-B20C10090.32/94.0090.57/94.18
ViT-B20IN-R76.53/82.2677.38/83.15
ViT-B50C10084.41/90.5386.30/91.11
ViT-B50IN-R71.28/76.9671.37/77.85
DeiT-S20C10072.72/81.0176.74/84.61
DeiT-S20IN-R48.13/56.6158.67/67.72
DeiT-S50C10068.45/79.0173.24/82.36
DeiT-S50IN-R43.75/52.9352.15/62.29

Q1.2: Matched parameter budgets

A1.2 Table 5 (p. 10) extends comparisons between InfLoRA and LANS. Both methods use the same training epochs and CA strategy, with matched parameter counts and comparable actual update ranks. InfLoRA ranks 24/48/96 correspond to LANS ranks 12/24/48.

Below are LANS minus InfLoRA (QKV) FAA/CAA differences (points). 1× is each backbone's lowest budget; M denotes millions of parameters.

DeiT-S budgetDeiT-S: CIFAR-100DeiT-S: ImageNet-RViT-B budgetViT-B: CIFAR-100ViT-B: ImageNet-R
1×, 0.332M+3.01 / +3.15+11.44 / +9.631×, 0.664M+0.60 / +0.48−0.10 / −1.50
2×, 0.664M+4.26 / +3.69+11.81 / +11.232×, 1.327M+0.87 / +0.75+1.58 / −0.76
4×, 1.327M+5.21 / +4.51+11.50 / +11.234×, 2.654M+1.13 / +0.87+5.37 / +0.88

LANS improves both metrics across DeiT-S budgets and on ViT-B CIFAR-100. ViT-B ImageNet-R gains depend on budget, reaching 5.37/0.88 points at 4×. These results support consistent benefits with different budgets.

Q2.1: Comparisons with recent methods

We added SplitLoRA (Qiu et al., 2026), SplitLoRA+CA, and SD-LoRA+CA (Wu et al., 2025). Results are from Tables 1–3 (pp. 7–8). Cells report FAA/CAA (%); CUB denotes CUB-200. Bold marks the best among the listed methods for each metric. Analysis and references follow in Part 2.

ViT-B/16-IN21K, 10 tasks (Table 1).

MethodC100IN-RCUB
InfLoRA87.06±0.25/91.59±0.1375.65±0.14/80.82±0.2480.26±1.21/84.04±0.55
InfLoRA+CA91.67±0.15/94.46±0.4878.63±0.28/83.01±0.5087.82±0.47/89.53±0.78
SplitLoRA89.27±0.18/92.89±0.5478.26±0.29/82.81±0.4481.52±0.32/82.92±0.53
SplitLoRA+CA90.55±0.21/93.50±0.5579.36±0.29/83.92±0.6285.80±0.60/85.73±0.77
SD-LoRA+CA91.13±0.16/94.15±0.5479.01±0.43/83.60±0.5188.04±0.29/89.19±0.62
LANS92.02±0.14/94.91±0.5579.05±0.41/83.28±0.3788.67±0.49/92.01±0.65

DeiT-S/16-IN611, 10 tasks (Table 2).

MethodC100IN-RCUB
InfLoRA62.99/75.1055.07/65.4844.86/59.03
InfLoRA+CA72.20/79.5551.63/59.5162.46/66.56
SplitLoRA68.69/78.8354.10/61.4734.86/48.35
SplitLoRA+CA70.68/79.6255.25/62.1445.86/54.09
SD-LoRA+CA73.43/82.0356.18/65.1161.62/66.77
LANS78.58/86.3461.82/70.5172.58/78.69

ViT-B/16-IN21K, 20/50 tasks (Table 3).

Method20T C10020T IN-R50T C10050T IN-R
InfLoRA83.37/89.5272.32/79.9556.67/70.5661.38/70.97
InfLoRA+CA90.32/94.0076.53/82.2684.41/90.5371.28/76.96
SplitLoRA87.25/92.2576.80/81.3878.67/85.7762.23/67.35
SplitLoRA+CA88.28/92.7678.07/82.3385.33/89.7669.80/74.06
SD-LoRA+CA89.23/92.7576.20/83.0782.98/89.0965.77/75.97
LANS90.57/94.1877.38/83.1586.30/91.1171.37/77.85

DeiT-S/16-IN611, 20/50 tasks (Table 3).

Method20T C10020T IN-R50T C10050T IN-R
InfLoRA50.46/67.3947.62/60.5126.13/48.9526.47/40.31
InfLoRA+CA72.72/81.0148.13/56.6168.45/79.0143.75/52.93
SplitLoRA61.37/74.4647.83/54.6043.32/58.0127.87/31.96
SplitLoRA+CA65.89/76.6151.00/57.2365.98/72.9349.93/50.20
SD-LoRA+CA70.25/80.2150.58/60.6860.71/74.7038.67/51.64
LANS76.74/84.6158.67/67.7273.24/82.3652.15/62.29
AuthorsOpenReview ↗

Response to Reviewer 49Sp (Part 2/2)

Q2.2: Competitiveness and experimental settings

Overall competitiveness. LANS leads FAA and CAA on ViT-B CIFAR-100 and CUB-200 (Table 1, p. 7), but its ImageNet-R gains are limited. Its advantages are clearer with DeiT-S: it leads all six metrics in Table 2 (p. 7), with FAA/CAA gains of 3.17/2.45, 1.09/3.61, and 1.92/3.65 points over the strongest competing result per metric on CIFAR-100, ImageNet-R, and CUB-200. It also leads 15 of 16 long-sequence metrics across both backbones (Table 3, p. 8). Preserving update freedom may be especially useful when limited pre-training or model capacity requires greater adaptation.

Interpreting the limited advantage on ImageNet-R with ViT-B/IN21K. We examine two factors that may explain the gap to SplitLoRA+CA, the strongest baseline on 10-task ImageNet-R in Table 1.

(1) Experimental settings. SplitLoRA's official script uses IN21K pre-training followed by IN1K fine-tuning, whereas ours uses IN21K pre-training only; its default task order also differs. We therefore reproduce SplitLoRA with our checkpoint, data splits, and task order.

Originally reported SplitLoRA results (Qiu et al., 2026): CIFAR-100 90.33/93.70; ImageNet-R 81.00/85.84 FAA/CAA (%). These use different settings and are excluded from best-result comparisons.

Aligned 10-task results (Table 1, p. 7; mean FAA/CAA %; best values in bold):

MethodCIFAR-100ImageNet-R
SplitLoRA89.27/92.8978.26/82.81
SplitLoRA+CA90.55/93.5079.36/83.92
LANS92.02/94.9179.05/83.28

With the same CA strategy, LANS leads on CIFAR-100 and trails SplitLoRA+CA by only 0.31/0.64 points on ImageNet-R. These aligned results support its competitiveness without implying an advantage in every setting.

(2) Pre-training overlap and stability mechanisms. We hypothesize that category or distribution overlap between ImageNet-R and ImageNet-21K may reduce adaptation needs, favoring strategies that preserve pre-trained representations. SplitLoRA scales a LoRA factor's initialization magnitude by 1/(t+1)1/(t+1) in subsequent tasks. Adding similar task-dependent scaling to LANS yields the following ImageNet-R results with ViT-B/16-IN21K (Appendix C.5, Table 6, p. 24). All methods use CA; LANS uses r=48r=48. Entries are means; bold marks the best within each task count.

TasksMethodFAA (%)CAA (%)
10SplitLoRA+CA79.3683.92
10LANS79.0583.28
10LANS + task scaling81.0285.18
20SplitLoRA+CA78.0782.33
20LANS77.3883.15
20LANS + task scaling78.9384.39
50SplitLoRA+CA69.8074.06
50LANS71.3777.85
50LANS + task scaling72.2379.18

Scaling improves LANS at all three sequence lengths, exceeding SplitLoRA+CA in both metrics. This supports its compatibility with LANS's attention-aware constraints.

These results are reported only in the appendix; Tables 1–3 retain default LANS without task scaling. We use a consistent default approach to assess the core method across task settings, rather than add a strategy based on a particular benchmark. Since scaling has been evaluated only in the settings above, we do not adopt it as a default component of the main experiments.

Q3: Connecting theoretical and practical update spaces

Following your feedback, we added implementation details and experiments linking theoretical flexibility to practical updates.

Approximation and truncation. Appendices B.1–B.2 (pp. 17–19) clarify spectral selection, truncation, and zero padding. Low-response directions satisfy the constraints approximately. Excess directions are truncated by retaining those with the lowest responses; zero padding adds no effective directions. Appendix C.3 and Figure 8 (p. 22) visualize this process and quantify the retained basis's relative residual.

Post-truncation dimensions. Figure 6 (p. 21; analysis in Appendix C.3, p. 22) compares LANS and InfLoRA-form using the same checkpoint statistics, spectral threshold, and matched branch caps. At task 10 on ImageNet-R, LANS retains 17.6%–86.0% larger layer-wise effective dimensions with DeiT-S and 31.9%–58.9% with ViT-B.

Empirical validation. We also added threshold sensitivity (Figure 5, p. 10), representation drift (Appendix C.4 and Figure 9, p. 23), and matched-CA/budget comparisons (Table 5, p. 10). Together, these support additional practical flexibility and competitive stability in the evaluated settings.

Thank you again for helping improve our manuscript.

References

Qiu et al. SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting. ICLR, 2026.

Wu et al. SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning. ICLR, 2025.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank you for your valuable comments, which we found very constructive for improving the rigor and quality of our paper. We have tried our best to address your concerns and would like to know whether our responses have addressed them. If anything remains unclear, we are more than happy to discuss it further.

Reviewer_CnrWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies continual learning for pre-trained ViTs, testing whether attention-specific constraints give a larger feasible update space than standard input-null-space constraints without forgetting prior tasks. LANS constrains value updates via attention weights and query/key updates via first-order bounds, learning increments with fixed bases and trainable coefficients that merge into the backbone at inference. Experiments on CIFAR-100, ImageNet-R, and CUB-200 cover two backbones, varying task lengths, ablations, representation drift, feasible space, and approximation error.

Critical strengths
  1. Section 4.1 exploits the interactions within self-attention to derive less restrictive update constraints than treating each projection independently. Theorems 1 and 2 characterize the feasible spaces and their dimension gains under explicit conditions. Together with the parameterization in Section 4.2, this provides a coherent mathematical basis for the method and a useful perspective on the stability–plasticity trade-off.
  2. Section 4.2 merges the learned weight increments into the shared backbone after each task. Consequently, LANS does not require additional adapter branches or task-specific module routing at inference. This offers a practical deployment advantage over methods that maintain expanding module pools, while keeping the backbone architecture and its forward computation unchanged as additional tasks are learned.
Critical weaknesses
  1. Section 5 mentions five-run averages, but Tables 2–4 leave out variance for the smaller-backbone, long-sequence, and component-ablation runs. Without error bars, it's hard to tell if small gains (like some component improvements in Table 4) hold across seeds, making those finer-grained results less convincing than the main benchmarks.
  2. Appendix B.2 (Eq. 67) describes truncation, but Appendix C.1 doesn't list the actual rank values, per-head and query/key settings, or trainable parameter counts. Adding a short config table for both backbones and the compared low-rank baselines would make the setup much easier to reproduce.
Questions for authors
  1. Could Figure 4(a) include InfLoRA in the representation-drift comparison under the same experimental setting? This would help assess whether LANS maintains comparable stability to the closest null-space baseline while allowing more flexible updates.

  2. Figure 4(b) demonstrates larger theoretical feasible spaces. Could the authors briefly explain how this additional flexibility is utilized under the fixed-rank parameterization in Appendix B.2? This would help clarify the connection between the feasible-space analysis and practical adaptation.

  3. Section 5.1 (Tables 1–3) compares LANS with InfLoRA, while Appendix C.1 describes classifier calibration. Do these results use the same calibration protocol for both methods, and how does the comparison change under matched calibration and adaptation budgets?

3 comments
AuthorsOpenReview ↗

Response to Reviewer CnrW

We thank the reviewer for recognizing our structural motivation, theory, and deployment benefits. We try our best to address your questions as follows.

Q1: Representation-drift comparison with InfLoRA

Figure 9 (p. 23) compares LANS with InfLoRA-form. The table reports mean and final historical drift:

DatasetMethodMean historical drift ↓Final historical drift ↓
CIFAR-100InfLoRA-form0.003900.00454
CIFAR-100LANS0.002240.00246
ImageNet-RInfLoRA-form0.032190.03544
ImageNet-RLANS0.033770.03703

LANS has lower mean and final drift on CIFAR-100. On ImageNet-R, they are 4.9%/4.5% higher but remain comparable in magnitude, supporting similar empirical representation stability to InfLoRA-form.

Q2: Additional flexibility under fixed-rank parameterization

Figure 6 (p. 21) compares post-truncation feasible dimensions of LANS and InfLoRA-form using the same checkpoint statistics, spectral threshold, and truncation caps for corresponding branches. On ImageNet-R, LANS increases layer-wise effective dimensions by 17.6%–86.0% with DeiT-S/16-IN611 and 31.9%–58.9% with ViT-B/16-IN21K. These results support additional update freedom after truncation under the evaluated thresholds and branch budgets.

Q3: InfLoRA calibration and matched-budget comparisons

A3.1 (Calibration protocol clarification) The entries labeled InfLoRA in the original Tables 1–3 did not include classifier alignment (CA), whereas LANS used CA. We now add InfLoRA+CA and clarify the calibration protocol in Appendix C.1.

In the updated 10-task results (Tables 1–2), LANS improves over InfLoRA+CA by the following FAA / CAA differences in percentage points:

BackboneCIFAR-100ImageNet-RCUB-200
ViT-B/16-IN21K+0.35 / +0.45+0.42 / +0.27+0.85 / +2.48
DeiT-S/16-IN611+6.38 / +6.79+10.19 / +11.00+10.12 / +12.13

LANS leads InfLoRA+CA, with larger gains under DeiT-S/16-IN611.

A3.2 (Matched-budget comparisons). All methods below use CA.

DeiT-S/16-IN611

MethodRankParameter budgetCIFAR-100: FAA / CAAImageNet-R: FAA / CAA
InfLoRA (official)100.092M72.20 ± 0.35 / 79.55 ± 0.2051.63 ± 0.40 / 59.51 ± 0.30
InfLoRA (QKV)241×, 0.332M74.72 ± 0.43 / 82.38 ± 0.1749.68 ± 0.45 / 59.69 ± 0.33
LANS (QKV)121×, 0.332M77.73 ± 0.29 / 85.53 ± 0.0961.12 ± 0.22 / 69.32 ± 0.17
InfLoRA (QKV)482×, 0.664M73.56 ± 0.11 / 81.87 ± 0.0649.46 ± 0.18 / 58.26 ± 0.05
LANS (QKV)242×, 0.664M77.82 ± 0.34 / 85.56 ± 0.0861.27 ± 0.29 / 69.49 ± 0.18
InfLoRA (QKV)964×, 1.327M73.37 ± 0.15 / 81.83 ± 0.0850.32 ± 0.19 / 59.28 ± 0.07
LANS (QKV)484×, 1.327M78.58 ± 0.30 / 86.34 ± 0.4061.82 ± 0.25 / 70.51 ± 0.39

ViT-B/16-IN21K

MethodRankParameter budgetCIFAR-100: FAA / CAAImageNet-R: FAA / CAA
InfLoRA (official)100.184M91.67 ± 0.15 / 94.46 ± 0.4878.63 ± 0.28 / 83.01 ± 0.50
InfLoRA (QKV)241×, 0.664M90.88 ± 0.40 / 94.15 ± 0.1875.90 ± 0.78 / 83.27 ± 0.25
LANS (QKV)121×, 0.664M91.48 ± 0.07 / 94.63 ± 0.0475.80 ± 0.16 / 81.77 ± 0.13
InfLoRA (QKV)482×, 1.327M90.60 ± 0.06 / 93.89 ± 0.0274.18 ± 0.18 / 82.68 ± 0.14
LANS (QKV)242×, 1.327M91.47 ± 0.07 / 94.64 ± 0.0375.76 ± 0.06 / 81.92 ± 0.14
InfLoRA (QKV)964×, 2.654M90.89 ± 0.15 / 94.04 ± 0.0673.68 ± 0.23 / 82.40 ± 0.15
LANS (QKV)484×, 2.654M92.02 ± 0.14 / 94.91 ± 0.5579.05 ± 0.41 / 83.28 ± 0.37

LANS outperforms InfLoRA (QKV) in FAA and CAA on both datasets at all three DeiT-S budgets. The lowest-budget gains are 3.01/3.15 points on CIFAR-100 and 11.44/9.63 points on ImageNet-R; the highest-budget gains are 5.21/4.51 and 11.50/11.23 points, respectively. With ViT-B, LANS improves CIFAR-100 results at every budget. On ImageNet-R, it gains 5.37/0.88 points at the highest budget, while both metrics are slightly lower at the lowest budget and only FAA is higher at the middle budget. These results indicate that LANS's gains are primarily attributable to its effective attention-aware constraint design, rather than simply increasing the number of trainable parameters.

Additional clarifications

To address the concern about result variability (in W.1), we have added standard deviations to the component ablations (Table 4, p. 8) and matched-budget comparisons (Table 5, p. 10). For W.2, we have expanded Appendix C.1 to specify the default rank and per-head Q/K and value-branch budgets; Table 5 reports the ranks and trainable parameter counts for the matched-budget comparisons.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank you for your valuable comments, which we found very constructive for improving the rigor and quality of our paper. We have tried our best to address your concerns and would like to know whether our responses have addressed them. If anything remains unclear, we are more than happy to discuss it further.

As the deadline for the rebuttal discussion period approaches, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns. We would be happy to clarify any points that remain unclear.

Reviewer_iGFzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses the issue that traditional null space constraints may overly restrict the plasticity of continual learning. It proposes LANS, which expands the feasible parameter update space through attention-weighted value constraints and query/key bilinear constraints. The authors provide theoretical derivations and validate the effectiveness of the method through multiple continual learning benchmarks, component ablation, and stability analyses.

Critical strengths
  • This paper derives a more relaxed null space constraint based on the internal structure of self-attention, with a clear theoretical motivation and provides a relatively complete mathematical proof.
  • The experiments covered multiple datasets, pre-trained models, and task sequence lengths, achieving notable improvements in some settings. Ablation and stability analyses further supported the method design.
Critical weaknesses
  • The theoretical derivation targets an exact null space, while practical methods rely on spectral energy truncation to construct an approximate null space. The paper does not yet fully explain the extent to which the actual increase in degrees of freedom comes from numerical approximation, and the impact of this approximation on stability guarantees.
  • The existing ablation studies have not adequately isolated the contribution of attention-aware constraints themselves, and performance improvements may also be affected by the parameterization method and the actual update rank. Experimental verification of the core mechanism is still insufficient.
Questions for authors
  • Could you further analyze the dimensional differences between the exact null space and the approximate null space, and explain the impact of different spectral energy thresholds on the update degrees of freedom, historical task stability, and final performance?
  • Could you directly compare the traditional input null space constraints with the proposed attention-aware constraints, given the matching number of trainable parameters, actual update rank, and training budget, to verify the independent contribution of the core design?
5 comments
AuthorsOpenReview ↗

Response to Reviewer iGFz

We sincerely thank the reviewer for recognizing the theoretical motivation and for identifying the need to strengthen the connection between theory and implementation and the validation of the core mechanism. We address your concerns point by point below.

Q1: Exact versus approximate null spaces and the effect of spectral thresholds

A1.1 (Exact versus approximate null spaces). In practice, accumulated feature matrices can have full column rank, leaving only the zero vector in the exact null space. We follow the spectral approximation strategy used in prior null-space continual learning work [1–3] and select low-response directions with a spectral-energy threshold. We have expanded the explanation of spectral selection in Appendices B.1–B.2, and added spectral approximation and residual analyses in Appendix C.3. Under the illustrated cumulative eigenvalue-mass threshold of η=0.99\eta=0.99 and a rank cap of 48, the retained bases have relative residuals ∥XV∥F/∥X∥F\|XV\|_F/\|X\|_F ranging from 1.81×10−71.81\times10^{-7} to 0.09950.0995. These relative residuals are all below 0.1, indicating relatively small null-space approximation errors in practice.

A1.2 (Practical update freedom). To address your question about practical update freedom, we have added a post-truncation dimension comparison (Figure 6, p. 21). It compares LANS and InfLoRA using the same checkpoint-derived statistics, the same spectral threshold, and matched truncation caps for corresponding branches. As a result, LANS increases layer-wise effective dimensions by 17.6%–86.0% with DeiT-S/16-IN611 and 31.9%–58.9% with ViT-B/16-IN21K. Thus, the additional flexibility is not confined to the untruncated theoretical space: it remains under the evaluated spectral thresholds and branch budgets.

A1.3 (Threshold sensitivity). Following your suggestion to examine different spectral-energy thresholds, we have added threshold sensitivity experiments in Figure 5 (p. 10), evaluating 10- and 20-task CIFAR-100 and ImageNet-R with DeiT-S/16-IN611. Its three rows show average forgetting, accuracy over all seen classes (with FAA at the endpoint), and relative dimension gains after rank truncation throughout the task sequence.

For 20-task ImageNet-R, the following table gives final-task results:

Spectral threshold ϵ\epsilonFinal forgetting (pp) ↓FAA (%) ↑Relative dimension gain after truncation (%)
0.819.6256.22+34.18
0.9511.9658.60+58.13
0.9998.7457.92+60.67

A larger threshold generally reduces forgetting but also restricts directions available for learning new tasks. Increasing ϵ\epsilon from 0.8 to 0.95 reduces final forgetting from 19.62 to 11.96 percentage points and improves FAA from 56.22% to 58.60%. Increasing it further to 0.999 reduces forgetting again but lowers FAA to 57.92%. Relative dimension gains remain positive across thresholds but are not monotonic in the threshold. We use ϵ=0.95\epsilon=0.95 to balance stability and plasticity.

Q2: Comparison with input-null-space constraints under matched budgets

We agree that an ablation against ordinary LoRA alone does not isolate the contribution of attention-aware constraints. We therefore added Table 5 (p. 10), comparing traditional input null space constraints (InfLoRA) and our LANS across different benchmarks at three different budgets.

The following table reports FAA / CAA differences in percentage points, computed as LANS−InfLoRALANS- InfLoRA.

BudgetDeiT-S: CIFAR-100DeiT-S: ImageNet-RViT-B budgetViT-B: CIFAR-100ViT-B: ImageNet-R
1×, 0.332M+3.01 / +3.15+11.44 / +9.631×, 0.664M+0.60 / +0.48−0.10 / −1.50
2×, 0.664M+4.26 / +3.69+11.81 / +11.232×, 1.327M+0.87 / +0.75+1.58 / −0.76
4×, 1.327M+5.21 / +4.51+11.50 / +11.234×, 2.654M+1.13 / +0.87+5.37 / +0.88

With DeiT-S/16-IN611, LANS improves both FAA and CAA on both datasets at all three budgets. With ViT-B/16-IN21K, it improves CIFAR-100 results at every budget. On ImageNet-R, the outcome depends on the budget: FAA/CAA are lower by 0.10/1.50 points at the lowest budget; FAA is higher by 1.58 points and CAA lower by 0.76 points at the middle budget; both improve by 5.37/0.88 points at the highest budget.

These matched-budget comparisons support the role of attention-aware constraints in preserving update freedom and alleviating the plasticity restrictions of input-null-space constraints.

References

[1] Wang et al. Training Networks in Null Space of Feature Covariance for Continual Learning. CVPR, 2021.

[2] Liang and Li. InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learning. CVPR, 2024.

[3] Lu et al. Visual Prompt Tuning in Null Space for Continual Learning. NeurIPS, 2024.

AuthorsOpenReview ↗

Official Comment by Authors

We sincerely thank you for your valuable comments, which we found very constructive for improving the rigor and quality of our paper. We have tried our best to address your concerns and would like to know whether our responses have addressed them. If anything remains unclear, we are more than happy to discuss it further.

As the rebuttal discussion period is coming to a close, we would greatly appreciate your feedback on whether our responses have addressed your concerns. We would be glad to clarify any remaining points.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers

We sincerely thank all three reviewers for their thorough reviews and constructive feedback. We appreciate their recognition LANS's innovations in the theoretical motivation, practical deployment advantage, and the effectiveness through multiple continual learning benchmarks.

Key Improvements and Clarifications

1. Exact and Approximate Null Spaces in Practice (iGFz, CnrW, 49Sp)

  • Clarified how spectral selection and rank truncation determine the update directions used in training, and quantified the approximation error of the retained directions (Appendices B.1–B.2 and Figure 8).
  • Showed that the dimension advantage persists after truncation. At the start of task 10 on ImageNet-R, LANS retains 17.6%–86.0% more effective update dimensions than InfLoRA-form across layers with DeiT-S and 31.9%–58.9% more with ViT-B (Figure 6).
  • Added a spectral-threshold sensitivity analysis requested by iGFz and a representation drift comparison with InfLoRA requested by CnrW (Figures 5 and 9).

2. Calibration and Matched-Budget Comparisons (iGFz, CnrW, 49Sp)

  • Added InfLoRA+CA to compare LANS and InfLoRA under the same classifier alignment protocol (Tables 1–3).
  • Compared LANS with InfLoRA at three matched trainable parameter budgets using the same calibration protocol and number of training epochs (Table 5).

3. Competitiveness (49Sp)

  • Added SplitLoRA and SD-LoRA as baselines (Tables 1–3).
  • Additionaly compared LANS with GR-LoRA and LoDA across five pretraining settings (Appendix C.6).

4. Practical Efficiency (49Sp)

  • Measured training time, peak GPU memory, additional training and inference parameters, and inference throughput under common hardware and training settings (Appendix C.6, Table 10). We also clarified how we count effective data passes and distinguish training time from end-to-end runtime.

The detailed responses appear under each review, and revisions are highlighted in red in the manuscript. We thank the reviewers again for helping us improve our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.