- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes LANS for class-incremental learning with pretrained ViTs. It replaces layerwise input null-space constraints with attention-aware constraints: V updates preserve attention-weighted outputs, while Q/K updates preserve first-order attention scores. It uses fixed subspace bases and trainable coefficients to enable mergeable low-rank updates. The paper derives the feasible space and its dimensionality gains.
Critical strengths
1.The characterization of the Q/K update space provides a concrete technical contribution. Expressing the solution to the two-sided constraint as the sum of two null spaces yields both verifiable dimensionality conditions and a direct training parameterization.
2.The experimental gains in more challenging settings are compelling. In the 50-task DeiT-S setting, LANS leads the strongest listed baselines in FAA by 7.56 and 5.10 percentage points on CIFAR-100 and ImageNet-R, respectively.
Critical weaknesses
-
The comparison with the most relevant baseline is subject to a key confound. LANS uses classifier alignment (CA), but the main table’s ImageNet-R results for InfLoRA, 75.65/80.82, correspond to the version without CA in the original paper; the InfLoRA paper reports 79.78/83.38 with CA. Results across papers cannot be used to directly determine which method is better, but this means the claimed gains over InfLoRA cannot yet be attributed to null-space expansion. The standard LoRA ablation in Table 4 also cannot substitute for an input null-space comparison under the same calibration and parameter budget.
-
The experimental performance is not competitive enough: on ImageNet-R at t=20, it is only around 77, and comparisons with recent state-of-the-art methods are lacking.
-
The theoretical gains in the space have not yet been adequately linked to the space used in actual training. The dimensionality result concerns the exact null space and explicitly applies before rank truncation; the implementation, however, uses a spectrally thresholded approximation to the null space and then truncates it further. After these operations, the original space-inclusion relationship and dimensionality advantage do not automatically hold.
Questions for authors
The main issue is that I’m not impressed by how the paper’s method performs in the experiments.
16 comments
Response to Reviewer 49Sp (Part 2/2)
Q2.2: Competitiveness and experimental settings
Overall competitiveness. LANS leads FAA and CAA on ViT-B CIFAR-100 and CUB-200 (Table 1, p. 7), but its ImageNet-R gains are limited. Its advantages are clearer with DeiT-S: it leads all six metrics in Table 2 (p. 7), with FAA/CAA gains of 3.17/2.45, 1.09/3.61, and 1.92/3.65 points over the strongest competing result per metric on CIFAR-100, ImageNet-R, and CUB-200. It also leads 15 of 16 long-sequence metrics across both backbones (Table 3, p. 8). Preserving update freedom may be especially useful when limited pre-training or model capacity requires greater adaptation.
Interpreting the limited advantage on ImageNet-R with ViT-B/IN21K. We examine two factors that may explain the gap to SplitLoRA+CA, the strongest baseline on 10-task ImageNet-R in Table 1.
(1) Experimental settings. SplitLoRA's official script uses IN21K pre-training followed by IN1K fine-tuning, whereas ours uses IN21K pre-training only; its default task order also differs. We therefore reproduce SplitLoRA with our checkpoint, data splits, and task order.
Originally reported SplitLoRA results (Qiu et al., 2026): CIFAR-100 90.33/93.70; ImageNet-R 81.00/85.84 FAA/CAA (%). These use different settings and are excluded from best-result comparisons.
Aligned 10-task results (Table 1, p. 7; mean FAA/CAA %; best values in bold):
| Method | CIFAR-100 | ImageNet-R |
|---|---|---|
| SplitLoRA | 89.27/92.89 | 78.26/82.81 |
| SplitLoRA+CA | 90.55/93.50 | 79.36/83.92 |
| LANS | 92.02/94.91 | 79.05/83.28 |
With the same CA strategy, LANS leads on CIFAR-100 and trails SplitLoRA+CA by only 0.31/0.64 points on ImageNet-R. These aligned results support its competitiveness without implying an advantage in every setting.
(2) Pre-training overlap and stability mechanisms. We hypothesize that category or distribution overlap between ImageNet-R and ImageNet-21K may reduce adaptation needs, favoring strategies that preserve pre-trained representations. SplitLoRA scales a LoRA factor's initialization magnitude by in subsequent tasks. Adding similar task-dependent scaling to LANS yields the following ImageNet-R results with ViT-B/16-IN21K (Appendix C.5, Table 6, p. 24). All methods use CA; LANS uses . Entries are means; bold marks the best within each task count.
| Tasks | Method | FAA (%) | CAA (%) |
|---|---|---|---|
| 10 | SplitLoRA+CA | 79.36 | 83.92 |
| 10 | LANS | 79.05 | 83.28 |
| 10 | LANS + task scaling | 81.02 | 85.18 |
| 20 | SplitLoRA+CA | 78.07 | 82.33 |
| 20 | LANS | 77.38 | 83.15 |
| 20 | LANS + task scaling | 78.93 | 84.39 |
| 50 | SplitLoRA+CA | 69.80 | 74.06 |
| 50 | LANS | 71.37 | 77.85 |
| 50 | LANS + task scaling | 72.23 | 79.18 |
Scaling improves LANS at all three sequence lengths, exceeding SplitLoRA+CA in both metrics. This supports its compatibility with LANS's attention-aware constraints.
These results are reported only in the appendix; Tables 1–3 retain default LANS without task scaling. We use a consistent default approach to assess the core method across task settings, rather than add a strategy based on a particular benchmark. Since scaling has been evaluated only in the settings above, we do not adopt it as a default component of the main experiments.
Q3: Connecting theoretical and practical update spaces
Following your feedback, we added implementation details and experiments linking theoretical flexibility to practical updates.
Approximation and truncation. Appendices B.1–B.2 (pp. 17–19) clarify spectral selection, truncation, and zero padding. Low-response directions satisfy the constraints approximately. Excess directions are truncated by retaining those with the lowest responses; zero padding adds no effective directions. Appendix C.3 and Figure 8 (p. 22) visualize this process and quantify the retained basis's relative residual.
Post-truncation dimensions. Figure 6 (p. 21; analysis in Appendix C.3, p. 22) compares LANS and InfLoRA-form using the same checkpoint statistics, spectral threshold, and matched branch caps. At task 10 on ImageNet-R, LANS retains 17.6%–86.0% larger layer-wise effective dimensions with DeiT-S and 31.9%–58.9% with ViT-B.
Empirical validation. We also added threshold sensitivity (Figure 5, p. 10), representation drift (Appendix C.4 and Figure 9, p. 23), and matched-CA/budget comparisons (Table 5, p. 10). Together, these support additional practical flexibility and competitive stability in the evaluated settings.
Thank you again for helping improve our manuscript.
References
Qiu et al. SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting. ICLR, 2026.
Wu et al. SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning. ICLR, 2025.
Official Comment by Authors
We sincerely thank you for your valuable comments, which we found very constructive for improving the rigor and quality of our paper. We have tried our best to address your concerns and would like to know whether our responses have addressed them. If anything remains unclear, we are more than happy to discuss it further.
Response to Reviewer 49Sp (Part 1/2)
We thank the reviewer for recognizing LANS's contributions and for the constructive feedback. We try our best to address your questions as follows.
Q1.1: Calibration protocol
A1.1 Original InfLoRA results omitted CA, whereas LANS used it. We added InfLoRA+CA with the same CA strategy and clarified the protocol in Appendix C.1 (pp. 19–20). LANS retains these FAA/CAA gains (percentage points; Tables 1–2, p. 7):
Long-task results (Table 3, p. 8) also favor LANS. Entries give FAA/CAA (%); C100 and IN-R denote CIFAR-100 and ImageNet-R. Bold marks the better result within each setting.
Q1.2: Matched parameter budgets
A1.2 Table 5 (p. 10) extends comparisons between InfLoRA and LANS. Both methods use the same training epochs and CA strategy, with matched parameter counts and comparable actual update ranks. InfLoRA ranks 24/48/96 correspond to LANS ranks 12/24/48.
Below are LANS minus InfLoRA (QKV) FAA/CAA differences (points). 1× is each backbone's lowest budget; M denotes millions of parameters.
LANS improves both metrics across DeiT-S budgets and on ViT-B CIFAR-100. ViT-B ImageNet-R gains depend on budget, reaching 5.37/0.88 points at 4×. These results support consistent benefits with different budgets.
Q2.1: Comparisons with recent methods
We added SplitLoRA (Qiu et al., 2026), SplitLoRA+CA, and SD-LoRA+CA (Wu et al., 2025). Results are from Tables 1–3 (pp. 7–8). Cells report FAA/CAA (%); CUB denotes CUB-200. Bold marks the best among the listed methods for each metric. Analysis and references follow in Part 2.
ViT-B/16-IN21K, 10 tasks (Table 1).
DeiT-S/16-IN611, 10 tasks (Table 2).
ViT-B/16-IN21K, 20/50 tasks (Table 3).
DeiT-S/16-IN611, 20/50 tasks (Table 3).