- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TIP-LoRA for exemplar-free class-incremental learning with a pretrained ViT. Its stability branch projects LoRA updates onto the complements of input- and output-side subspaces estimated from cumulative historical updates. A separate plasticity branch adapts within leading singular subspaces of the pretrained weights. The paper motivates this design through a local analysis of old-task loss changes and evaluates it across seven settings on five benchmarks, with branch ablations and interference diagnostics.
Critical strengths
The local analysis provides a reasonable motivation for considering both layer inputs and backpropagated loss responses when controlling interference. The controlled diagnostics and equal-total-rank ablations also help establish the value of the proposed components within the evaluated setup. These are useful aspects of the paper, although they do not resolve its positioning against existing methods.
Critical weaknesses
1. The literature review misses directly relevant work, weakening the claimed methodological contribution.
The omission of E²-LoRA [1] is particularly consequential. E²-LoRA already studies interference through output feature drift and constructs an SVD-based spectral space for LoRA orthogonalization. This is directly relevant to a paper motivated by the limitations of input-side constraints. However, TIP-LoRA neither discusses this connection nor compares against E²-LoRA.
The paper therefore does not adequately establish what its two-sided projection and cumulative-update subspace estimation contribute beyond this closely related approach. The local analysis and two-branch construction need to be positioned against E²-LoRA, rather than assessed only relative to the selected baselines. Missing TUNA [2], GOD [3], and MOS [4] further limits the coverage of relevant pretrained-model-based continual-learning methods. These omissions affect the assessment of both novelty and significance.
2. Several omitted methods report higher accuracy, so the state-of-the-art positioning is not supported.
The abstract emphasizes the highest final accuracy across all seven evaluated settings, but this conclusion depends on an incomplete selection of baselines. The following comparison uses reported final accuracy (%) for 10-task settings:
| Benchmark | TIP-LoRA | Omitted method | Reported accuracy | Difference |
|---|---|---|---|---|
| CIFAR-100 | 88.44 | E²-LoRA | 92.13 | +3.69 |
| ImageNet-R | 79.61 | E²-LoRA | 82.77 | +3.16 |
| CUB-200 | 78.95 | E²-LoRA | 89.77 | +10.82 |
| ImageNet-A | 57.56 | TUNA | 64.78 | +7.22 |
| ImageNet-A | 57.56 | GOD | 59.38 | +1.82 |
| ImageNet-A | 57.56 | MOS | 60.04 | +2.48 |
These results materially change the picture of empirical competitiveness. In particular, E²-LoRA reports higher accuracy on three shared benchmarks, while several omitted methods report higher ImageNet-A accuracy. The paper needs to address these results and establish its advantage under comparable conditions. Leading the selected comparison tables is insufficient to establish state-of-the-art performance. If the main advantage is instead efficiency, that trade-off also needs to include these stronger methods.
References:
- [1] E²-LoRA, Energy-Structured Low-Rank Adaptation for Continual Learning: https://arxiv.org/abs/2605.27482 — https://github.com/kiddo127/E2-LoRA
- [2] TUNA, Integrating Task-Specific and Universal Adapters for Pre-Trained Model-based Class-Incremental Learning: https://arxiv.org/abs/2508.08165
- [3] GOD, Geometry-driven OOD Detectors Are Class-Incremental Learners: https://github.com/Wangwang-Jia/GOD
- [4] MOS, Model Surgery for Pre-Trained Model-Based Class-Incremental Learning: https://arxiv.org/abs/2412.09441 — https://github.com/sun-hailong/AAAI25-MOS
Questions for authors
- How does TIP-LoRA advance beyond E²-LoRA’s output-feature-drift-based orthogonalization? Please explain the distinction in the protected subspaces, the interference being controlled, and the resulting practical benefit.
- How do the authors reconcile the performance positioning with the higher reported results above? What evidence establishes an accuracy or efficiency advantage over these omitted methods under comparable conditions?
2 comments
Response to Reviewer kAsx (2/2)
W2 & Q2. How does TIP-LoRA compare with the additional baselines?
The expanded comparison shows that enhanced TIP-LoRA improves both Last and Avg over our reproduction of E²-LoRA across all three benchmarks in our three-run evaluation. The main experiments use parameter-efficient baselines such as InfLoRA, SD-LoRA, and BiLoRA to examine the end-to-end effect of LoRA projection within a simple class-incremental training pipeline. We further evaluate enhanced TIP-LoRA to establish its performance with complementary training components.
E²-LoRA supplements its LoRA update with self-distillation and classifier alignment (CA). Enhanced TIP-LoRA adds two common components, CA and patch-level feature alignment (PA). PA aligns local backbone features with the previous model on current-task images, complementing the protection applied to the backbone updates.
The gains over E²-LoRA hold in all nine corresponding runs, and enhanced TIP-LoRA achieves the highest Last accuracy across all four benchmarks in the table.
| Method (Last / Avg) | CIFAR-100 10T (%) ↑ | CUB-200 10T (%) ↑ | ImageNet-R 10T (%) ↑ | ImageNet-A 10T (%) ↑ |
|---|---|---|---|---|
| Enhanced TIP-LoRA | 91.85 ± 0.13 / 94.82 ± 0.21 | 90.26 ± 0.05 / 93.15 ± 0.75 | 83.18 ± 0.09 / 87.21 ± 0.37 | 66.95 ± 0.75 / 74.49 ± 1.70 |
| E²-LoRA (our reproduction) | 91.23 ± 0.34 / 94.34 ± 0.43 | 88.93 ± 0.55 / 92.46 ± 0.46 | 82.81 ± 0.15 / 86.98 ± 0.45 | — |
| TUNA | 91.79 ± 0.15 / 94.88 ± 0.06 | 88.40 ± 0.42 / 92.01 ± 0.68 | 79.44 ± 0.38 / 84.80 ± 0.37 | 64.78 / — |
| MOS | 90.53 ± 0.35 / 94.09 ± 0.91 | 89.91 ± 0.25 / 92.87 ± 0.56 | 78.10 ± 0.12 / 83.61 ± 0.40 | 60.04 / — |
| GOD | — | — | — | 59.38 / — |
| Gain over reproduced E²-LoRA (pp) | +0.61 / +0.48 | +1.33 / +0.70 | +0.37 / +0.23 | — |
TIP-LoRA and our E²-LoRA reproduction ort mean ± sample SD over three runs; gains use unrounded means. TUNA, MOS, and GOD follow the results listed in the review with uncertainty where reported. “—” marks entries not included here. Bold denotes the best value among the displayed entries.
On ImageNet-A, enhanced TIP-LoRA exceeds all three results cited by the reviewer: Last accuracy improves by 2.17 points over TUNA, 6.91 over MOS, and 7.57 over GOD. It also exceeds E²-LoRA’s reported Last accuracy by 0.49 and 0.41 points on CUB-200 and ImageNet-R 10T, respectively, and achieves comparable CIFAR-100 accuracy (91.85% versus 92.13%). We will name the comparison set in the abstract.
Response to Reviewer kAsx (1/2)
Thank you for asking us to clarify the related-work coverage, the distinction from E²-LoRA, and the resulting practical benefit.
W1 & Q1. How does TIP-LoRA differ from E²-LoRA, and what practical benefit does it provide?
Regarding “missing directly relevant work,” we respectfully disagree that the omitted E²-LoRA discussion diminishes TIP-LoRA’s methodological contribution. The submitted manuscript already cites GPM, InfLoRA, SD-LoRA, BiLoRA, LoDA, and SplitLoRA, covering directly relevant work on projection, low-rank adaptation, and stability–plasticity design. TIP-LoRA retains a distinct technical advance when E²-LoRA is included in the comparison: it derives paired input–loss-response protection from the old-task loss and jointly estimates the two protection subspaces from cumulative updates. We will add an explicit E²-LoRA citation; the following comparison details the differences in protection targets, subspace construction, and practical benefits.
Our advance is to connect output-side protection directly to the old-task loss and enforce it jointly with input-side protection. E²-LoRA uses the principal directions of output drift ΔWtXt on proxy features to preserve drift energy under rank truncation and recycle low-energy ranks. TIP-LoRA identifies the interaction that determines first-order interference: e⊤ΔWh, where h is the old-task layer input and e is the backpropagated loss response. Output perturbations of equal magnitude can have different loss effects because their alignment with e differs. This analysis identifies a paired protection target: the historical input directions that activate an update and the historical response directions that make its output consequential for the loss. We turn this target into an explicit two-sided constraint on the stability update.
A second advance is estimating both protection subspaces from cumulative learned updates. The gradient factorization ∇Wℓ=eh⊤ explains how input and loss-response signals enter weight updates as pairs. TIP-LoRA uses the left and right singular directions of Mt−1=∑τ<tΔWτ to estimate these paired historical spaces and constructs the stability update as ΔWs=(I−UU⊤)ΔW(I−VV⊤). Thus, a single decomposition of the accumulated Q/V update supplies both sides of the constraint directly from the learning history. This links the old-loss analysis to a concrete projection rule without collecting proxy features for projector construction. A separate branch uses fixed pretrained-weight subspaces for plasticity.
Practical benefit. The enhanced TIP-LoRA model, combining this projection with CA and PA, achieves higher Last and Avg than our E²-LoRA runs in all nine corresponding runs across three benchmarks. The mean Last / Avg gains are +0.61 / +0.48 points on CIFAR-100, +1.33 / +0.70 on CUB-200, and +0.37 / +0.23 on ImageNet-R 10T. For the additional methods raised in the review, enhanced TIP-LoRA reaches 66.95 ± 0.75% Last accuracy on ImageNet-A, exceeding the cited results of TUNA, MOS, and GOD by 2.17, 6.91, and 7.57 percentage points, respectively. Q2 reports the three-run means, standard deviations, and component settings.