acceptodds
Under review as a conference paper at ICLR 2027

TIP-LoRA:Two-Sided Interference-Aware Projection for Exemplar-Free Class-Incremental Learning

Abstract

Continual learning (CL) requires a model to acquire new knowledge without forgetting old knowledge, yet low-rank adaptation (LoRA) applied to pre-trained models remains vulnerable to interference from sequential updates. Existing LoRA-based CL methods constrain an update from the input side; with a finite-rank subspace, however, residual output changes can remain, and their interference also depends on the loss response. In this paper, we propose Two-Sided Interference-Aware Projection LoRA (TIP-LoRA), a two-branch framework that controls this interference through paired input- and output-side projections. A local Taylor analysis shows that first-order interference is jointly determined by the layer input and the backpropagated loss response. This analysis suggests controlling both the input- and output-side directions of a new update. Since old-task signals are unavailable, the stability branch estimates these directions from cumulative task updates, and restricts new updates to their two-sided orthogonal complement. In parallel, the plasticity branch learns within the leading singular subspaces of the frozen pre-trained weights to retain transferable structure for new knowledge. We further provide a local bound on the old-task loss change under the projected update. Across seven settings on five benchmarks, TIP-LoRA achieves the highest final accuracy in all seven settings and competitive average incremental accuracy. Controlled diagnostics further show that two-sided projection removes residual interference left by input-only control and that cumulative updates retain the paired directions used by the stability branch.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Parameter-Efficient Fine-Tuning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_kAsxOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes TIP-LoRA for exemplar-free class-incremental learning with a pretrained ViT. Its stability branch projects LoRA updates onto the complements of input- and output-side subspaces estimated from cumulative historical updates. A separate plasticity branch adapts within leading singular subspaces of the pretrained weights. The paper motivates this design through a local analysis of old-task loss changes and evaluates it across seven settings on five benchmarks, with branch ablations and interference diagnostics.

Critical strengths

The local analysis provides a reasonable motivation for considering both layer inputs and backpropagated loss responses when controlling interference. The controlled diagnostics and equal-total-rank ablations also help establish the value of the proposed components within the evaluated setup. These are useful aspects of the paper, although they do not resolve its positioning against existing methods.

Critical weaknesses

1. The literature review misses directly relevant work, weakening the claimed methodological contribution.

The omission of E²-LoRA [1] is particularly consequential. E²-LoRA already studies interference through output feature drift and constructs an SVD-based spectral space for LoRA orthogonalization. This is directly relevant to a paper motivated by the limitations of input-side constraints. However, TIP-LoRA neither discusses this connection nor compares against E²-LoRA.

The paper therefore does not adequately establish what its two-sided projection and cumulative-update subspace estimation contribute beyond this closely related approach. The local analysis and two-branch construction need to be positioned against E²-LoRA, rather than assessed only relative to the selected baselines. Missing TUNA [2], GOD [3], and MOS [4] further limits the coverage of relevant pretrained-model-based continual-learning methods. These omissions affect the assessment of both novelty and significance.

2. Several omitted methods report higher accuracy, so the state-of-the-art positioning is not supported.

The abstract emphasizes the highest final accuracy across all seven evaluated settings, but this conclusion depends on an incomplete selection of baselines. The following comparison uses reported final accuracy (%) for 10-task settings:

BenchmarkTIP-LoRAOmitted methodReported accuracyDifference
CIFAR-10088.44E²-LoRA92.13+3.69
ImageNet-R79.61E²-LoRA82.77+3.16
CUB-20078.95E²-LoRA89.77+10.82
ImageNet-A57.56TUNA64.78+7.22
ImageNet-A57.56GOD59.38+1.82
ImageNet-A57.56MOS60.04+2.48

These results materially change the picture of empirical competitiveness. In particular, E²-LoRA reports higher accuracy on three shared benchmarks, while several omitted methods report higher ImageNet-A accuracy. The paper needs to address these results and establish its advantage under comparable conditions. Leading the selected comparison tables is insufficient to establish state-of-the-art performance. If the main advantage is instead efficiency, that trade-off also needs to include these stronger methods.

References:

Questions for authors
  1. How does TIP-LoRA advance beyond E²-LoRA’s output-feature-drift-based orthogonalization? Please explain the distinction in the protected subspaces, the interference being controlled, and the resulting practical benefit.
  2. How do the authors reconcile the performance positioning with the higher reported results above? What evidence establishes an accuracy or efficiency advantage over these omitted methods under comparable conditions?
2 comments
AuthorsOpenReview ↗

Response to Reviewer kAsx (1/2)

Thank you for asking us to clarify the related-work coverage, the distinction from E²-LoRA, and the resulting practical benefit.

W1 & Q1. How does TIP-LoRA differ from E²-LoRA, and what practical benefit does it provide?

Regarding “missing directly relevant work,” we respectfully disagree that the omitted E²-LoRA discussion diminishes TIP-LoRA’s methodological contribution. The submitted manuscript already cites GPM, InfLoRA, SD-LoRA, BiLoRA, LoDA, and SplitLoRA, covering directly relevant work on projection, low-rank adaptation, and stability–plasticity design. TIP-LoRA retains a distinct technical advance when E²-LoRA is included in the comparison: it derives paired input–loss-response protection from the old-task loss and jointly estimates the two protection subspaces from cumulative updates. We will add an explicit E²-LoRA citation; the following comparison details the differences in protection targets, subspace construction, and practical benefits.

Our advance is to connect output-side protection directly to the old-task loss and enforce it jointly with input-side protection. E²-LoRA uses the principal directions of output drift ΔWtXt\Delta W_t X_t on proxy features to preserve drift energy under rank truncation and recycle low-energy ranks. TIP-LoRA identifies the interaction that determines first-order interference: e⊤ΔWhe^\top\Delta W h, where hh is the old-task layer input and ee is the backpropagated loss response. Output perturbations of equal magnitude can have different loss effects because their alignment with ee differs. This analysis identifies a paired protection target: the historical input directions that activate an update and the historical response directions that make its output consequential for the loss. We turn this target into an explicit two-sided constraint on the stability update.

A second advance is estimating both protection subspaces from cumulative learned updates. The gradient factorization ∇Wℓ=eh⊤\nabla_W\ell=e h^\top explains how input and loss-response signals enter weight updates as pairs. TIP-LoRA uses the left and right singular directions of Mt−1=∑τ<tΔWτM_{t-1}=\sum_{\tau<t}\Delta W_\tau to estimate these paired historical spaces and constructs the stability update as ΔWs=(I−UU⊤)ΔW(I−VV⊤)\Delta W_s=(I-UU^\top)\Delta W(I-VV^\top). Thus, a single decomposition of the accumulated Q/V update supplies both sides of the constraint directly from the learning history. This links the old-loss analysis to a concrete projection rule without collecting proxy features for projector construction. A separate branch uses fixed pretrained-weight subspaces for plasticity.

Practical benefit. The enhanced TIP-LoRA model, combining this projection with CA and PA, achieves higher Last and Avg than our E²-LoRA runs in all nine corresponding runs across three benchmarks. The mean Last / Avg gains are +0.61 / +0.48 points on CIFAR-100, +1.33 / +0.70 on CUB-200, and +0.37 / +0.23 on ImageNet-R 10T. For the additional methods raised in the review, enhanced TIP-LoRA reaches 66.95 ± 0.75% Last accuracy on ImageNet-A, exceeding the cited results of TUNA, MOS, and GOD by 2.17, 6.91, and 7.57 percentage points, respectively. Q2 reports the three-run means, standard deviations, and component settings.

AuthorsOpenReview ↗

Response to Reviewer kAsx (2/2)

W2 & Q2. How does TIP-LoRA compare with the additional baselines?

The expanded comparison shows that enhanced TIP-LoRA improves both Last and Avg over our reproduction of E²-LoRA across all three benchmarks in our three-run evaluation. The main experiments use parameter-efficient baselines such as InfLoRA, SD-LoRA, and BiLoRA to examine the end-to-end effect of LoRA projection within a simple class-incremental training pipeline. We further evaluate enhanced TIP-LoRA to establish its performance with complementary training components.

E²-LoRA supplements its LoRA update with self-distillation and classifier alignment (CA). Enhanced TIP-LoRA adds two common components, CA and patch-level feature alignment (PA). PA aligns local backbone features with the previous model on current-task images, complementing the protection applied to the backbone updates.

The gains over E²-LoRA hold in all nine corresponding runs, and enhanced TIP-LoRA achieves the highest Last accuracy across all four benchmarks in the table.

Method (Last / Avg)CIFAR-100 10T (%) ↑CUB-200 10T (%) ↑ImageNet-R 10T (%) ↑ImageNet-A 10T (%) ↑
Enhanced TIP-LoRA91.85 ± 0.13 / 94.82 ± 0.2190.26 ± 0.05 / 93.15 ± 0.7583.18 ± 0.09 / 87.21 ± 0.3766.95 ± 0.75 / 74.49 ± 1.70
E²-LoRA (our reproduction)91.23 ± 0.34 / 94.34 ± 0.4388.93 ± 0.55 / 92.46 ± 0.4682.81 ± 0.15 / 86.98 ± 0.45—
TUNA91.79 ± 0.15 / 94.88 ± 0.0688.40 ± 0.42 / 92.01 ± 0.6879.44 ± 0.38 / 84.80 ± 0.3764.78 / —
MOS90.53 ± 0.35 / 94.09 ± 0.9189.91 ± 0.25 / 92.87 ± 0.5678.10 ± 0.12 / 83.61 ± 0.4060.04 / —
GOD———59.38 / —
Gain over reproduced E²-LoRA (pp)+0.61 / +0.48+1.33 / +0.70+0.37 / +0.23—

TIP-LoRA and our E²-LoRA reproduction ort mean ± sample SD over three runs; gains use unrounded means. TUNA, MOS, and GOD follow the results listed in the review with uncertainty where reported. “—” marks entries not included here. Bold denotes the best value among the displayed entries.

On ImageNet-A, enhanced TIP-LoRA exceeds all three results cited by the reviewer: Last accuracy improves by 2.17 points over TUNA, 6.91 over MOS, and 7.57 over GOD. It also exceeds E²-LoRA’s reported Last accuracy by 0.49 and 0.41 points on CUB-200 and ImageNet-R 10T, respectively, and achieves comparable CIFAR-100 accuracy (91.85% versus 92.13%). We will name the comparison set in the abstract.

Reviewer_T16hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TIP-LoRA combines a stability branch projected onto the left and right complements of cumulative-update singular subspaces with a plasticity branch restricted to leading pretrained-weight subspaces. A local Taylor analysis motivates controlling both input features and backpropagated loss responses. Across seven settings on five benchmarks, the method reports the highest final accuracy among the compared methods, with branch ablations, threshold sensitivity, and interference diagnostics.

Critical strengths
  1. The two-sided formulation has a clear analytical motivation. The local loss expansion exposes the roles of both layer inputs and output gradients, and the conditional residual bound explains what each projection controls. This provides a concrete design rationale for extending input-side protection.
  2. The empirical coverage is substantial. Three-run main results span five datasets and several task lengths, with consistent final-accuracy improvements. The equal-total-rank CIFAR-100 branch ablation also supports cooperation between stability and plasticity beyond simply adding rank (Figure 3; Appendix B.4).
Critical weaknesses
  1. The connection from historical updates to protected old-task knowledge remains unverified. The recovery argument requires Assumption 2's dominant paired component and small residual, including optimizer effects. Signed update coherence in Appendix D.2 does not measure alignment with old-task input/response subspaces or verify that spectral condition. Moreover, the local bound covers only the stability update, while the plasticity branch can modify the protected directions. The analysis therefore provides limited support for retention under the complete learned update.
  2. The key output-side mechanism is not isolated in actual continual training. Figure 4(a) constructs projectors from old test images, rather than the historical-update projectors used by TIP-LoRA. Adding an orthogonal left projection necessarily cannot increase the reported Frobenius residual energy, so this diagnostic alone does not show that the learned output subspace provides useful protection. Branch-removal ablations do not replace a controlled input-only versus two-sided comparison with the same plasticity branch and training budget.
Questions for authors
  1. Evidence needed for assessment: How closely do the actual cumulative-update subspaces align with held-out old-task input and loss-response subspaces across task boundaries, and how much old-task loss change comes from each branch and their sum?
  2. Causal clarification: With the actual historical projectors, identical total rank, plasticity branch, and training settings, how do input-only, two-sided, and rank-matched random-output projection compare in final accuracy and forgetting across seeds?
2 comments
AuthorsOpenReview ↗

Response to Reviewer T16h

Thank you for the thoughtful review and constructive suggestions.

W1 & Q1. How do actual historical subspaces relate to old-task knowledge, and how does each branch affect old-task loss?

At matched ranks and update norms, historical output bases retain 99.1% of the original cumulative displacement's old-task loss reduction, compared with 19.3% for random bases and 69.5% for pretrained bases.

On CUB-200 10T, we reconstruct all Q/V cumulative updates using different output bases, while fixing the historical input bases, actual ranks, and update norms. We evaluate the full model on 128 held-out old-task images at one task boundary in one training seed:

Output basisOld-task accuracy (%) ↑Old-task cross-entropy ↓Retained loss reduction (%) ↑
Actual historical90.630.310699.1
Same-rank random88.54 ± 0.450.3622 ± 0.006019.3 ± 9.3
Same-rank pretrained89.840.329869.5

Historical output bases recover the original accuracy of 90.63%, exceeding random bases by 2.08 percentage points. The last column measures the fraction of the original loss reduction recovered, using removal of the cumulative displacement as the baseline.

The two branches balance retention and adaptation through different update subspaces. At task 0, plasticity uses pretrained principal subspaces, while stability uses their two-sided orthogonal complement, yielding orthogonal updates. Subsequently, both branches contribute to the history used to refresh the stability projectors, while the plasticity bases remain fixed. Reconstruction assesses the predictive function of this combined history; The equal-total-rank ablation in Figure 3 and Appendix B.4 further supports the benefit of combining the two branches.

W2 & Q2. Do historical output directions and two-sided projection improve retention?

On ImageNet-R 10T, historical output projection outperforms same-rank random projection in both seeds: mean Last increases by 0.43 percentage points, forgetting decreases by 0.51 points, and final old-class accuracy increases by 0.51 points.

Historical and random output projections use matched plasticity settings, projection ranks, and training budgets. The input-only and output-only rows apply the side ablation to both branches. All variants use CA, zero PA weight, and a drift-compensation coefficient of 10−610^{-6}; results are mean ± SD over two seeds:

Projection settingComparison scopeLast (%) ↑Forgetting (pp) ↓Final old-class accuracy (%) ↑
Historical input + historical outputShared reference78.70 ± 0.1813.49 ± 0.1977.45 ± 0.20
Historical input + same-rank random outputMatched plasticity settings78.27 ± 0.2114.00 ± 0.1876.94 ± 0.18
Input onlyBoth branches varied77.83 ± 0.1814.65 ± 0.1076.51 ± 0.17
Output onlyBoth branches varied76.83 ± 0.4215.74 ± 0.4375.38 ± 0.45

Two-sided projection also achieves higher Last and lower forgetting than both one-sided variants in both seeds. These comparisons support the contribution of historical output directions and the benefits of the complete two-sided design, respectively.

Thanks for the responses. My concerns have been resolved. Therefore, I maintain my positive score.

Reviewer_6XXKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates interference in LoRA updates for exemplar-free class-incremental learning. The authors formulate the first-order change in the old-task loss as a bilinear coupling between layer inputs and backpropagated loss responses, and propose TIP-LoRA. The motivation is clear, and the method does not retain old-task samples. However, its novelty is insufficiently positioned, and the paper does not compare against closely related recent work.

Critical strengths
  1. The authors separately analyze stability and plasticity in LoRA updates and provide the necessary empirical evidence.

  2. The experiments cover a broad range of datasets and use three random seeds.

Critical weaknesses
  1. The novelty of the method requires further clarification. Null-space LoRA, two-sided low-rank bases, stability-plasticity decomposition, and the use of historical subspaces all have related precedents. Although the paper cites LoDA and SplitLoRA, it does not sufficiently explain the differences between these methods.

  2. The experimental results are not compared with recent work. The abstract explicitly claims that TIP-LoRA achieves the highest final accuracy in all seven settings, but the experiments do not include the accepted LoDA, which uses consistent settings, or the closely related SplitLoRA.

  3. The paper lacks an end-to-end ablation of the core “two-sided historical projection.” It compares the dual-branch model with the plasticity-only and stability-only variants, but does not distinguish between right-side, left-side, and two-sided projection within the stability branch.

##Questions That Need the Authors to Answer

  1. How does TIP-LoRA differ technically from recent related methods, particularly LoDA and SplitLoRA?

  2. Why are LoDA and SplitLoRA not included in the experimental comparison, despite the claim that TIP-LoRA achieves the highest final accuracy across all seven settings?

  3. Can the authors provide end-to-end ablations comparing right-side-only, left-side-only, and two-sided historical projection under the same rank and training settings?

References

[1]. Qiu H, Zhang M, Qiao Z, et al. Splitlora: Balancing stability and plasticity in continual learning through gradient space splitting[C]//International Conference on Learning Representations. 2026, 2026: 48458-48480.

Questions for authors

See Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 6XXK (1/2)

Thank you for the thoughtful review and for asking us to clarify the distinction from related work and the role of each projection side.

W1 & Q1. What does TIP-LoRA add beyond LoDA and SplitLoRA?

TIP-LoRA derives two-sided protection from the old-task loss and builds both projectors from accumulated LoRA updates, without collecting additional old-task information.

Compared with LoDA, we explicitly constrain loss-sensitive output directions. LoDA collects input feature statistics to select shared and isolated directions, then applies gradient alignment and recalibration. After finite-rank input projection, residual inputs can still change loss-sensitive outputs. For an old layer input hh and its backpropagated loss gradient ee, our stability update has the first-order effect

= \big[(I-UU^\top)e\big]^\top \Delta W\big[(I-VV^\top)h\big].$$ The right projection removes selected input components. The left removes output changes along protected loss-response directions. **SplitLoRA chooses the size of an update subspace. We ask which input and output directions jointly drive old-task interference.** SplitLoRA uses old- and new-task loss estimates and separately collected statistics. The squared norm in its bound measures update size. Freezing a LoRA factor restricts directions. We derive both constraints from $e^\top\Delta W h$. The paired gradient term $eh^\top$ motivates estimating both bases with one truncated SVD per cumulative Q/V update. Under Assumption 2, the leading left and right singular spaces approximate the dominant historical response and input spaces. A separate branch adapts in fixed pretrained subspaces. # **W2 & Q2. Why these baselines, and how does TIP-LoRA compare with LoDA and SplitLoRA?** We use parameter-efficient baselines to study LoRA projection under standard class-incremental training. On ImageNet-R 10T, TIP-LoRA improves both accuracy metrics over InfLoRA and SD-LoRA at the same inference cost of 35.12 GFLOPs (Table 3). We will specify the comparison scope of Tables 1–2 in the abstract. LoDA adds gradient alignment and update recalibration, while SplitLoRA balances stability and plasticity in subspace selection. For these comparisons, enhanced TIP-LoRA retains two-sided projection and adds two common components: classifier alignment (CA) and patch-level feature alignment (PA). **Across five settings on four datasets, enhanced TIP-LoRA optimizes 31.0%–84.1% fewer parameters on the first task and 53.9%–89.4% fewer per subsequent task than LoDA+CA.** We count adapter and current-classifier parameters actively optimized during the main training stage; each entry gives first / subsequent task. | Setting | Enhanced TIP-LoRA (M) ↓ | LoDA+CA (M) ↓ | Reduction (first / subsequent task) | | --- | ---: | ---: | ---: | | CUB-200 10T | **0.458 / 0.458** | 1.195 / 1.785 | 61.7% / 74.4% | | CIFAR-100 10T | **0.819 / 0.819** | 1.187 / 1.777 | 31.0% / 53.9% | | ImageNet-A 10T | **0.384 / 0.384** | 1.195 / 1.785 | 67.9% / 78.5% | | ImageNet-R 10T | **0.384 / 0.384** | 2.375 / 3.554 | 83.8% / 89.2% | | ImageNet-R 20T | **0.376 / 0.376** | 2.367 / 3.547 | 84.1% / 89.4% | **With substantially fewer training parameters, enhanced TIP-LoRA achieves comparable or better performance than LoDA+CA.** On ImageNet-R 10T, it optimizes **89.2% fewer** parameters per subsequent task while improving Last by **0.63 points** and also achieving higher Avg. On ImageNet-A 10T, it optimizes **78.5% fewer** parameters per subsequent task with higher Last and Avg. On CUB-200 10T, the reduction is **74.4%**, with Last / Avg differences of only **0.31 / 0.26 points**. Compared with SplitLoRA, enhanced TIP-LoRA achieves higher Last and Avg on CIFAR-100 10T and ImageNet-R 5T, 10T, and 20T, with Last gains of **1.52–2.68 points**. | Setting | Enhanced TIP-LoRA Last / Avg (%) | LoDA+CA Last / Avg (%) | SplitLoRA Last / Avg (%) | | --- | ---: | ---: | ---: | | CIFAR-100 10T | 91.85 ± 0.13 / 94.82 ± 0.21 | **92.29 ± 0.05 / 95.13 ± 0.17** | 90.33 ± 0.73 / 93.70 ± 0.32 | | CUB-200 10T | 90.26 ± 0.05 / 93.15 ± 0.75 | **90.57 ± 0.43 / 93.41 ± 0.44** | — | | ImageNet-A 10T | **66.57 ± 1.07 / 73.96 ± 1.48** | 66.54 ± 0.69 / 73.66 ± 1.42 | 58.22 / 66.11 | | ImageNet-R 5T | **83.88 ± 0.11** / 87.33 ± 0.34 | 83.69 ± 0.38 / **87.45 ± 0.42** | 81.92 ± 0.29 / 85.83 ± 0.55 | | ImageNet-R 10T | **83.18 ± 0.09 / 87.21 ± 0.37** | 82.55 ± 0.55 / 87.18 ± 0.42 | 81.00 ± 0.17 / 85.84 ± 0.62 | | ImageNet-R 20T | 81.50 ± 0.06 / 85.72 ± 0.37 | **81.70 ± 0.27 / 86.60 ± 0.31** | 78.82 ± 0.28 / 84.57 ± 0.44 | Multi-run results are mean ± sample SD. TIP-LoRA uses five runs on ImageNet-A and three on the other settings. In a separate CUB-200 10T control with LoDA’s rank set to 12 (seed 1993), enhanced TIP-LoRA and LoDA+CA optimize **0.458M and 0.679M** parameters per subsequent task, respectively. TIP-LoRA still uses **32.6% fewer** parameters, with Last accuracies of **90.29% and 90.54%**, respectively.
AuthorsOpenReview ↗

Response to Reviewer 6XXK (2/2)

W3 & Q3. What changes when we use one projection side?

Compared with the input-only and output-only settings, two-sided projection raises final old-class accuracy by 0.94 and 2.07 points on ImageNet-R and reduces forgetting by 1.16 and 2.54 points on ImageNet-A. W2 uses the full enhanced recipe. Here we keep CA and disable patch-level feature alignment (PA) to test how projection affects retention without patch supervision. We switch both branches together at fixed LoRA rank and budget, with a drift blend of 10−610^{-6}. Values are mean ± sample SD over paired seeds 1993–1994 (ImageNet-R) and 1993–1995 (ImageNet-A).

DatasetProjectionLast (%)Avg (%)Forgetting (pp)
ImageNet-R 10TTwo-sided78.70 ± 0.1884.57 ± 0.4813.49 ± 0.19
ImageNet-R 10TInput only77.83 ± 0.1884.22 ± 0.5214.65 ± 0.10
ImageNet-R 10TOutput only76.83 ± 0.4283.63 ± 0.2615.74 ± 0.43
ImageNet-A 10TTwo-sided64.69 ± 0.5072.73 ± 1.8611.22 ± 0.78
ImageNet-A 10TInput only64.34 ± 1.4572.70 ± 1.7712.38 ± 2.08
ImageNet-A 10TOutput only63.07 ± 0.8071.90 ± 1.9413.76 ± 0.94

Dear reviewer:

Thank you again for your tremendous effort and valuable feedback. We have responded carefully and thoroughly to your main questions and hope you are satisfied with our response. Given the limited time for rebuttal, we would like to know if you have any further concerns, which we would be happy to address. We look forward to receiving your further feedback.

Sincerely,

Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Response to Reviewer kAsx (1/2)

Dear Reviewer kAsx,

1. The submitted TIP-LoRA matches or exceeds E²-LoRA under both CUB protocols with substantially fewer trainable parameters

Main-stage trainable parameters on CUB, standard Caltech split, E²’s IN21K checkpointSubmitted TIP-LoRAE²-LoRA without KD or CAE² / TIP
First task, MLP adapters alone0141.56M—
First task, total0.458M226.51M494.9×
Tasks 2–10, mean per task0.458M18.73M40.9×
Final task0.458M8.22M18.0×
All ten tasks, mean per task0.458M39.51M86.3×

TIP adapts only the attention Q and V weights. E² adapts the joint QKV projection and the first MLP layer in every Transformer block. E²’s first-task MLP adapters alone contain 309× as many trainable parameters as TIP’s entire per-task budget, including the classifier. Even excluding the first task, E² trains 40.9× as many parameters per task on average.

The 78.95% and 81.80% cited in your assessment come from different checkpoints and data splits. We tested both methods under each protocol at seed 1993. Within each row, the methods share the checkpoint, split, and class order while retaining their respective training recipes. CA and PA are disabled for TIP, and KD and CA are disabled for E².

CUB-200 protocolTIP LastE² LastTIP AvgE² Avg
TIP checkpoint (IN21K→IN1K), submission split79.22%77.40%87.01%84.56%
E² checkpoint (IN21K only), standard Caltech split82.05%82.03%89.12%88.84%

Under our protocol, TIP leads by 1.82 points in Last and 2.46 in Avg (from unrounded values). Under E²’s protocol, the methods are level in Last, and TIP achieves a slightly higher Avg while training 1.16% as many parameters per task on average. Neither matched comparison shows a deficit that TIP needed the enhanced pipeline to reverse.

2. Classifier and feature alignment can yield large gains with a small trainable-parameter budget

LoDA (ICML 2026) provides a direct precedent on the same CUB ten-task benchmark. Across three seeds, adding classifier alignment (CA) alone raised LoDA’s mean Last accuracy from 81.74% to 90.67%, a gain of 8.93 percentage points. This shows that classifier calibration can materially improve final accuracy in a parameter-efficient LoRA system. We measured the contribution of CA and patch-level feature alignment (PA) to TIP directly.

On our original CUB protocol, the submitted TIP trained 0.458M parameters per task and achieved 78.95% mean Last. The CA+PA configuration reached 90.26%.

To separate the two components, we enabled CA and PA successively while holding the remaining settings fixed within each row.

Controlled ablation, seed 1993, Last accuracyBoth offCACA + PA
CUB, standard Caltech split81.96%87.31% (+5.35)90.26% (+2.95)
ImageNet-A, TIP checkpoint58.66%64.58% (+5.92)67.54% (+2.96)
AuthorsOpenReview ↗

Response to Reviewer kAsx (2/2)

3. Historical output directions preserve old-task performance, and both projection sides contribute to retention

On E²’s CUB checkpoint and split, full enhanced TIP reaches 89.63% Last / 93.18% Avg, compared with 88.73% / 92.87% for full E². To evaluate the projection itself, we performed two separate controls.

First, at one CUB task boundary, we reconstructed all 24 cumulative Q/V updates while holding the input bases, ranks, and update norms fixed. On 128 held-out old-task images, the historical output bases retained 99.1% of the reduction in old-task loss produced by the original updates, compared with 69.5% for pretrained bases and 19.3% for random bases. This measures how much of the original updates’ function each reconstruction preserves.

Second, we tested the effect of each projection side during subsequent learning. In end-to-end ImageNet-A training, we fixed the rank and training budget, held CA constant, and disabled PA. Mean forgetting across three seeds was 11.22 points with two-sided projection, 12.38 with input-only projection, and 13.76 with output-only projection. Removing either side increased forgetting.

4. The choice of baselines should reflect the research question.

The submitted TIP-LoRA studies how to constrain LoRA updates. Even E²-LoRA, the closest additional baseline, adapts the LoRA projection and trains an additional LoRA adapter in the first MLP layer of every Transformer block. This changes both where adaptation occurs and how much capacity it uses. Its MLP adapters alone train 141.56M parameters on the first task, and its total trainable budget averages 86.3× TIP’s across the ten Caltech tasks. We therefore report both complete-system results and comparisons of the original methods under shared checkpoints and splits.

TUNA and MOS study how to train, combine, and retrieve task-specific nonlinear bottleneck adapters. TUNA combines predictions from a selected task-specific adapter and a universal adapter. MOS merges adapters and refines their retrieval at inference. We included both comparisons as requested. Their scores measure complete systems with different adaptation and inference procedures. Those rankings alone cannot establish the relative merit of TIP’s LoRA projection rule. The submitted TIP achieved the highest final accuracy among the baselines evaluated in each of the seven settings. We will make that scope explicit in the abstract.

We ask you to reconsider the rating of 1 in light of this evidence. During the rebuttal period, we tested both CUB protocols, verified the per-task parameter budgets, separated CA and PA, and evaluated the projection under fixed-rank controls. If you maintain the rating, please identify the specific technical claim that remains unsupported and explain why the corresponding controlled experiment leaves it unresolved.

AuthorsOpenReview ↗

TIP-LoRA Rebuttal Summary

Evidence for TIP-LoRA’s two-sided projection and parameter efficiency

Dear PC, SAC, and AC,

Reviewer 6XXK asked how TIP differs from LoDA and SplitLoRA, how it compares with them, and whether both projection sides matter. TIP estimates both input and loss-sensitive output directions from accumulated LoRA updates.

  • Accuracy and parameter efficiency. On ImageNet-R 10T, enhanced TIP surpassed LoDA+CA in final accuracy (83.18% vs. 82.55%) and average accuracy (87.21% vs. 87.18%) while training 89.2% fewer parameters per subsequent task (0.384M vs. 3.554M). TIP also exceeded SplitLoRA on both metrics across four settings.
  • Contribution of both sides. At fixed rank on ImageNet-A, mean forgetting was 11.22 points with two-sided projection, 12.38 with input-only projection, and 13.76 with output-only projection.

We addressed all three questions and received no further reply after our follow-up.

Reviewer T16h asked whether TIP’s learned output directions preserve old-task performance. At one CUB task boundary, we held the input bases, ranks, and update norms fixed and varied only the output bases. On 128 held-out old-task images, the historical directions preserved 99.1% of the loss reduction achieved by the original updates, compared with 69.5% for pretrained directions and 19.3% for random directions. Historical directions also reduced forgetting relative to rank-matched random directions in both ImageNet-R training runs. T16h confirmed that their concerns were resolved and maintained their positive score.

Reviewer kAsx requested comparisons with methods that also change the adaptation architecture or training procedure. Our submission used a simple pipeline to isolate the effect of the LoRA projection rule. TUNA and MOS combine or retrieve task-specific adapters at inference. E²-LoRA extends adaptation from Q/V to the full QKV projection and the MLP, and its full system adds self-distillation and classifier alignment. We therefore evaluated both the underlying methods and the full systems.

  • Underlying methods. With enhancements disabled for both methods, the apparent CUB deficit disappeared under both matched protocols. Under TIP’s protocol, TIP led E² by 1.82 points in final and 2.46 points in average accuracy. Under E²’s own checkpoint and split, the results were:
CUB under E²’s protocolSubmitted TIPE²
Final accuracy82.05%82.03%
Average accuracy89.12%88.84%
Mean trainable parameters per task0.458M39.51M

E² trained 86.3× as many parameters per task. TIP had already matched its final accuracy and exceeded its average accuracy before adding either alignment component.

  • Full systems. We retained TIP’s projection rule and evaluated an enhanced configuration with classifier alignment and patch-level feature alignment. On the original CUB protocol, three-seed mean final accuracy rose from 78.95% to 90.26% (+11.31 points). LoDA reports a comparable gain on ten-task CUB, where classifier alignment alone raised final accuracy from 81.74% to 90.67% (+8.93 points, Table 1). Enhanced TIP exceeded our full E² reproduction in both final and average accuracy in all nine paired runs across CIFAR-100, CUB, and ImageNet-R. It also exceeded the cited TUNA, MOS, and GOD results in our exploratory ImageNet-A evaluation.

Reviewer kAsx reaffirmed the rating of 1 before we posted the matched CUB comparisons and parameter counts. We believe the evidence resolves questions. The fixed-rank experiments show that TIP’s learned projection directions reduce forgetting. The matched comparisons show that the submitted method already achieves competitive accuracy at a fraction of E²’s trainable budget.

Sincerely,
The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.