acceptodds
Under review as a conference paper at ICLR 2027

Towards Efficient Continual Learning: Task Discrimination via Incremental LDA for VLMs

Abstract

Continual learning (CL), which enables vision-language models (VLMs) to incrementally acquire new knowledge from non-stationary data streams while avoiding catastrophic forgetting (CF), is commonly addressed by parameter-efficient fine-tuning (PEFT) with task-specific low-rank adapters. However, in the more realistic task-agnostic setting, such methods either rely on explicit task identities or resort to auxiliary trainable task-discrimination modules, that are computationally expensive and unreliable in performance. In this work, we identify a previously overlooked phenomenon, supported by a theoretical analysis: owing to large-scale pre-training, the intermediate-layer representations of VLMs naturally exhibit strong between-task separability. Building on this insight, we propose ILDA-CL, a lightweight framework that decouples CL into two independent pathways: (i) a training-free distribution discriminator based on Incremental Linear Discriminant Analysis (ILDA) that incrementally refines task prototypes into a discriminative subspace, thereby enabling accurate task-identity prediction; and (ii) a set of parameter-isolated task-specific adapters that, guided by the predicted task identity, acquire new knowledge without interfering with previously learned tasks. Extensive experiments demonstrate that ILDA-CL achieves state-of-the-art accuracy while incurring substantially lower computational cost compared with existing approaches.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Continual Learning, Vision-Language Models, Task-Agnostic Continual Learning, Incremental Linear Discriminant Analysis, Task Discrimination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_4C8wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper is a clean, well-motivated idea: route samples to frozen per-task LoRA adapters using a closed-form incremental LDA over intermediate CLIP features, with no gradient training in the router. The core observation (intermediate CLIP features separate tasks well) is real and the write-up is easy to follow. The statistical appendix is also, to my relief, mostly correct.

Critical strengths
  1. Taking router features from a frozen backbone is the right constraint. It makes the "no forgetting in the router" argument essentially trivial, and it is the honest way to build a task-agnostic router.
  2. Refusing to over-claim from the Fisher trace is good practice. The three-task counterexample in Appendix C (Eq. 34) is the right observation to make, and more papers should make it.
Critical weaknesses

The released code does not implement the LoRA described in Eq. 10; The 5-shot Transfer row is identical to the full-shot Transfer row, digit for digit;The router layer and the adapter layer are hard-wired together, and "intermediate feature" has two different definitions in the same codebase;In the 5-shot setting, checkpoint selection uses the full validation set;

Questions for authors
  1. Which of Eq. 8 and lda.py:194-200 produced Tables 1-3? If the Mahalanobis form, please correct Eq. 8.
  2. Can you regenerate the 5-shot Transfer block (M2), and confirm the beta used in each of Tables 1 and 2 (M9)?
  3. Was Table 5's ILDA-CL-679 row measured, or copied from ILDA-CL (M4)?
4 comments
AuthorsOpenReview ↗

Part 1/3: Adapter Implementation and Routing Consistency

Thank you for the careful code-level review and for recognizing the motivation for frozen-feature routing and the scope of our statistical analysis. We agree that the implementation, equations, and evaluation protocol must be consistent. We address your questions and implementation concerns below.

1. Adapter implementation and correction to Eq. (10).

You are correct that Eq. (10) does not accurately describe the implemented adapter. The implementation uses a nonlinear bottleneck adapter, whereas the original equation describes a linear LoRA weight update. We will correct both the equation and the associated terminology.

The implemented mapping is Mt(m)(h)=M0(m)(h)+Bt(m)(Dropout⁡0.1(ReLU⁡(At(m)(h)))).(10)M_t^{(m)}(h)=M_0^{(m)}(h)+B_t^{(m)}\left(\operatorname{Dropout}_{0.1}\left(\operatorname{ReLU}\left(A_t^{(m)}(h)\right)\right)\right).\qquad(10) Here, hh is the post-attention hidden representation, and M0(m)M_0^{(m)} and Mt(m)M_t^{(m)} denote the frozen and adapted residual FFN mappings, respectively. At(m)A_t^{(m)} and Bt(m)B_t^{(m)} are affine down- and up-projections, including biases, with the fixed scaling absorbed into Bt(m)B_t^{(m)}. Dropout is disabled at inference.

We will consistently describe the implemented module as a task-specific nonlinear bottleneck adapter. This correction makes the manuscript accurately describe the architecture used in the reported experiments.

2. Q1: Which routing rule produced Tables 1-3?

Tables 1-3 were generated using the Mahalanobis distance implementation in lda.py, lines 194-200. We will replace the Euclidean expression in Eq. (8) with the implemented routing rule.

For s≥2s\geq2, define vt,s(x)=Ws⊤[g(l∗)(x)−μ^t],v_{t,s}(x)=W_s^\top\left[g^{(l^\ast)}(x)-\hat{\mu}_t\right], Cs=Ws⊤S~W,sWsN≤s−s+ϵIks.C_s=\frac{W_s^\top\widetilde{S}_{W,s}W_s}{N_{\leq s}-s}+\epsilon I_{k_s}. Then dt,s2(x)=vt,s(x)⊤Cs−1vt,s(x),t^s(x)=arg⁡min⁡1≤t≤sdt,s2(x).(8)d_{t,s}^{2}(x)=v_{t,s}(x)^\top C_s^{-1}v_{t,s}(x),\qquad \hat{t}_s(x)=\arg\min_{1\leq t\leq s}d_{t,s}^{2}(x).\qquad(8) Here, S~W,s\widetilde{S}_{W,s} denotes the unnormalized weighted matrix stored as self.Sw, N≤s=∑j=1sNjN_{\leq s}=\sum_{j=1}^{s}N_j, and ϵ\epsilon corresponds to reg_param. We will also align the surrounding definitions and normalization conventions with the implementation, so that the revised equations consistently describe the router used for the reported results.

3. Router layer, adapter placement, and feature definitions.

The motivation for coupling the router layer and adapter start layer was to permit reuse of the frozen prefix. The intended shared boundary is the input to the first adapted block, equivalently the output of the preceding frozen block. Its CLS representation supports routing, while the full token sequence can be retained for the selected adapter branch.

The released interfaces use different conventions: the routing hook captures a block output, whereas the auxiliary interface returns an adapter-start block input. We have checked these conventions and will explicitly document the feature-extraction boundary and adapter start layer used in the reported experiments, with consistent block indexing and interface definitions.

The coupling is an implementation choice rather than a requirement of ILDA. We will also distinguish the intended prefix-reuse strategy from the execution path implemented in the released code.

AuthorsOpenReview ↗

Part 2/3: Verified Transfer Results, Beta, and Five-Shot Model Selection

4. Q2: Identical Transfer results and beta.

We have re-evaluated the five-shot Transfer block under the original experimental setting. The regenerated results exactly match the original per-dataset entries in Table 2, including the aggregate Transfer accuracy of 65.9%. These entries also coincide with the full-shot Transfer results in Table 1.

This agreement follows from the evaluation pipeline for unseen-task samples:

  1. Frozen zero-shot gate. At each stage, the frozen CLIP model predicts over the global candidate vocabulary. If the predicted class belongs to the seen-class set, the sample enters the seen-task branch; otherwise, it enters the OOD branch.

  2. OOD branch. The sample is classified using the frozen CLIP model with the evaluation dataset's class vocabulary. Neither the task-specific adapters nor the ILDA statistics participate in this prediction.

  3. Seen-task branch. An unseen-task sample assigned to a seen-task expert is counted as incorrect under the evaluation rule, regardless of which expert ILDA selects.

Thus, for an unseen-task sample (x,y)(x,y), its contribution to Transfer accuracy is cs(x,y)=1[Gs(x)=OOD]1[f0(x;Ceval)=y],c_s(x,y)=\mathbf{1}\left[G_s(x)=\mathrm{OOD}\right]\mathbf{1}\left[f_0(x;\mathcal{C}_{\mathrm{eval}})=y\right], where 1[⋅]\mathbf{1}[\cdot] denotes the indicator function, which equals 1 if the enclosed condition holds and 0 otherwise. Here, GsG_s is the frozen zero-shot gate at stage ss, and f0f_0 is the frozen classifier evaluated over the dataset's candidate classes.

With the test data, preprocessing, candidate vocabularies, prompt templates, and task order fixed, both factors remain unchanged between full-shot and five-shot training. Consequently, reducing the adapter training samples or changing the estimated ILDA statistics does not change this Transfer metric. The identical scores are therefore consistent with the evaluation procedure, and the repeated evaluation confirms the reported entries.

AuthorsOpenReview ↗

Part 3/3: Verified Provenance and Detailed Comparison of ILDA-CL-679

6. Q3: Was the ILDA-CL-679 row measured or copied?

The ILDA-CL-679 row was independently measured; it was not copied from the full-dimensional ILDA-CL results. We have now completed an independent re-evaluation of this configuration, retaining the first 679 coordinates of the original 768-dimensional CLIP feature.

The table below reports the per-dataset differences between ILDA-CL-679 and full-dimensional ILDA-CL for Transfer, Average, and Last.

Table R1. Accuracy differences between the 679-dimensional and 768-dimensional configurations.

We define Δ = accuracy of ILDA-CL-679 - accuracy of full-dimensional ILDA-CL. All entries are absolute accuracy differences in percentage points (pp). Positive values favor the 679-dimensional configuration, and negative values favor the 768-dimensional configuration.

DatasetΔ Transfer (pp)Δ Average (pp)Δ Last (pp)
Aircraft—0.00000.0000
Caltech1010.0000+0.01730.0000
CIFAR1000.00000.00000.0000
DTD0.0000-0.0904-0.0532
EuroSAT0.00000.00000.0000
Flowers1020.0000+0.0065+0.0163
Food1010.0000-0.0059-0.0079
MNIST0.00000.00000.0000
Oxford Pets0.0000-0.0245-0.1090
Stanford Cars0.0000-0.0087-0.0871
Avg.0.0000-0.0106-0.0241

"—" denotes an inapplicable Transfer entry for the first task in the sequence.

Transfer remains unchanged across all applicable datasets, consistent with the evaluation pipeline explained above. The mean differences are -0.0106 pp in Average accuracy and -0.0241 pp in Last accuracy. The largest absolute per-dataset difference is 0.1090 pp, observed for Last accuracy on Oxford Pets. These results show a small measured effect of reducing the input-feature dimensionality from 768 to 679 on the evaluated benchmarks.

We will update Table 5 using the verified results, provide the corresponding evaluation records and higher-precision details in the supplementary material.

We thank the reviewer for their careful examination of our manuscript and code and for their constructive feedback. We hope our rebuttal has clarified the implementation details and addressed their questions about the reported results. We would welcome any remaining questions and would be happy to provide further clarification or supporting evidence.

Reviewer_dv73OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

ILDA-CL combines an incremental LDA router over frozen CLIP features with task-specific LoRA adapters. It reports strong performance on a ten-dataset continual-learning stream and provides a population-level interpretation of task discrimination under a shared-covariance Gaussian model.

Critical strengths
  1. The router requires neither gradient-based optimization nor historical-image replay, offering a practical alternative to trainable task selectors.

  2. Full-data and few-shot results demonstrate strong aggregate performance.

Critical weaknesses
  1. The methodological and theoretical novelty is limited. The framework combines established incremental LDA with parameter-isolated adapters, while the analysis mainly applies classical Gaussian-overlap bounds and Fano’s inequality rather than establishing a guarantee specific to the incremental algorithm.

  2. A substantial gap remains between the population analysis and the implemented router. The latter uses estimated means and covariances, regularization, and a potentially truncated discriminant subspace. Their effects on routing error are not quantified, particularly in high-dimensional few-shot settings.

  3. Deterministic closed-form updates do not imply preservation of old-task predictions. Adding tasks changes the projection, covariance metric, and candidate prototypes, potentially altering previous routing decisions even when old adapters are frozen. The paper provides no corresponding stability characterization.

  4. The shared-covariance Gaussian assumption is restrictive for tasks containing multiple semantic classes. Moreover, the theory covers discrimination among seen tasks, not errors introduced by the separate seen/unseen gate. Its implications for end-to-end classification therefore remain limited.

  5. Evaluation uses markedly different datasets that favor domain recognition, leaving performance on overlapping task distributions unclear. Efficiency reporting also needs complete accounting for covariance statistics, projection updates, accumulated adapters, and the zero-shot gating pass.

Questions for authors
  1. Can the authors bound or empirically characterize the gap between population Bayes error and the estimated, regularized, projected router, including sensitivity to sample size and projection dimension?

  2. Under what conditions do incremental updates preserve old-task routing decisions? How frequently do these decisions change in practice, and how much accuracy is lost?

  3. How robust is routing when task distributions are multimodal or have unequal covariances? Can the analysis or evaluation separately account for seen/unseen gating errors?

  4. Do the gains persist on within-dataset class increments and strongly overlapping tasks, with complete storage and runtime accounting?

3 comments
AuthorsOpenReview ↗

Part 1/3: Contribution and Q1

Thank you for recognizing the practical value of gradient-free, replay-free routing. Our contribution is the design and evaluation of an incremental statistical router over frozen intermediate CLIP features for selecting task-specific adapters. We agree that LDA and the underlying classification inequalities are established tools. Our theory provides a population-level interpretation of task discrimination; the following experiments characterize the implemented router.

Q1. Estimation, subspace dimension, and regularization.

We distinguish input-feature dimension dd from retained discriminant dimension kk. On the controlled shared-covariance Gaussian benchmark with d=768d=768, requested k=9k=9, and ϵ=10−4\epsilon=10^{-4}, the known-parameter Bayes classifier has a Monte Carlo error estimate of 22.958%. Across five training draws, router errors at 100 and 500 samples per task are 73.870 ± 0.497% and 35.714 ± 0.151%, corresponding to estimated gaps of 50.912 and 12.756 percentage points. These results quantify empirical excess routing error under this synthetic model.

On real frozen CLIP features, the router-only results at k=9k=9 are:

Samples per semantic classMacro accuracy (%)Overall accuracy (%)
184.824 ± 1.88390.110 ± 2.063
599.016 ± 0.06699.586 ± 0.028
2099.251 ± 0.03499.681 ± 0.001
All99.31699.714

Macro accuracy weights tasks equally; overall accuracy weights samples equally. The real-data sampling budget is specified per semantic class, whereas the synthetic sampling budget is specified per task.

With five samples per semantic class, macro accuracies for k=1,3,5,9k=1,3,5,9 are 37.863%, 51.968%, 87.503%, and 99.016%, respectively. These results demonstrate the importance of retaining sufficient discriminant directions on the evaluated benchmark. Adapters remain fixed throughout this ablation.

Regularization on real data. We additionally evaluate sensitivity to regularization around the reference value ϵ=10−4\epsilon=10^{-4} in a separate real-data ablation:

Regularization parameter ϵ\epsilon10−610^{-6}10−510^{-5}10−410^{-4}10−310^{-3}
LDA classification accuracy (%)99.399.399.399.3

Accuracy is unchanged at the reported precision over three orders of magnitude. This supports low sensitivity to regularization within the tested range in the evaluated real-data setting. We will include this ablation in the appendix.

AuthorsOpenReview ↗

Part 2/3: Q2 and Q3

Q2. Old-task routing stability.

Deterministic updates do not automatically preserve routing decisions. Let qt,s(x)q_{t,s}(x) denote routing distance and t∗t^\ast the unique pre-update winner. For s≥2s \geq 2, let ms(x)m_s(x) be the minimum margin over competing old tasks and ηs(x)\eta_s(x) the largest absolute change in any old task's distance. A sufficient preservation condition is

\qquad q_{s+1,s+1}(x) > q_{t^\ast,s+1}(x).$$ The first condition preserves the winner among old tasks, and the second excludes replacement by the new task. This a posteriori criterion covers **60.9%-100.0%** of old samples across eligible updates, rounded to one decimal place, with zero observed flips among covered samples. Across nine updates, we compute sample-weighted metrics within each update and then average updates equally. The mean routing-flip rate is **<0.1%**, and the mean net routing-accuracy decrease is **<0.1 pp**. With gate decisions, checkpoints, and branch predictions fixed, the mean classification decrease is **<0.1 pp** under both sample and task weighting. The largest sample-weighted classification decrease is **<0.1 pp**; the largest individual-task classification decrease is approximately **1.0 pp on DTD**, both at update $6 \rightarrow 7$. These results characterize stability on the evaluated sequence without assuming invariant predictions. **Q3. Distributional mismatch.** Using the synthetic setup in Q1 with 500 samples per task, requested $k=9$, and $\epsilon=10^{-4}$: | Distribution | Bayes error estimate (%) | Router error (%) | Estimated gap (pp) | |---|---:|---:|---:| | Shared covariance | 22.958 | 35.714 | 12.756 | | Mild unequal covariance | 2.730 | 36.062 | 33.332 | | Strong unequal covariance | 0.000 | 35.964 | 35.964 | | Two-component mixture | 21.078 | 35.936 | 14.858 | | Four-component mixture | 19.016 | 35.901 | 16.885 | Bayes errors are Monte Carlo estimates; **0.000% means no observed test errors**. Router errors remain near 36%, but the larger Bayes gaps under unequal covariances show that similar absolute errors do not establish general robustness to distributional mismatch.
AuthorsOpenReview ↗

Part 3/3: Q4 and efficiency

Q4. Within-dataset and semantically related tasks.

We partition CIFAR-100 into 5, 10, and 20 tasks, and 42 vehicle-related ImageNet classes into six tasks.

Task partitionILDA-CLDDASDDAS + CLIP featuresiCaRL
CIFAR-586.356.966.265.8
CIFAR-1084.355.866.066.4
CIFAR-2085.059.774.274.0
ImageNet-Vehicle4289.866.580.679.9

Reported accuracies are percentages. ILDA-CL exceeds the strongest competing result by 20.1, 17.9, 10.8, and 9.2 pp, respectively. DDAS improves with CLIP features, but ILDA-CL retains an advantage. These results extend the empirical evidence beyond distinct source datasets to within-dataset increments and semantically related categories.

Efficiency and accumulated storage.

We additionally evaluate **DIKI [a] ** and **IAP [b] **. Both use fewer parameters and less GPU memory than the baselines in our original comparison, such as LwF, WiSE-FT, and ZSCL, providing stronger efficiency references. Efficiency comparisons with the original baselines are reported in Section 5.4, Table 4 of the main paper. The following table complements those results with these two additional methods.

MetricDIKI [a]IAP [b]ILDA-CL
Single-image latency (ms)19.3226.814.57
Batch-256 latency (ms)288.26369.03274.01
Throughput (images/s)888694934
Training time/step (ms)78.9583.7476.86
Peak GPU memory (MiB)8030100627848
Accumulated trainable parameters (M)1.62.313.8
Saved checkpoints (MiB)13.15818.452.8

Single-image speedups over DIKI/IAP are 4.23x/5.87x, while batch-256 latency reductions are 4.94%/25.75%. ILDA-CL also achieves the highest measured throughput and lowest measured peak GPU memory among these methods.

We explicitly report the storage tradeoff: 13.8M accumulated parameters exceed both baselines, and 52.8 MiB of checkpoints exceed DIKI's 13.15 MiB while remaining below IAP's 818.4 MiB. Our claim is competitive practical efficiency rather than superiority on every metric.

References

[a] Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models, ECCV202

[b] IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting, IEEE TIP2026

Reviewer_EsxtOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The topic of this paper is about continual learning(CL) of vision-language models(VLMs) and the authors mainly focus on the parameter-efficient fine-tuning (PEFT) adapter-based CL methods. They propose a ILDA-CL framework with a distribution discriminator and a set of parameter-isolated task-specific adapters. The proposed method has been evaluated on CL benmark and achieves good performance.

Critical strengths
  • The problem of VLM continual learning is meaningful.
  • The performance of ILDA-CL method outperforms that other methods by a margin.
Critical weaknesses
  • Missing the important cost analysis. For CL of VLMs, both the performance and efficiency are important. It is necessary to give a detailed cost analysis (e.g, training cost, inference cost, the extra parameters cost).
  • MIssing the comparison to PEFT methods(e.g, prompt-based CL methods[a]-[b]). This paper mainly made a comparison among different adapter-based CL methods. However, recent some prompt-based methods have achieved good performance. Make a comparison among these methods (the performance and efficiency aspects).

[a] Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models, ECCV2024

[b] IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting, IEEE TIP2026

  • The idea of enabling a task distribution discriminator has been proposed in previous methods(e.g, [a]-[b]). It is necessary to highlight the novelty of the proposed module.
Questions for authors

Q1: This paper mainly focuses on the adapter-based CL methods. Prompt-based methods has also been utilized for VLMs. How to select the appropriate PEFT method in such scenarios?

Q2: As described in the work [b], the pluging location and strategy of the extra PEFT modules can also be task-specific and further decrease the parameter cost. Such adaptvie pluging location and strategy can also be integrated to the proposed framework.

5 comments
AuthorsOpenReview ↗

Part 1/3: Computational Efficiency and Storage Costs

We sincerely thank the reviewer for recognizing the importance of VLM continual learning and the performance improvements achieved by ILDA-CL. We appreciate the constructive suggestions regarding efficiency, comparisons with recent PEFT methods, and methodological novelty. We conducted additional experiments with DIKI [a] and IAP [b] under a unified evaluation protocol, together with cross-backbone evaluations of the ILDA discriminator. Our responses are organized below.

W1. Comprehensive efficiency analysis

We agree that practical VLM continual learning should be evaluated in terms of both accuracy and efficiency. We therefore report training cost, inference latency, throughput, peak GPU memory, accumulated trainable parameters, and checkpoint storage.

Table R1. Efficiency comparison with recent PEFT methods.

MetricDIKI [a]IAP [b]ILDA-CL
Single-image latency (ms)19.3226.814.57
Batch-256 latency (ms)288.26369.03274.01
Throughput (images/s)888694934
Training time/step (ms)78.9583.7476.86
Peak GPU memory (MiB)8030100627848
Cumulative params (M)1.62.313.8
Saved checkpoints (MiB)13.15818.452.8

All cumulative parameter counts refer to the weights retained after the complete continual learning sequence. The batched latency and throughput measurements use batch size 256. The reported checkpoint sizes reflect the saved artifacts in the current implementations.

Inference efficiency. ILDA-CL achieves a single-image latency of 4.57 ms, approximately 4.2x and 5.9x faster than DIKI and IAP, respectively. Under batch-256 inference, it also achieves the lowest batch latency (274.01 ms) and highest throughput (934 images/s), corresponding to 5.2% and 34.6% higher throughput than DIKI and IAP. Thus, its advantage holds for both single-image and batched inference.

Training efficiency. ILDA-CL has the lowest training time per step (76.86 ms) and peak GPU memory (7848 MiB). Its per-step training time is 2.6% and 8.2% lower than DIKI and IAP, respectively. The ILDA router is built through incremental statistics and a generalized eigenvalue problem, without gradient-based training of an additional neural discriminator.

Parameter and storage costs. We acknowledge that ILDA-CL retains more cumulative trainable parameters (13.8M) than DIKI (1.6M) and IAP (2.3M). This is the explicit cost of task-specific parameter isolation. Importantly, however, only the new task's adapter is optimized at each stage, while previously learned adapters remain frozen; cumulative parameter count therefore does not directly measure per-step training cost. Furthermore, parameter count alone does not capture storage overhead from non-trainable statistics. IAP maintains additional class-distribution covariance information, leading to an 818.4 MiB saved checkpoint, versus 52.8 MiB for ILDA-CL's saved adapter checkpoints (a 15.5x difference). DIKI has the smallest checkpoint (13.15 MiB). We therefore emphasize ILDA-CL's accuracy-computation-storage trade-off, rather than claiming the fewest parameters.

AuthorsOpenReview ↗

Part 2/3: PEFT Comparisons and Method Selection

W2. Comparison with DIKI [a] and IAP [b]

We thank the reviewer for these suggestions. We reproduced both methods under our evaluation protocol, where task identities are unavailable at inference. All methods use the same batch size of 256 for the reported evaluations. We consider both class-incremental learning (CIL) and task-agnostic continual learning.

CIL comparison. CIL evaluation is restricted to learned tasks and classes. ILDA-CL outperforms DIKI and IAP by 1.9 and 2.8 percentage points, respectively, in both Average and Last accuracy.

Table R2. CIL accuracy (%).

MethodAverageLast
DIKI [a]86.486.3
IAP [b]85.585.4
ILDA-CL88.388.2

Task-agnostic comparison. This setting additionally evaluates unseen tasks. Transfer measures generalization to unseen tasks; Average aggregates accuracy over seen and unseen tasks throughout continual learning; Last reports final-stage performance.

Table R3. Task-agnostic accuracy (%).

MethodTransferAverageLast
DIKI [a]68.776.286.3
IAP [b]69.976.185.4
ILDA-CL (original)65.976.288.2
ILDA-CL (+ Batch Voting)68.277.588.3

Stability-plasticity trade-off. Task-specific adapters favor adaptation to acquired tasks, as reflected in ILDA-CL's higher learned-task performance. DIKI and IAP preserve stronger zero-shot transfer, as reflected in their higher Transfer accuracy. These results reveal complementary strengths rather than universal superiority of one PEFT paradigm. Importantly, ILDA-CL achieves stronger learned-task performance together with lower measured inference and training costs.

Batch-voting extension. We observed that the implementations of DIKI and IAP employ batch voting during inference. To ensure a fair comparison, we therefore also evaluate ILDA-CL with inference-time batch voting, without incorporating any other DIKI/IAP modules. It improves ILDA-CL's Transfer from 65.9% to 68.2%, Average from 76.2% to 77.5%, and Last from 88.2% to 88.3%. The extension narrows the Transfer gap while achieving the highest Average and Last accuracy. It adds no trainable parameters.

Q1. How should one choose a PEFT method?

The choice depends on the desired balance between learned-task plasticity, zero-shot knowledge preservation, runtime efficiency, and storage budget. ILDA-CL is attractive when learned-task accuracy and low inference latency matter most. DIKI and IAP may be preferable when transfer to unseen tasks or minimizing trainable parameters is the priority. Our comparison also shows that fewer trainable parameters do not necessarily imply lower latency or smaller checkpoints, particularly when non-trainable distribution statistics are retained. We therefore advocate comparing the complete accuracy-computation-storage trade-off.

AuthorsOpenReview ↗

Part 3/3: Novelty and Adaptive Module Placement

W3. Novelty of the ILDA discriminator

We acknowledge that distribution-aware modeling has been explored in DIKI and IAP. Our contribution is not distribution awareness itself or the invention of LDA, but an incremental LDA-based task-routing mechanism that exploits separability in frozen visual representations. We clarify the differences in four respects.

(1) Explicit discriminative routing. DIKI employs distribution-aware calibration for residual knowledge integration, whereas IAP uses instance-aware and distribution-guided prompting. In contrast, ILDA-CL builds a shared Fisher-discriminative subspace from frozen intermediate visual features and uses covariance-aware prototype matching to select task-specific adapters. The resulting module directly addresses task identification for parameter-isolated CL.

(2) Incremental, optimization-free updates. ILDA accumulates task-level first- and second-order statistics and updates the shared projection through a generalized eigenvalue problem. The routing discriminator does not require gradient-based training or replay of previous training images. Its frozen visual feature space remains stable as task-specific adapters are sequentially trained.

(3) Generalization across vision backbones. ILDA task discrimination uses intermediate visual features rather than relying on a CLIP-specific text-image matching interface. To test whether this property is restricted to CLIP, we evaluate the discriminator with different pretrained visual backbones.

Table R4. Cross-backbone task-discrimination accuracy (%).

Vision backboneMacroOverall
SigLIP[c]99.699.8
MetaCLIP[d]99.399.6
TIPS[e]99.399.8
Qwen3.5-VL (ViT)[f]99.699.8

The verified results show over 99% Macro and Overall task-discrimination accuracy on SigLIP[c], MetaCLIP[d], TIPS[e] and Qwen3.5-VL[f] vision-encoder. This supports the generality of the feature geometry exploited by ILDA across pretrained vision backbones. Importantly, these experiments validate the ILDA discriminator, not full continual-learning classification accuracy on all backbones.

(4) Statistical interpretation. Under a shared-covariance Gaussian model, we relate pairwise Mahalanobis separation to an upper bound on Bayes task-classification error and a corresponding lower bound on mutual information. This motivates covariance-aware routing, while its finite-sample performance is validated empirically. Thus, the contribution combines incremental discriminative subspace construction, optimization-free task routing, and cross-backbone applicability.

Q2. Could adaptive PEFT module placement be integrated?

We agree. ILDA-CL decouples task discrimination in frozen intermediate visual representations from task adaptation via downstream task-specific adapters. This modularity leaves room to incorporate IAP-style adaptive placement in the later transformer blocks without changing the fixed features used by the ILDA router. Such a design may reduce cumulative adapter parameters, although placement selection could introduce its own optimization and inference costs. We consider this a valuable direction for future work rather than claiming it has been experimentally established here.

We thank the reviewer again for these constructive comments. We will include the expanded PEFT comparisons, detailed efficiency measurements, cross-backbone discrimination results, and a clearer discussion of methodological novelty and trade-offs in the revised manuscript.

Reference

[a] Mind the Interference: Retaining Pre-trained Knowledge in Parameter Efficient Continual Learning of Vision-Language Models, ECCV202

[b] IAP: Improving Continual Learning of Vision-Language Models via Instance-Aware Prompting, IEEE TIP2026

[c] Sigmoid Loss for Language Image Pre-Training, ICCV2023

[d] Demystifying CLIP Data, ICLR2024

[e] TIPS: Text-Image Pretraining with Spatial Awareness, ICLR2025

[f] Qwen3.5: Towards Native Multimodal Agents, Qwen Blog 2026

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Respones

Dear AC/SAC/PC,

We sincerely thank the ICLR reviewers, ACs, and PCs for their time and constructive feedback. Below, we would like to provide a final summary of our contributions, discussion, and revisions.

Contributions and Strengths.

ILDA-CL selects a task-specific adapter without a task label at inference. Its router applies incremental linear discriminant analysis to frozen intermediate CLIP features, while downstream adapters learn each task. Storing the means in their original feature space lets us reproject earlier task prototypes as new tasks arrive.

Routing design. Reviewer dv73 recognized the value of a selector that requires neither gradient training nor replay of past images. Reviewer 4C8w highlighted the fixed feature space used for routing.

Benchmark performance. Reviewer Esxt noted the gains over baselines, and dv73 highlighted the full-data and few-shot results. Across the ten-dataset stream, final class-incremental accuracy was 88.2% with full data and 78.8% with five images per class (Table 3).

Statistical interpretation. Reviewer 4C8w highlighted the Appendix C counterexample showing that a large Fisher trace can coexist with two indistinguishable tasks.

Summary of the Discussion.

Accuracy and efficiency. At Reviewer Esxt’s suggestion, we evaluated ILDA-CL alongside DIKI and IAP under a common protocol. ILDA-CL exceeded DIKI by 1.9 percentage points and IAP by 2.8 points in final class-incremental accuracy. Single-image inference took 4.57 ms for ILDA-CL, compared with 19.32 ms for DIKI and 26.81 ms for IAP. Reviewer Esxt subsequently raised their rating by one point, noting that our response had addressed their concerns.

Overlapping tasks and routing stability. To address Reviewer dv73’s question about tasks drawn from the same dataset, we split CIFAR-100 into 5, 10, and 20 tasks and divided 42 ImageNet vehicle classes into six tasks. Across these four settings, ILDA-CL’s margin over the strongest baseline on each split ranged from 9.2 to 20.1 percentage points. On the original ten-dataset stream, more than 99.9% of examples from earlier tasks retained their routing assignment after an update, averaged across all nine updates.

Scope of the statistical analysis. We complemented the population analysis with controlled experiments varying sample size and covariance structure. Under the shared-covariance Gaussian model, increasing the training set from 100 to 500 samples per task reduced the router’s estimated excess error over the Bayes classifier by approximately 75%. On real data, the router achieved over 99% macro task-identification accuracy on each of four additional pretrained vision encoders.

Implementation and reproducibility. In response to Reviewer 4C8w, we detailed the projected Mahalanobis routing rule and task-specific nonlinear bottleneck adapters used in Tables 1–3. We also independently reproduced the five-shot Transfer score of 65.9%. Its agreement with the full-data result follows from the evaluation protocol. An unseen-task example earns credit when the frozen gate routes it to the zero-shot branch and frozen CLIP predicts the correct class from the evaluation dataset’s vocabulary. In a separate rerun, the 679- and 768-dimensional configurations differed by approximately 0.024 percentage points in mean final accuracy.

We will incorporate the additional comparisons, routing analyses, and implementation details into the revised manuscript. Thank you for considering our submission.

Best regards,

Submission 56608 authors.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.