acceptodds
Withdrawn from ICLR 2027

Distributive Similarity Composition for CLIP-based Class-Incremental Learning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning (PEFT) provides an effective way to adapt vision-language models to sequential tasks while preserving their pre-trained backbones. However, when separate task-specific modules are maintained, task-agnostic inference in class-incremental learning typically relies on routing or retrieval to activate the appropriate module, making final predictions vulnerable to selection errors. In this paper, we exploit a distinctive property of CLIP to replace selection-based inference with task-representation composition. Since CLIP performs classification through inner-product similarities, and the inner product distributes over vector addition, the similarity of a composed visual representation can be decomposed into the additive contributions of individual task-wise representations. Based on this observation, we propose Distributive Similarity Composition for Incremental Learning (DiSCIL), a three-stage rehearsal-free framework. Specifically, DiSCIL freezes the CLIP textual space and trains a visual LoRA for each incoming task. Motivated by the observation that visual features are more crowded than their corresponding text anchors, each LoRA is optimized to produce high responses to relevant textual anchors while suppressing unrelated responses. To improve inference efficiency and stability, we further introduce an incremental two-teacher consolidation strategy that distills the previous merged LoRA and the current task LoRA into a new merged LoRA using mean squared error and cosine objectives. Consequently, the final merged LoRA requires only one visual forward pass, and its output is classified using all seen-class text anchors together with an auxiliary cumulative cosine classifier. Extensive experiments on multiple benchmarks demonstrate that DiSCIL achieves strong continual recognition performance with favorable parameter and inference efficiency.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
Class-Incremental Learning, CLIP, Parameter-Efficient Fine-Tuning, LoRA Consolidation.

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_p6deOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes DiSCIL, which replaces task-specific module routing in CLIP-based class-incremental learning with additive representation composition based on the distributive property of CLIP’s inner-product classifier. It further introduces a two-teacher LoRA consolidation strategy to merge historical and current LoRAs into a single module for task-agnostic inference.

Critical strengths

The observation that CLIP’s visual features are more crowded than textual anchors is intuitive and provides a reasonable rationale for freezing the text space and adapting only the visual encoder.

Critical weaknesses

1)The distributive property in Eq. (3) holds for any linear inner product and does not by itself justify treating task-specific modules as “complementary contributions” rather than competing candidates. Eq. (4) is only an ideal selectivity condition, not a proven property. 2)The proposed framework largely combines existing ideas: textual anchors, LoRA, supervised contrastive learning, negative similarity suppression, prototype-initialized cosine classifiers, and two-teacher distillation. 3)Table 1 shows DiSCIL is not consistently superior, especially on Aircraft and Cars, where TPPT-V achieves better Last accuracy. Yet the paper claims “strong and consistent” performance. Only Avg and Last are reported.

Questions for authors

As the weakness.

Reviewer_JW26OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The authors propose a method to reduce the number of extra LoRAs designed and trained for each task. They hope to use a single LoRA that can represent the semantic features of different tasks for class-incremental learning. The whole training process for CIL still needs to first train different LoRAs for different tasks and then transfer the ability of different LoRAs into a combined single LoRA for final inference with the help of distillation.

Critical strengths

The motivation is interesting. If there is only a single LORA that can sperate different tasks for CIL, the computation cost will definitely be reduced, and the inference speed will be increased.

Critical weaknesses

The reason why the task-wise representation can be decoupled as the weighted-average of different task representations. Besides, the training process is quite complex. It still needs to train different LoRAs for different tasks, respectively first. Then, the different LoRAs are combined for a single LoRA to represent task-wise representation.

Questions for authors
  1. The writing needs to be improved. First, there is no definition of K, k in Eq. (2), making it difficult to understand the meaning of the equation. Same problem also appears at Eq. (10), what is meaning of B. Moreover, the formulation of the valid anchor set seems awkward. Based on the formulation, P(i) is a set, representing the other samples share the same category as i-th sample in the same task. If you want to describe that P(i) should be empty. Maybe you can try to use the sign for empty set.
  2. As I mentioned in critical weakness, could you please explain more about Eq.(2). It is difficult for me to understand why the task-wise representation can be described as the weighted average of different task representations. Moreover, instead of using complex weights for different tasks, you directly use (k-1):1 in Eq. (21). Why do you choose to use this ratio. Please provide more explanation behind the two equations.
  3. Following question 2, Why the inference of new task will not be influenced by the old task through the consolidated single-LoRA.
  4. In Appendix D, the authors attempt to justify why the task-wise representation can be decomposed into a weighted average of different task representations. However, Eq. (32) appears to be based on an assumption rather than a theoretical derivation, as the weighting ratio of (k-1):1 is introduced without sufficient justification. Given this assumption, the subsequent simplification naturally leads to a representation consisting of the previous task representation and a residual term involving both previous and current task representations. Therefore, I do not find this explanation sufficiently convincing, as the conclusion seems to follow directly from the assumed formulation rather than providing a theoretical justification for it. The authors should first clarify why the teacher target can be formulated as Eq. (32), particularly the rationale behind the weighting ratio of (k-1):1, before using it to justify the proposed decomposition.
  5. Please provide the ablation study on the ratio (k-1):1. Ablation study of different combination of the distillation loss of Eq. (24). Ablation study of auxiliary cumulative cosine classifier.
  6. The whole training process still requires to train multiple LoRAs. Would it be possible to merge the previously trained LoRA adapters while simultaneously training a new LoRA adapter for the current task?
  7. Based on the results presented in Table 1, the proposed method does not appear to achieve competitive performance on most datasets. Furthermore, Tables 2 and 3 show only marginal performance differences across various experimental settings. Given these observations, it remains unclear what the key advantages of the proposed method are and whether it offers meaningful improvements over existing approaches. The authors are encouraged to clarify the main benefits and contributions of their method. If the advantage of the convenience and fast, please provide the training time and inference speed. Anyway, based on the description in the main paper, the training process is very complex.
Reviewer_Q85SOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses module-selection errors in task-agnostic inference for PEFT-based CIL methods that maintain separate task-specific modules. The proposed method uses frozen CLIP text anchors, built from class-name prompts, as a fixed classifier. It exploits the linearity of the inner product to combine task-wise visual LoRA representations by composition instead of selection. To reduce inference cost, the previous merged LoRA and the current task LoRA are recursively distilled into a single merged LoRA. As a result, module selection is removed at inference.

Critical strengths

S1. Simple and interpretable formulation. Starting from the fact that CLIP classifies by inner-product similarity, the paper shows that the contributions of task-wise representations decompose additively (Eq. 3). It uses this as a design principle and derives a structure that requires no task identity, prompt retrieval, expert routing, or replay buffer at inference.

S2. Inference cost independent of the number of tasks. All task knowledge is consolidated into a single merged LoRA. The number of trainable parameters and visual forward passes (one) remains constant regardless of the number of tasks. Consolidation uses only the previous merged LoRA and the current LoRA, without past data or all previous LoRAs, which is consistent with the rehearsal-free setting.

S3. Empirical support for consolidation and module generality. In Table 3, the single merged LoRA performs close to explicit composition (EMC), which requires one forward pass per task. On CIFAR-100, the proposed method obtains 88.34/80.86 (Avg/Last), compared with 88.52/80.57 for EMC. This partially supports the claim that recursive distillation approximates the composition target. Table 2 shows similar performance when LoRA is replaced with Prompts or Adapters, indicating that the method is not tied to a specific PEFT architecture.

Critical weaknesses

W1. The proposed method circumvents rather than resolves the module-selection problem. The paper attributes the removal of module selection to representation composition (Eq. 3). However, the final method does not perform composition at inference. Stage II distills the previous merged LoRA and the current task LoRA into a single LoRA, and the authors acknowledge that this is not an exact realization of Eq. 3 (Sec. 3.3). The deployed model therefore contains only one module, and no selection target exists by construction. The selection problem is bypassed by reducing to a single-module architecture, not resolved. This places the proposed method in the same category as merging- or fusion-based methods, which do not face selection errors in the first place. Under this view, comparison with retrieval-based methods (L2P, DualPrompt, CODA-P) does not support the central claim. The relevant baselines are methods that also fuse task-specific parameters, such as RAPF and BOFA. Both are discussed in the Appendix but are absent from Table 1.

W2. Insufficient analysis in the direct comparison. Sec. 4.2 only states that the proposed method is “strong and consistent” and “remains competitive,” without method-wise analysis. The paper does not explain why it underperforms TPPT-V on the fine-grained benchmarks (Aircraft, Cars). It also does not identify which component accounts for the gains over MG-CLIP on CIFAR-100 and ImageNet-R. In addition, it is unclear whether all baselines use the same OpenCLIP ViT-B/16 (LAION-400M) backbone, which is specified only in the Appendix.

W3. Marginal gains without variance reporting. In Table 1, the proposed method achieves the best result in only 5 of 10 metrics. On CUB (Last), Aircraft, and Cars, it trails TPPT-V or CODA-P by up to 1.43%p (Aircraft, Avg). Where it ranks first, the margin over the strongest baseline is at most 0.73%p (CIFAR-100, Last). No standard deviations or numbers of runs are reported, so the significance of these differences cannot be assessed. The claim of “strong and consistent performance across all five benchmarks” is therefore not supported. Mean and standard deviation over multiple seeds should be reported.

W4. The feature-space asymmetry is not connected to the CL problem. The paper presents two motivations: module-selection errors, and the observation that visual features are more crowded than text anchors (Fig. 1(b)). No logical connection between the two is provided. Moreover, the asymmetry is a static property measured on frozen CLIP and does not by itself indicate a CIL-specific problem. To establish it as a key CL problem, the paper should show the following:

4.1 whether the asymmetry intensifies as tasks progress

4.2 whether it correlates with forgetting or degraded CIL performance in existing CLIP-based, prompt-based, or MoE-based CL methods

Questions for authors

Q1. The paper presents the distributive property (Eq. 3) as a distinctive property of CLIP. However, it is the linearity of any fixed linear classifier. In the proposed method, the CLIP text encoder only provides fixed per-class weights. It is therefore necessary to separate whether the performance comes from (i) the fixed linear head structure or (ii) CLIP’s pre-trained image–text alignment. Please report results when the text anchors are replaced with a fixed random head or an ETF head under the same pipeline.

Q2. What does Figure 3 show, and on which dataset is it computed?

Color scale: The color scale is restricted to [0.23,0.34][0.23, 0.34], which visually amplifies small differences.

Significance: For image task 2, the diagonal entry is 0.315 and the largest off-diagonal entry is 0.286 (LoRA 6), a gap of 0.029. With τtext=0.07\tau_{text}=0.07, this corresponds to a logit difference of about 0.41. Overall, unmatched LoRAs produce about 80% of the matched response. Is this difference large enough to support task-selective responses? Please also report the response of the frozen CLIP encoder without LoRA as a reference.

Scope: The matrix reports only the similarity to the GT anchor. Selectivity in the sense of Eq. 4 also requires low responses to non-GT anchors, which Figure 3 does not show.

Q3. The Abstract and Conclusion claim “favorable parameter and inference efficiency,” but the only evidence is the number of forward passes in Table 3. Parameter counts, FLOPs, latency, memory usage, and complexity analysis are not provided, so space and computational efficiency cannot be verified.

Training cost: Each task requires Stage I (5 epochs), cosine classifier training (5 epochs), and Stage II (5 epochs). Stage II performs three forward passes per image, for two teachers and one student (Table 3). The training cost per task is therefore several times that of training a single LoRA. Please report the added training cost in GPU hours.

Capacity: All tasks are compressed into a single fixed rank-4 LoRA, so space efficiency is obtained at the cost of an upper bound on representational capacity. Please examine how this constraint affects Last accuracy as the number of tasks increases, together with the long-sequence experiment in Q4.

Q4. Recent CIL studies evaluate not only 10-task settings but also 20 or more tasks. What are the results of the proposed method in a 20-task setting?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.