acceptodds
Under review as a conference paper at ICLR 2027

CL-CLIP: CLIP-Based Continual Learning Framework with Cost-Volume Category Decoupling for Object Detection

Abstract

Continual Object Detection (COD) requires a detector to acquire new categories over time while preserving previously learned ones. This goal is closely related to open-vocabulary detection, since both settings require reasoning over categories that are not fully covered by the annotations available at the current training stage. Recent CLIP-based open-vocabulary detectors have shown strong zero-shot generalization, and frameworks such as F-ViT demonstrate that vision-language pretraining can provide powerful zero-shot detection ability for unseen categories. However, real-world deployments cannot remain purely zero-shot: once these detectors are continually updated on newly introduced categories, they suffer severe catastrophic forgetting and quickly lose their previously calibrated detection ability. We therefore propose CL-CLIP, a CLIP-based COD framework that equips open-vocabulary detectors with better continual learning ability through cost-volume-guided category decoupling. Specifically, following CAT-Seg, we compute a CLIP image-text similarity cost volume, defined as dense category-wise response maps between visual tokens and class text embeddings. This zero-shot spatial prior decomposes shared region features into class-specific pathways, which are then processed by a Multi-Expert RoI head. Extensive experiments on PASCAL VOC and MS-COCO show that CL-CLIP substantially improves the F-ViT baseline under continual fine-tuning and achieves competitive performance with existing continual object detectors, especially in adapting to newly introduced categories while preserving competitive base-class performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Continual Object Detection, Open-Vocabulary Detection, CLIP, Cost Volume, Multi-Expert Head

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_CV3xOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes CL-CLIP, a continual object detection framework that leverages CLIP-based spatial similarity priors to reduce catastrophic forgetting. It introduces cost-volume-guided category decoupling, class-specific RoI experts, and drift regularization to separate category representations during sequential learning. Experiments on Pascal VOC and MS-COCO demonstrate improvements over CLIP-based baselines and competitive performance against existing continual detection methods.

Critical strengths
  1. The paper addresses a well-motivated problem by leveraging CLIP's pretrained visual-language representations for continual object detection.

  2. The category-specific expert design provides a practical approach to reducing interference between previously learned and newly introduced classes.

  3. Extensive experiments on VOC and COCO demonstrate strong performance, supported by matched-backbone comparisons and meaningful component ablations.

Critical weaknesses
  1. Although CLIP is frozen, the cost-volume refinement modules remain trainable, potentially changing old-class spatial responses. The authors should quantify cross-task response drift and compare frozen versus updated refinement modules.

  2. Table 3 removes both components jointly, making their individual effects unclear. Separate ablations of cost-volume gating and per-category experts under comparable model capacity are needed.

  3. Table 5 measures cross-category response overlap, but low overlap does not necessarily imply correct category localization. The authors should evaluate spatial alignment with ground-truth regions across continual learning stages.

  4. A shared cosine logit scale does not necessarily guarantee calibration between old- and new-class experts. The authors should analyze cross-task score distributions and evaluate the impact of simple score calibration on detection performance.

Questions for authors

I would appreciate further quantitative evidence addressing the key concerns raised above, particularly regarding cross-task spatial response drift caused by trainable cost-volume refinement modules, the independent contributions of cost-volume gating and category-specific experts under comparable model capacity, the spatial alignment between category responses and ground-truth regions across continual learning stages, and the calibration of old- and new-class expert scores. These issues are important for validating the effectiveness of the proposed components and their ability to mitigate catastrophic forgetting. If the authors cannot provide convincing quantitative experiments to adequately address these concerns, I may consider lowering my score.

Reviewer_uXBNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3(+2)
Read review
Paper summary

This paper studies class-incremental object detection using frozen CLIP representations. It proposes CL-CLIP, which constructs category-wise spatial responses from CLIP image–text similarities, uses these responses to gate RoI features, and assigns a separate convolutional classification expert to each category. Experts for previously learned classes are frozen, while EWC constrains updates to shared FPN components; a cost-volume-guided Spatial-RPN is also introduced. The method is evaluated on PASCAL VOC and multi-task MS-COCO protocols, with ablations of the expert pathway, regularization, and auxiliary components. It improves all-class AP over its matched F-ViT/FineCLIP baseline, although the underlying anti-forgetting mechanisms and several implementation details remain insufficiently established.

Critical strengths
  1. The paper is generally well organized, and the motivation for combining frozen CLIP representations with class-specific detection pathways is easy to follow.
  2. The category-specific expert design is intuitive and addresses a relevant challenge in class-incremental detection.
  3. The evaluation includes established PASCAL VOC and MS-COCO incremental benchmarks, relevant ablations, and efficiency measurements; the reported gains over the matched frozen-CLIP baseline are encouraging.
Critical weaknesses

Method Section

Comment 1 (Figure 1 and Eqs. (1), (4), (5): inconsistent notation and incomplete definitions). Figure 1 writes the total loss as "L = L_ortho + L_drift + L_reg + L_cls," but Eq. (5) includes a weight λ on L_drift (λ = 1000 in Table 8). The two formulas should match. Also, Eq. (1), "Z_cv = Cost(F(I))," does not show the text embeddings used to compute category similarities, while Eq. (4) does not explain how each expert output is pooled or projected to match the CLIP text embedding. Please make these key definitions more explicit. In addition, Figure 1 uses "Rol" instead of "RoI".

Comment 2 (Sec. 3.1 vs. Sec. 3.4: identification and treatment of old-object proposals). The paper says "proposals matching previous categories are excluded from the classification loss rather than treated as negatives" (Sec. 3.4). However, Sec. 3.1 says that only current classes have annotations. Without old-class labels or pseudo-labels, how can these old-object proposals be identified? Please clarify whether the method merely skips old-class classification terms or also ignores particular proposals. Please also clarify whether unlabeled old-class objects are still treated as background during RPN training, and how the proposed spatial attention guidance addresses this issue.

Comment 3 (Sec. 3.2: what exactly is claimed to be stable). The authors claim that the cost volume provides a "stable per-category spatial signal that does not degrade as new tasks arrive" (Sec. 3.2). Yet they also state that "all other modules (cost-volume pathway, FPN, RPN, current-task experts) are trainable" (Sec. 3). Freezing CLIP preserves the raw image–text similarities, but it does not ensure that the refined response maps produced by trainable aggregation and decoding remain unchanged. The orthogonality loss reduces overlap between classes, not changes within the same class across tasks. Please clarify which spatial signal is claimed to be stable and show how old-class responses change after learning new tasks.

Comment 4 (Sec. 3.4: EWC implementation details are missing). Regarding EWC, the paper says, "The diagonal Fisher information is computed after each task from the total loss gradients" (Sec. 3.4). However, it does not explain exactly how Fisher importance is estimated and normalized, which earlier parameters are used as references, or how information is carried across tasks. Table 8 provides the weight λ = 1000, but not these update rules. As Table 11 shows large benefits from drift regularization, the complete loss equation and step-by-step update procedure are important for reproducing the method.

Experiments Section

Comment 5 (Table 3: cost-volume gating and per-category experts are conflated). Table 3 explicitly states that "Cost-volume gating and per-category experts are ablated jointly." Therefore, the experiment shows that the combined design helps, but not whether the gains mainly come from the cost-volume gate or from having separate experts whose old weights are frozen. The additional comparison of convolutional and MLP experts does not answer this question. Please report the two components separately, ideally with similar model capacity and the same EWC setting.

Comment 6 (Table 6: all-class AP does not match the class-wise averaging rule). In Table 6 (COCO 70+10, FG-CLIP ViT-L/14), the reported entries are "40.6" for current-class AP, "21.2" for previous-class AP, and "21.7" for all-class AP. Under the standard class-wise averaging rule, the all-class value should be (10 × 40.6 + 70 × 21.2) / 80 = 23.625, approximately 23.6, rather than 21.7. Please check the numbers or explain whether a different evaluation rule was used.

Comment 7 (Sec. 4.4, Tables 7 and 14: better initialization is conflated with better retention). The paper states that CL-CLIP "improves old-class and all-class AP over the corresponding baseline" (Sec. 4.4). However, its initial detector is already stronger. On VOC 10+10, the FineCLIP baseline has 80.2 AP after Task 1 (Table 7), while CL-CLIP has 86.2 (Table 14, seed 0). After Task 2, old-class AP is 55.8 versus 64.8. Thus, the old-class AP drops are 24.4 versus 21.4 points: a 3.0-point difference in forgetting-related decline, rather than the full 9.0-point final gap. On COCO Task 4, old-class AP improves only from 17.4 to 18.6. Please report forgetting measures that separate better initial accuracy from better retention.

Appendix Section

Comment 8 (Appendix F: gradient path of frozen old-class experts). Appendix F says that "each frozen old-class expert still contributes forward activations (and gradients through the shared features it feeds)." However, Sec. 3.4 defines the classification loss only for current-class scores, and no loss using the old experts' outputs is clearly described. A frozen expert can pass gradients to its inputs if its output is used by a loss, but simply running its forward pass does not create such gradients. Which loss, if any, directly uses old-expert outputs? Please clarify the actual gradient path.

Comment 9 (Appendix F and Table 13: cost scaling with present vs. learned categories). Appendix F says that "the cost per image scales with the categories actually present rather than the full vocabulary." The same section also says that the expert pipeline operates over "all seen categories." If every learned expert runs on every image, computation should depend on the number of learned categories, not on how many object classes appear in that image. No mechanism for skipping irrelevant experts is described, and Table 13 shows FLOPs increasing from 289.48G to 594.69G as the vocabulary grows from 19 to 80 classes. Please explain which interpretation is correct.

Minor Presentation and Formatting Issues

Comment 10. Figure 4 labels the first visualization "Original CLIP," while the text and caption refer to "Original FineCLIP". The bibliography uses inconsistent venue naming conventions (e.g., abbreviated versus full ECCV titles) and DOI formats (plain identifiers versus full URLs), and some entries lack complete publication details. The Conclusion contains "confirm substantially improvements," which should be "confirm substantial improvements". "Human participations" in the Ethics Statement should be "human participants".

Questions for authors
  1. With no labels for old classes, how are the old-object proposals mentioned in Sec. 3.4 identified or ignored? Please explain the RoI and RPN training labels and the complete detection loss.
  2. Does "stable per-category spatial signal" mean the raw frozen-CLIP similarities or the maps produced by the trainable decoder? How much do old-class response maps change between tasks?
  3. What is the exact EWC loss, how is diagonal Fisher importance calculated, and how are old-task weights and importance estimates maintained across tasks?
  4. Can you compare cost-volume gating alone, per-category experts alone, and the full design under comparable settings?
  5. Can you report old-class forgetting relative to each model's own Task-1 AP, including results across the COCO task sequence?
  6. In Appendix F, what loss sends gradients through the frozen old-expert branches, if any?
  7. Are all learned experts run on every image, or does the system skip experts for classes not present? How does this relate to Table 13?
  8. Can you verify the FG-CLIP ViT-L/14 70+10 row in Table 6, where the reported all-class AP is 21.7 rather than the usual weighted value of about 23.6?
Reviewer_vV2nOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes CL-CLIP, a continual object detection framework that constructs category-specific spatial responses from a CLIP cost volume. It combines category-specific experts, freezing of previously learned experts, and EWC regularization to mitigate catastrophic forgetting. Experiments on PASCAL VOC and MS-COCO show clear improvements over the sequentially fine-tuned F-ViT baseline.

Critical strengths

1.The paper clearly demonstrates that open-vocabulary detection ability does not naturally translate into continual learning ability. 2.Cost-volume-based category decoupling is well motivated for reducing cross-category interference in continual object detection. 3.The method consistently outperforms the matched F-ViT baseline across multiple VOC splits and the four-task COCO setting. 4.The paper provides extensive ablations, evaluations with different CLIP backbones, and efficiency analysis.

Critical weaknesses

1.The paper describes the cost volume as a stable routing signal, but its spatial aggregation and decoder modules are still updated in subsequent tasks, making this explanation insufficiently justified. 2. Cost-volume gating and per-category experts are ablated jointly, so it is unclear whether the gains come from the cost volume or mainly from the independent frozen experts. 3. The implementation of excluding old-class proposals from the classification loss is unclear because old-class annotations are generally unavailable in the current task. 4. The method maintains and executes a separate expert for every category, leading to computation that grows linearly with the number of seen classes and making it difficult to scale to large-vocabulary settings.

Questions for authors
  1. If the cost-volume aggregation and decoder are continually updated, how are old-class spatial responses kept stable?
  2. Could the authors separately ablate cost-volume gating and per-category experts?
  3. How are old-class proposals identified and excluded when old-class annotations are unavailable?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.