acceptodds
Withdrawn from ICLR 2027

Knowledge Recovery for Novel Classes in Open-Vocabulary Object Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Pretrained open-vocabulary object detection (OVD) models can recognize novel classes without ground truth annotations in a target dataset. However, adapting these models with only base class supervision tends to compromise their generalization to novel classes. We find that treating unlabeled novel-class instances as background during adaptation suppresses two types of knowledge: class-agnostic foreground knowledge and class-discriminative semantic knowledge, leading to reduced detection performance for novel classes. To address this issue, we propose KR-OVD (Knowledge Recovery for Open-Vocabulary Detection), which provides two distinct recovery pathways. One pathway uses class-aware pseudo labels to directly supply the missing classification and localization supervision for novel classes. The other pathway introduces an Inter-Class Attention Path Aggregation Network (ICA-PAN) that models inter-class relationships to enable base-to-novel semantic transfer, indirectly recovering impaired novel-class knowledge. Experiments on OV-COCO and VisDrone2019-DET demonstrate that KR-OVD recovers novel-class recognition impaired during adaptation while preserving base-class performance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary Object Detection, Model Adaptation, Pseudo-Labeling, Vision-Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Y8PWOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies the degradation of novel-category detection when a pretrained YOLO-World-L v2.1 model is adapted using ground-truth annotations only for base classes. On OV-COCO, base-only fine-tuning reduces novel AP50 from 68.0 to 25.7. The authors interpret this degradation as suppression of foreground and category-discriminative knowledge and propose KR-OVD with two recovery pathways: novel-category pseudo-labels generated by a frozen YOLO-World teacher, and ICA-PAN, which combines inter-class text attention with text-guided visual feature fusion. Experiments on OV-COCO and VisDrone2019-DET report base/novel AP50 of 66.5/50.7 and 40.2/17.9, respectively. Ablations examine the contributions of pseudo-supervision and the architectural components.

Critical strengths

The paper reports both base- and novel-category performance before and after adaptation, making the loss of pretrained capabilities explicit. The pseudo-label-only and ICA-PAN-only ablations also help distinguish the empirical contributions of the two components.

Critical weaknesses
  1. Limited evidence for the recovery mechanisms.
    • The paper shows partial recovery over base-only adaptation, but does not compare against preservation strategies applied to the same pretrained checkpoint, such as feature- or logit-level distillation from the frozen model, partial freezing, or weight regularization/interpolation. Comparisons with heterogeneous OVD models do not establish a better adaptation–retention trade-off.
    • On OV-COCO, pseudo-labeling alone achieves 67.5/50.5 base/novel AP50, while the full method achieves 66.5/50.7. Thus, standard pseudo-labeling accounts for nearly all the novel-category improvement. ICA-PAN helps when combined with pseudo-labeling on VisDrone, but alone reduces novel AP50 from 13.6 to 4.7.
    • These results support a dataset-dependent architectural benefit, but do not establish base-to-novel semantic transfer as its cause. Controls isolating cross-category interaction and variability estimates for small gains, such as ICTE’s 0.7 novel AP50 improvement, are needed.
  2. Unequal supervision assumptions and limited validation.
    • KR-OVD uses the exact evaluation novel-class list to generate category-specific training labels, so it receives direct supervision for precisely the classes on which it is evaluated. NoOVD does not require this list, while MSPL generates region labels without a predefined novel vocabulary. Table 2 does not distinguish these materially different assumptions, so its reported margins cannot establish methodological superiority under comparable conditions.
    • Evaluation covers only 17 novel classes on OV-COCO and 3 on VisDrone, omitting OV-LVIS without explanation. No experiment tests retention of categories excluded from pseudo-label generation or robustness to inference vocabulary changes. The evidence therefore remains limited to a small, explicitly targeted novel vocabulary.
Questions for authors
  1. OV-LVIS, the standard benchmark used by NoOVD and other OVD work, is not evaluated, and the paper does not explain why. The indirect pathway models relationships across the whole input vocabulary, yet it is validated only at a limited vocabulary scale (65 OV-COCO and 10 VisDrone classes). Could you report OV-LVIS results (e.g., AP on the 337 rare classes), or explain why it was excluded? This would show whether ICTE and SoftmaxGate still help with a large vocabulary and a long-tailed class distribution.

  2. ICTE and SoftmaxGate make each category’s features depend on the other categories in the input vocabulary. Does ICA-PAN retain its benefit when this vocabulary changes? For example, please compare novel-category AP under novel-only and standard base-plus-novel prompts for both KR-OVD and the original YOLO-World baseline.

  3. Could you generate pseudo-labels for only a subset of novel classes and compare performance on the remaining novel classes against base-only fine-tuning? In particular, does training with these pseudo-labels cause additional degradation on novel classes that receive no pseudo-label supervision?

Reviewer_GVfuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes that when pre trained open vocabulary detectors are fine tuned using only base classes, they may lose their previously learned foreground perception and category discrimination abilities due to unlabeled novel objects being used as backgrounds. The author recovers this knowledge through two paths: one uses category aware pseudo labels to provide direct supervision, and the other uses ICA-PAN to model category relationships and inject semantic information into visual features.

Critical strengths
  1. The method combines pseudo-supervision with semantic transfer through two separate recovery pathways. This design is straightforward and makes the framework easy to follow.

  2. KR-OVD greatly improves the performance of new class detection, increasing the Novel AP50 on OV-COCO from 25.7 to 50.7, and on VisDrone2019 DET from 13.6 to 17.9, while maintaining basic class performance to a large extent.

Critical weaknesses
  1. Table 5 shows that adding ICA-PAN to class-specific pseudo-supervision improves Novel AP50 by only 0.2 points (50.5 to 50.7) on OV-COCO, while Base AP50 drops from 67.5 to 66.5. Moreover, ICA-PAN alone reduces Novel AP50 from 13.6 to 4.7 on VisDrone2019-DET. These results raise questions about the necessity of ICA-PAN and whether the two pathways are truly complementary.

  2. The paper claims to recover two types of knowledge, but the experiments mainly rely on detection performance to support this claim. Table 6 also shows that most of the improvement comes from SoftmaxGate (25.7 to 43.2 Novel AP50), while ICTE brings only a 0.7-point gain. Further analysis is needed to clarify whether the method actually recovers inter-class semantic knowledge or simply improves feature representations.

  3. The direct recovery pathway assumes that novel-class names are known and reliable pseudo labels can be generated during training. However, these conditions may not hold in real-world. Table 5 also shows that most improvements come from class-specific pseudo-supervision, while ICA-PAN alone performs poorly on VisDrone2019-DET.

Questions for authors
  1. What evidence shows that class-agnostic foreground knowledge and class-discriminative semantic knowledge are actually recovered?

  2. Table 6 shows that ICTE improves Novel AP50 by only 0.7 points over SoftmaxGate alone. How can the authors show that the improvement comes from inter-class semantic modeling rather than general feature enhancement?

  3. Why does ICA-PAN alone improve novel-class detection on OV-COCO but significantly reduce performance on VisDrone2019-DET?

  4. How does KR-OVD perform when novel-class names are unavailable during adaptation and class-specific pseudo labels cannot be generated?

Reviewer_ft8DOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes KR-OVD, an open-vocabulary object detection framework designed to mitigate the significant degradation in novel-class detection performance when pretrained detectors are fine-tuned using only base-class annotations. The authors attribute this degradation to unlabeled novel-class instances being treated as background, which suppresses foreground knowledge and class-discriminative semantic knowledge.

Critical strengths
  1. Clear and Meaningful Research Motivation. This paper addresses the significant degradation in novel-class detection performance when pretrained OVD models are fine-tuned using only base-class annotations.
  2. Significant Performance Improvements. Experiments on OV-COCO and VisDrone2019-DET demonstrate substantial improvements in novel-class AP50, from 25.7 to 50.7 and from 13.6 to 17.9, respectively.
Critical weaknesses
  1. Weak Connection Between Motivation and Method Design, with Limited Methodological Novelty. The paper analyzes the degradation of novel-class detection performance from the perspectives of class-agnostic foreground knowledge and class-discriminative semantic knowledge. However, the proposed method does not explicitly design targeted recovery mechanisms for these two types of knowledge. The direct recovery pathway mainly relies on conventional pseudo-label supervision, while the indirect recovery pathway combines existing operations such as Self-Attention, Cross-Attention, Gating, and FiLM. The connections between these design choices and the goal of knowledge recovery are not sufficiently explained. In particular, it remains unclear how these operations facilitate base-to-novel knowledge transfer, weakening the method's novelty and technical justification.
  2. Insufficient Validation of the Core Modules and Knowledge Transfer Mechanism. On OV-COCO, pseudo-label supervision alone achieves 50.5 novel-class AP50, while adding ICA-PAN only marginally improves it to 50.7. Moreover, the ICTE module, which explicitly models inter-class relationships, provides an additional improvement of only 0.7 points. On VisDrone, using ICA-PAN alone decreases novel-class AP50 from 13.6 to 4.7. These results suggest that the contribution of ICA-PAN is limited and dataset-dependent. Furthermore, the current experiments do not sufficiently demonstrate whether the performance gains arise from inter-class knowledge transfer rather than general vision-language feature fusion.
Questions for authors
  1. Connection Between Research Motivation and Knowledge Recovery Mechanisms. The paper attributes the degradation in novel-class detection performance to the suppression of class-agnostic foreground knowledge and class-discriminative semantic knowledge. However, the connections between the proposed recovery pathways and these two types of knowledge remain unclear. Could the authors further explain how each component specifically contributes to recovering these two types of knowledge? Is there any more direct experimental evidence demonstrating that the corresponding knowledge is indeed recovered, rather than simply improving detection performance through additional supervision or feature fusion?
  2. Inter-Class Semantic Interaction During ICA-PAN Training. ICA-PAN employs Self-Attention to model semantic relationships among different categories, but it is not sufficiently explained how this mechanism operates under the base-only training setting. Are both base and novel class text embeddings included during training? If only base classes are used, how does the model learn to transfer knowledge to unseen novel classes? If both are included, how can the inter-class attention mechanism effectively facilitate semantic transfer without introducing interference from irrelevant categories, particularly in the absence of ground-truth annotations for novel classes?
  3. Contribution and Effectiveness of ICA-PAN. On VisDrone, using ICA-PAN alone reduces novel-class AP50 from 13.6 to 4.7. How do the authors explain this substantial performance degradation? Furthermore, could the authors provide comparisons with standard Cross-Attention, Gating, or other conventional feature fusion methods to better demonstrate that the improvements of ICA-PAN arise from base-to-novel semantic transfer rather than general vision-language feature enhancement?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.