acceptodds
Withdrawn from ICLR 2027

Towards Less Base-Biased Open-Vocabulary Object Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Frozen-backbone detectors have emerged as an effective paradigm for Open-Vocabulary Object Detection (OVOD), as they preserve transferable knowledge from pretrained vision-language models. However, freezing the backbone alone does not eliminate detector-level base bias: base-only optimization can distort novel-region representations, while region proposal networks trained on base categories often produce spatially inaccurate proposals for unseen objects. To address these issues, we propose ToLB, a framework Towards Less base-Biased OVOD that tackles representation and localization bias at training and inference, respectively. Specifically, we propose a Relational Knowledge Preservation module that employs a frozen visual foundation model as a category-agnostic structural teacher and regularizes region representations through pairwise relational consistency, avoiding restrictive alignment between heterogeneous feature spaces. At inference, we further introduce a VLM-Guided Region Refinement module that extracts category-agnostic objectness cues from dense vision-language features to identify foreground-dominant regions and adaptively refine proposals, thereby reducing background interference for novel objects. Extensive experiments on OV-COCO and OV-LVIS demonstrate superior performance in novel-category generalization, achieving 48.2% Novel AP50 on OV-COCO and 39.5% Rare mask mAP on OV-LVIS, respectively.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary Object Detection, Vision-Language Models, Region Representation, Novel-Category Generalization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_h1cbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper is motivated by two problems of OVOD, including representation drift and spatial bias of novel regions. Two methods are proposed targeting these two problems. RKP aims to mitigate the representation drift by distilling the similarity correlations of DINOv2, and VRR tries to refine the bounding boxes generated by RPN.

Critical strengths
  1. This paper is well-motivated. I agree with the discussion on the representation drift and the spatial bias, which are important problems of OVOD.
  2. The experiments is concise and solid, including the comparison with SoTA and the ablation studies, which demonstrates the substantial improvements brought by RKP and VRR.
Critical weaknesses
  1. About RKP: The logical link between the representation drift and the proposal of RKP is not clear. The authors are suggested to answer my question #1.
  2. About VRR: To my opinion, the proposal of VRR is a heuristic post-process based on experience rather than a well-designed method. The authors are suggested to answer my question #2.
  3. About experiments: The comparison with SoTA faces potential unfairness due to the external model DINOv2, which requires further explanation.
Questions for authors
  1. About RKP: Please explain why RKP helps mitigate the representation drift. Are the vanilla RPN loss and classification loss unchanged during training? If so, would the RPN be trained to ignore novel regions and do not generate any proposal? Meanwhile, would the detection head be trained to misclassify the novel regions? These problems are likely due to the loss functions calculated on base classes, but RKP makes no modifications on existing loss functions and only adds a new one.
  2. About VRR: Please explain why some edges of bounding-boxes even expand after VRR (e.g., the upper boundary of the cake in Figure 5) ? Besides, Table 4 shows that VRR improves the AP50 by 1.8, but in Table 8, the gap between 40% and 100% Top-K values are only 0.3. Does VRR with a 100% Top-K value the same as the baseline? If so, why is there a gap between Table 4 and Table 8?
Reviewer_itkfOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies open-vocabulary detection in a frozen-representation setting, where pretrained foundation-model features are kept fixed to preserve their generalization ability and avoid expensive end-to-end adaptation. The authors argue that such frozen OVD pipelines still suffer from limited region-level knowledge transfer and inaccurate localization of novel objects.

To address these issues, the paper introduces two components: 1.RKP, which acts as a training-stage regularizer to encourage better region-level knowledge transfer from pretrained representations, 2.VRR, an inference-only proposal refinement module that uses CLIP-derived foreground/objectness cues to adjust proposal coordinates.

Built on a two-stage detector, the overall framework aims to improve detection performance while keeping the underlying foundation-model representations frozen.

Critical strengths
  1. The paper tackles an important challenge in frozen-representation open-vocabulary detection: improving novel-object localization without fine-tuning the underlying foundation models.

  2. The method is practically appealing, as it improves detection performance while preserving frozen foundation-model representations and avoiding expensive end-to-end retraining.

  3. The experimental results show consistent gains across the evaluated benchmarks, suggesting that the proposed components can effectively improve the frozen-representation OVD pipeline.

Critical weaknesses
  1. The choice of CLIP as the objectness source for VRR is insufficiently justified. According to the paper’s motivation, VRR aims to improve spatial foreground localization and proposal refinement. This task is arguably closer to class-agnostic objectness estimation or foreground segmentation than to CLIP’s image-text semantic alignment objective. In particular, the paper itself argues that DINOv2 provides stronger spatial coherence and structural priors than CLIP. Therefore, the authors should explain why VRR uses CLIP rather than DINOv2 or other objectness/localization-oriented alternatives, such as SAM-based segmentation. I encourage comparisons with DINOv2-based grouping or affinity cues, SAM-based masks/segmentation, RPN objectness, class-agnostic foreground segmentation or saliency, and other mask-based refinement methods.

  2. Direct evidence for VRR's localization benefit is missing. Moreover, the current evaluation only reports improvements in final AP. The paper should provide direct localization metrics, such as proposal recall, best proposal IoU, average recall (AR), and AP75, rather than relying only on final AP50.

  3. VRR may compensate for foreground evidence that RKP does not directly exploit during training. According to the authors' results, CLIP objectness is sufficiently reliable to directly modify proposal coordinates at inference time. If so, the authors should explain why this cue is not incorporated more directly during training, for example through soft objectness supervision, background-loss reweighting/ignoring, or training-time proposal refinement. Otherwise, VRR may appear to compensate for RKP's failure to directly exploit the mined foreground evidence, rather than addressing an independent localization failure mode.

Questions for authors
  1. Could the limited use of CLIP-derived foreground cues during training be a consequence of the frozen detector design? Would end-to-end optimization allow these cues to be incorporated more directly into objectness and localization learning?

  2. However, since the method is explicitly developed for frozen-representation OVD, the limitation in Question 1 may be inherent to this setting rather than specific to RKP/VRR. The paper should therefore better discuss the motivation and advantages of the frozen two-stage design, and position it relative to end-to-end OVD approaches that may exploit foreground cues more directly during training.

Reviewer_YFiPOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ToLB, a framework for mitigating base-category bias in frozen-backbone open-vocabulary object detection. The authors identify two limitations: representation drift caused by base-only supervision and inaccurate region proposals for novel objects. To address these issues, ToLB introduces Relational Knowledge Preservation (RKP), which uses a frozen visual foundation model to preserve inter-region relationships during training, and VLM-Guided Region Refinement (VRR), which leverages category-agnostic objectness cues to refine proposals at inference. Experiments on OV-COCO and OV-LVIS demonstrate improvements in novel-category detection performance across different backbone sizes.

Critical strengths
  1. Clear motivation. The paper addresses detector-level base bias in frozen-backbone OVOD, considering both representation preservation and proposal refinement.

  2. Consistent performance improvements. The proposed method improves novel-category performance on both OV-COCO and OV-LVIS across different backbone sizes, with ablation studies demonstrating the contributions of its components.

Critical weaknesses
  1. Limited methodological novelty. The potential novel-region mining strategy closely resembles NoOVD, while the relational knowledge preservation mechanism shares substantial similarities with DeCLIP. Although these techniques are applied at different stages, the proposed contributions appear largely incremental.

  2. Insufficient evidence for the claimed localization bias. The paper argues that base-trained RPNs exhibit significant localization bias toward novel objects, but provides no quantitative evidence to support this claim. Moreover, the observed AP50 improvements do not necessarily indicate better localization, as they may also result from improved classification.

  3. Insufficient validation of the proposed mechanisms. The paper does not clearly establish whether RKP effectively mitigates novel-category semantic drift or whether VRR improves localization through foreground selection. In particular, the near-optimal performance when retaining 100% of the objectness locations raises questions about the actual contribution of the proposed refinement mechanism.

Questions for authors
  1. Novelty: How does the proposed potential novel-region mining differ fundamentally from NoOVD? Similarly, what is the key ethodological distinction between RKP and the relational knowledge preservation strategy in DeCLIP?

  2. Localization bias: Could the authors provide quantitative evidence that RPN proposals exhibit greater localization errors for novel categories than for base categories? It would also be valuable to compare localization quality before and after RoI box regression to clarify whether the claimed bias persists in the final predictions.

  3. Effectiveness of VRR: Could the authors report localization-specific metrics, such as proposal IoU, recall, and AP75, before and after VRR? Additionally, why does retaining 100% of the objectness locations in Table 8 still achieve 42.2 Novel AP50, close to the best result of 42.5? What accounts for this improvement when no foreground locations are excluded?

  4. Effectiveness of RKP: Since preserving pairwise region similarities does not necessarily preserve their semantic alignment with text embeddings, could the authors provide more direct evidence that RKP mitigates novel-category semantic drift rather than merely improving relational consistency?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.