acceptodds
Under review as a conference paper at ICLR 2027

Neighborhood-Aware Representation Learning for Open-World Object Detection

Abstract

Open World Object Detection (OWOD) extends conventional object detection beyond a fixed label space, enabling unknown object discovery and incremental learning of new classes. Representative objectness-based approaches commonly rely on global probabilistic foreground modeling or reduce the coupling between objectness and known class prediction. However, they typically treat known class discrimination and foreground modeling separately, leaving the connection between the two objectives underexplored. We therefore introduce Prototype Relations for Open World object detection (PROW), a neighborhood-aware representation learning framework that exploits neighborhood relations among known class prototypes as a shared structure for local class discrimination and multi-component foreground modeling. PROW maintains a dynamic graph over class prototypes, with neighborhood relations updated from matched positive queries. The graph identifies locally confusable prototypes as informative negatives for local class discrimination, while the same graph organizes class prototypes into multiple neighborhood-conditioned foreground components. For each query, PROW selects the most compatible foreground component to estimate class-independent objectness. In subsequent tasks, confidence-filtered pseudo-labels from the previous detector and replay samples provide complementary supervision to preserve previously learned knowledge. Compared with the best prior results, PROW achieves absolute gains of - points in unknown object discovery, with a reduction in confusion between known and unknown objects.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open world object detection, incremental learning, representation learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_VrSwOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes PROW, a neighborhood-aware representation learning framework for open-world object detection (OWOD). Its key idea is to reuse neighborhood relations among known-class prototypes as a shared relational structure for two objectives: local class discrimination and multi-component foreground modeling.

Critical strengths

Complete incremental design: inherited prototypes/graph, frozen-teacher pseudo labels, and replay are combined; ablations show pseudo labels mainly preserve known classes while replay mainly improves U-Recall.

Strong empirical results: leading U-Recall and K-mAP on most M-OWODB tasks, clear A-OSE reduction, and reasonably thorough ablations and hyperparameter sensitivity.

Lightweight: only about 2% additional parameters relative to the base detector.

Critical weaknesses

Limited detector/backbone scope: only Deformable DETR + ResNet50 + DINO pretraining is evaluated. Other DETR variants or stronger backbones is unverified.

Graph construction depends on matched positive query quality: both graph updates and local discrimination rely on matched positives. Inaccurate matching, few samples, or class imbalance may introduce noisy relations.

Heuristic component threshold: the mean off-diagonal adjacency weight is used as the grouping threshold, and the number of components may fluctuate during training; stability and its effect on objectness are not fully analyzed.

Questions for authors

How does the number of components Mt evolve during training? Can the mean threshold cause large fluctuations in Mt? Could you report sensitivity of Mt?

What are the precision and recall of pseudo labels? Over longer task sequences, can frozen-teacher pseudo labels accumulate errors?

Can the method transfer to other DETR variants or CNN detectors?

1 comment

Q1. Table 6 reports the actual component counts and full size distributions on M-OWODB. As the known classes expand from 20 to 40, 60, and 80, the recorded graphs contain 3, 5, 7, and 9 components. Component sizes range from 4–12, 4–14, 4–14, and 3–21 prototypes, respectively. The largest component contains 60% of prototypes in Task 1 and at most 35% in Tasks 2–4. Across the reported tasks, the learned prototype graph produces multiple foreground components, with no collapse into a single connected component and no singleton components.

Figure 3 further shows how the connected-component structure of the learned prototype graph evolves during training. Task 1 maintains three components over epochs 13–39, while Task 2 maintains five components over epochs 78–109. Adjacent-epoch ARI remains 1.0 over epochs 14–39 in Task 1 and 80–109 in Task 2, indicating that class prototypes retain the same foreground-component assignments during these intervals. Edge Jaccard also exhibits smaller fluctuations during later training, showing more consistent overlap between consecutive edge sets.

Appendix D.2 examines threshold sensitivity on the same fixed Task-1 graph. Thresholds of 0.9, 1.0, and 1.1 times the mean off-diagonal edge weight all yield three components of sizes 12, 4, and 4. Thus, the component count and size distribution remain unchanged under the tested ±10%\pm 10\% threshold perturbations.

Manuscript revision: We add graph statistics and dynamics to Section 4.4 (Table 6 and Figure 3), with threshold sensitivity analysis in Appendix D.2.

Q2. As detailed in Appendix A.2, we report the precision and recall of old-class pseudo-labels across the three task transitions, which are 89.5%/33.4%, 94.3%/32.7%, and 91.1%/29.2%, respectively, at IoU≥0.5\mathrm{IoU} \geq 0.5. The consistently high precision suggests that pseudo-label noise is relatively limited, despite the lower recall. Moreover, Table A.2 shows consistent improvements with replay across subsequent tasks. Specifically, replay improves U-Recall and K-mAP from 35.7 to 38.7 and from 50.6 to 51.7 on Task 2, respectively. On Task 3, the corresponding improvements are from 29.8 to 32.7 and from 44.4 to 46.4. On Task 4, K-mAP increases from 42.1 to 44.8. These results suggest that replay helps mitigate the impact of imperfect pseudo-labels. While error propagation across tasks cannot be entirely ruled out [1], these results provide empirical support for the effectiveness of replay.

Manuscript revision: We add the ablation results of pseudo-label supervision and replay across incremental tasks in Table A.2 (Appendix A.2).

Q3. PROW can also transfer beyond DETR to a CNN detector. We implement it on RandBox-based Fast R-CNN using the same training and inference configuration as OrthogonalDet, which differs from the main Top-100 setting. Table B.1 shows that PROW improves U-Recall by 9.0/5.6/4.4 percentage points and reduces A-OSE by 584/136/238 on M-OWODB Tasks 1–3. K-mAP also increases by 1.5/1.3/1.1/0.2 percentage points across Tasks 1–4.

Manuscript revision: We add experiments evaluating transfer to a CNN-based detector in Appendix B (Table B.1).

References

[1] Qiuchen Wang, Zehui Chen, Chenhongyi Yang, Jiaming Liu, Zhenyu Li, and Feng Zhao. PseDet: Revisiting the Power of Pseudo Label in Incremental Object Detection. ICLR, 2025.

Reviewer_2diSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes PROW, a neighborhood-aware representation learning framework for open-world object detection. PROW maintains a dynamic graph over known-class prototypes and reuses the resulting neighborhood structure for two purposes: selecting locally confusable prototypes for metric discrimination and constructing multiple foreground components for class-independent objectness estimation. The method further employs pseudo-labeling and replay during incremental learning. The paper is clearly written, the central shared-structure motivation is intuitive, and the reported ablations suggest that both local discrimination and multi-component foreground modeling contribute to performance.

Critical strengths
  1. PROW provides a coherent unified formulation for OWOD. It reuses prototype-neighborhood relations for both local class discrimination and multi-component foreground modeling, giving the method a clear conceptual motivation rather than combining unrelated modules.

  2. The empirical results are strong and reasonably well supported. PROW improves unknown-object discovery over the included baselines while maintaining competitive known-class detection, and the ablations support the contributions of the main components.

Critical weaknesses
  1. The CVPR 2026 paper Detecting Unknown Objects via Energy-based Separation for Open World Object Detection evaluates OWOD on both M-OWODB and S-OWODB and reports substantially higher U-Recall scores: 65.1 / 66.2 / 69.0 on M-OWODB and 68.7 / 62.9 / 60.7 on S-OWODB. In comparison, PROW reports 31.9 / 34.4 / 31.5 and 24.6 / 28.0 / 33.1 , respectively. However, that work explicitly states that its M-OWODB results are based on a corrected evaluation addressing the annotation-duplication issue , whereas this paper still reports some baseline numbers from the older evaluation setting. Therefore, it is currently unclear whether PROW and this recent work are evaluated using exactly the same benchmark version, annotations, and evaluation protocol . The authors should clearly specify the version of M-OWODB and the evaluation implementation used in this paper, and provide comparisons with recent methods such as DEUS under directly comparable settings.

  2. Multi-component foreground modeling is one of the core designs of PROW, yet the paper does not directly analyze the connected-component structure actually induced by the learned prototype graph. In particular, the paper does not report the number of components MtM_t in each task, the distribution of component sizes, or how these structures evolve across incremental tasks. As a result, it remains unclear whether the graph consistently produces multiple meaningful foreground components, or whether it may degenerate in some cases into a single large connected component or a large number of singleton components. Since the subsequent foreground objectness modeling directly depends on these graph-induced component structures, reporting such statistics would provide more direct evidence that the proposed mechanism operates as intended.

Questions for authors
  1. Could the authors clarify the exact version of M-OWODB, annotation files, and evaluation implementation used for PROW? In particular, does the reported evaluation incorporate the correction for the annotation-duplication issue discussed in recent work such as DEUS? If PROW is evaluated under the corrected setting, could the authors provide directly comparable results against DEUS or other recent methods under the same protocol? This clarification is important for determining whether the current SOTA comparison and related performance claims are valid.

  2. Could the authors report the actual connected-component statistics induced by the learned prototype graph, including the number of components MtM_t for each task and the distribution of component sizes? It would also be helpful to clarify how these structures change across incremental tasks. These results are important for assessing whether the proposed multi-component foreground modeling consistently produces meaningful multiple foreground components, rather than occasionally collapsing into a single large component or fragmenting into many singleton components.

1 comment

Q1. Our original submission follows the earlier M-OWODB annotation setting. We now use the corrected M-OWODB test annotations addressing duplicate test-image IDs. For a fair comparison, we also impose a joint Top-100 final-detection cap per image, ranking known and unknown predictions together by detection score. This choice is motivated by IPOW (ICML 2026) [1], which notes that excessive unknown proposals can increase U-Recall and limits post-NMS proposals to 100 per image.

We re-evaluate PROW, PROB, and OrthogonalDet using the corrected M-OWODB test annotations and the same joint Top-100 final-detection cap. For PROB and OrthogonalDet, we use their publicly available implementations. PROW outperforms both. On M-OWODB Tasks 1–3, PROW achieves U-Recall of 36.9/38.7/32.7. Relative to the task-wise best baseline results reported in Tables 1 and 3, PROW improves U-Recall by 3.9–12.9 percentage points and reduces A-OSE by 31.6%–60.3%. We also re-evaluate OrthogonalDet on S-OWODB under the same cap. PROW achieves U-Recall of 24.6/28.0/33.1 on Tasks 1–3. Relative to the best baseline values in Table 2, it improves U-Recall by 0.6 and 2.2 percentage points on Tasks 1 and 3, respectively, while falling 0.7 points below the best value on Task 2. These table-level comparisons include both re-evaluated results and published reference values.

We acknowledge DEUS's strong unknown-detection performance and its higher reported U-Recall on both benchmarks. A DEUS re-evaluation under our joint Top-100 final-detection cap and filtering order is not yet available. We therefore discuss its published results separately in the main paper and omit them from Table 1, without claiming a controlled head-to-head comparison.

We also implement PROW on the RandBox-based Fast R-CNN architecture. Under the matched OrthogonalDet training and inference configuration, which differs from the main Top-100 setting, Table B.1 shows U-Recall gains of 9.0/5.6/4.4 percentage points on M-OWODB Tasks 1–3, alongside higher K-mAP and lower A-OSE. Table B.2 in Appendix B.1 further shows that, with 20 sampling passes and no post-NMS detection cap, increasing proposals per pass from 100 to 500 raises PROW's Task-1 U-Recall from 59.7 to 70.6 but also increases A-OSE from 412,931 to 648,939. OrthogonalDet exhibits the same trend. Over this change, K-mAP increases from 65.9 to 66.9 for PROW and from 64.5 to 64.6 for OrthogonalDet. These results show that increasing the number of proposals can substantially increase unknown recall while worsening unknown-to-known errors.

Manuscript revision: We clarify the evaluation protocol and update Tables 1–3 in Section 4.2, and add cross-architecture and proposal-budget analyses to Appendix B (Tables B.1 and B.2).

Q2. Table 6 reports the actual component counts and full size distributions on M-OWODB. As the known classes expand from 20 to 40, 60, and 80, the recorded graphs contain 3, 5, 7, and 9 components. Component sizes range from 4–12, 4–14, 4–14, and 3–21 prototypes, respectively. The largest component contains 60% of prototypes in Task 1 and at most 35% in Tasks 2–4. Across the reported tasks, the learned prototype graph produces multiple foreground components, with no collapse into a single connected component and no singleton components.

Figure 3 further shows how the connected-component structure of the learned prototype graph evolves during training. Adjacent-epoch ARI remains 1.0 over epochs 14–39 in Task 1 and 80–109 in Task 2, indicating that class prototypes retain the same foreground-component assignments during these intervals. Edge Jaccard also exhibits smaller fluctuations during later training, showing more consistent overlap between consecutive edge sets.

Manuscript revision: We add component statistics and training-dynamics analysis to Section 4.4 (Table 6 and Figure 3).

References

[1] Xueqiang Lv, Shizhou Zhang, Yinghui Xing, Di Xu, Peng Wang, and Yanning Zhang. Knowing the Unknown: Interpretable Open-World Object Detection via Concept Decomposition Model. ICML, 2026.

Reviewer_T9LqOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper builds a dynamic graph of class prototypes to model relationships between known classes. It uses neighboring prototypes to improve class discrimination and groups them to estimate foreground objectness. During incremental learning, pseudo labels and replay are used to reduce forgetting. Experiments on M-OWODB and S-OWODB show improvements in unknown object detection and known-class performance.

Critical strengths
  1. The method uses a shared prototype-neighborhood structure for both class discrimination and foreground objectness estimation. This provides a clear connection between known-class detection and unknown object discovery.

  2. PROW achieves competitive results on M-OWODB and S-OWODB, particularly in unknown recall and known–unknown confusion. The ablation studies and additional analyses also provide useful support for the proposed design.

Critical weaknesses
  1. In Eq. (3), the weights of the selected Top-k neighbors are set to 1, but Eq. (4) uses these weights in the discrimination loss. If the graph is updated before computing the loss, the weighting may reduce to standard Top-k hard-negative learning.

  2. The method forms foreground components from connected prototypes, but their number, size, and stability are not reported. It is unclear whether the graph consistently produces meaningful components or sometimes collapses into one dominant group. Table 6 also shows that graph weighting improves U-Recall but not A-OSE.

Questions for authors
  1. In Eqs. (3) and (4), are the edge weights updated before or after computing the loss? If the selected edges are set to 1, how does the proposed loss differ from standard Top-k hard-negative learning?

  2. How does the proposed loss compare with an unweighted Top-k contrastive or margin-based loss? How much of the improvement comes from graph weighting?

  3. How many foreground components are typically formed during training and incremental learning? Are these components stable, and how sensitive are the results to the grouping threshold?

  4. In Table 6, weighted components improve U-Recall but slightly worsen A-OSE. Could the authors explain this trade-off?

  5. The paper does not analyze the sensitivity to λlocal\lambda_{\mathrm{local}} and λobj\lambda_{\mathrm{obj}}. How were these values chosen, and were they validated across different OWOD benchmarks?

1 comment

Q1. We compute the local loss in Eq. (4) before updating the graph in Eq. (3). Each mini-batch therefore uses the graph state from preceding mini-batches. For each matched query, we select the closest graph-supported negative prototypes and weight their margin-loss terms using the pre-update edge values. The current graph update affects only subsequent mini-batches and does not automatically assign unit weights to the negatives in the current loss. PROW uses the historical graph to guide negative selection and weighting. Under the same margin and normalization, its local objective reduces to an unweighted Top-k margin loss when the negative sets match and all selected weights equal 1.

Manuscript revision: We clarify the graph update order in Section 3.2, following Eq. (3).

Q2. Table C.1 evaluates edge weighting in the local margin loss. The control replaces the local-loss edge coefficients with 1 while keeping the same graph-supported negative-selection rule. On M-OWODB Task 1, it achieves 34.5 U-Recall, 66.1 K-mAP, and 1,743 A-OSE, compared with 36.9, 67.3, and 1,230 for PROW. Historical edge weighting therefore improves U-Recall and K-mAP by 2.4 and 1.2 percentage points, respectively, and reduces A-OSE by 513. These results demonstrate the effectiveness of historical edge weighting under the same graph-supported negative-selection rule.

Manuscript revision: We add the local edge-weighting ablation to Appendix C.1 (Table C.1).

Q3. Table 6 reports 3, 5, 7, and 9 connected components across M-OWODB Tasks 1–4 as the number of known classes increases from 20 to 80, with no singleton components in the recorded graphs. Figure 3 shows that Task 1 maintains three components with a largest size of 12 over epochs 13–39, while Task 2 maintains five components with a largest size of 14 over epochs 78–109. Adjacent-epoch ARI remains 1.0 over epochs 14–39 in Task 1 and 80–109 in Task 2. Edge Jaccard also exhibits smaller fluctuations and settles into a relatively stable range during later training. Together, these trends indicate stable component membership alongside continued updates to within-component edges.

Appendix D.2 examines threshold sensitivity on the same fixed Task-1 graph. Thresholds of 0.9, 1.0, and 1.1 times the mean off-diagonal edge weight all yield three components of sizes 12, 4, and 4. These results demonstrate stable component counts and sizes under the tested threshold perturbations on this graph.

Manuscript revision: We add graph statistics and training dynamics to Section 4.4 (Table 6 and Figure 3), and threshold sensitivity analysis to Appendix D.2.

Q4. We re-evaluate both foreground-weighting variants using the corrected M-OWODB test annotations, as requested by the reviewer 2diS. Weighting improves U-Recall from 34.5 to 36.9 and K-mAP from 66.2 to 67.3, while increasing A-OSE from 1,198 to 1,230, a relative increase of 2.7%. Since missed unknown objects do not contribute to A-OSE, lower A-OSE alone does not establish better known–unknown discrimination. We therefore report this increase as a trade-off accompanying the gains in unknown recall and known-class detection.

Manuscript revision: We update the foreground-weighting ablation using corrected M-OWODB annotations (Table 7 (originally Table 6)).

Q5. Tables C.2 and C.3 examine sensitivity to the two loss coefficients using the corrected M-OWODB test annotations. On Task 2, increasing λlocal\lambda_{\mathrm{local}} from 1.0 to 1.5 raises U-Recall from 38.7 to 38.9 but reduces K-mAP from 51.7 to 51.3. We therefore retain λlocal=1.0\lambda_{\mathrm{local}}=1.0, which gives the highest K-mAP among the tested settings. On Task 1, λobj=10−3\lambda_{\mathrm{obj}}=10^{-3} gives the highest K-mAP and lowest A-OSE among the tested settings, reaching 67.3 and 1,230, respectively. Doubling λobj\lambda_{\mathrm{obj}} improves U-Recall by only 0.1 percentage points while worsening both other metrics. These results support the default coefficients, which we use on both M-OWODB and S-OWODB. The sensitivity analyses cover M-OWODB only.

Manuscript revision: We add sensitivity analyses for both loss coefficients to Appendix C.2 (Tables C.2 and C.3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.