acceptodds
Under review as a conference paper at ICLR 2027

Beyond Known-Class Prompts: Sparse Factor Objectness for Open-World Object Detection

Abstract

Open-world object detection (OWOD) requires recognizing known objects while discovering unknown ones. Open-vocabulary detectors (OVDs) provide strong visual representations but rely on class-name prompts for detection. Without unknown-class prompts, unknown objects may be missed even when their region representations retain informative object cues. We propose a Sparse-AutoEncoder-based Open-Vocabulary Open-World framework for OWOD (SAE-OVOW) to recover class-agnostic objectness directly from a frozen OVD without unknown-class annotations. SAE-OVOW uses a sparse autoencoder to decompose known-aligned region embeddings into sparse latent factors. It then selects factors that are reliably activated, exhibit similar distributions across known-aligned and unmatched regions, and have diverse decoder directions. The selected activations are aggregated into sparse factor objectness and combined with known-class uncertainty for unknown detection. Experiments on M-OWODB and S-OWODB show that SAE-OVOW consistently improves U-Recall while maintaining competitive known-class performance. On S-OWODB Task 2, it improves U-Recall by 12.7 points and overall mAP by 5.3 points over the best competing results. Latent removal-and-recovery experiments verify the functional relevance of the selected factors, while Task-1 post-hoc audits on M-OWODB and S-OWODB identify coherent visual patterns in 119 of 200 factors.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open world object detection, sparse autoencoder, unknown object detection, open-vocabulary object detection

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yHLbOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAE-OVOW, a framework for open-world object detection that extracts classagnostic objectness from frozen open-vocabulary detectors without unknown-class annotations or predefined attribute descriptions. A sparse autoencoder is trained on known-aligned region embeddings to learn latent factors. The method selects factors based on activation reliability, distributional similarity measured by Jensen Shannon divergence, and decoder diversity. Selected factor activations are aggregated into an objectness score and combined with known-class uncertainty for unknown detection. Experiments on M-OWODB, S-OWODB, and nu-OWODB demonstrate improvements in unknown-object recall while maintaining competitive known-class performance. Ablation studies, latent interventions, and semantic audits further examine the effectiveness of the selected factors.

Critical strengths
  1. Meaningful Methodological Contribution. The paper presents a well-motivated approach to extracting class-agnostic objectness from frozen open-vocabulary detectors without unknownclass annotations or predefined attribute descriptions. Learning sparse latent factors directly from region embeddings provides a promising alternative to language-dependent attribute modeling while preserving the original known-class prediction pathway.
  2. Comprehensive Experimental Validation. The paper provides extensive evaluations across multiple OWOD benchmarks, supported by component ablations, alternative representation comparisons, hyperparameter sensitivity analyses, and robustness experiments. Particularly, the latent removal-and-recovery experiments provide additional evidence that the selected factors are functionally relevant to unknown-object detection, strengthening the empirical support for the proposed approach.
Critical weaknesses
  1. Missing Comparison with the Closest Related Work. The paper lacks a direct comparison with OW-OVD (CVPR 2025), which employs a closely related JSD-based feature selection and redundancy reduction strategy. Without a controlled comparison, it remains unclear whether SAEderived factors provide meaningful improvements over existing attribute-based representations, weakening the empirical justification for the claimed contribution.
  2. Insufficiently Controlled Objectness Comparisons. In Table 5(c), alternative objectness branches achieve substantial unknown recall but suffer severe degradation in known-class mAP. Since unknown and known scores compete during post-processing, differences in score calibration may affect these results. Comparisons at matched operating points would better establish whether the improvements stem from superior objectness representations.
Questions for authors
  1. Relation to OW-OVD. OW-OVD (CVPR 2025) employs a similar JSD-diversity selection strategy and reports higher S-OWODB U-Recall under its own settings. Could the authors clarify the key methodological differences and explain its absence from the comparisons? A matchedbackbone comparison would help establish the added value of SAE-derived factors.
  2. Contribution to Incremental Detection. In the VOC incremental experiments, the detector remains frozen while new classes are recognized through vocabulary expansion. How much of the reported performance comes from the SAE branch rather than the pretrained detector's zero-shot capability? A baseline without SAE would help clarify this contribution.
1 comment
AuthorsOpenReview ↗

### Additional Clarification: Interpretability Beyond Detection Performance

Additional Clarification: Interpretability Beyond Detection Performance

We appreciate your recognition of our latent intervention and semantic audit analyses. We would like to further clarify their significance as evidence supporting an important interpretability contribution of SAE-OVOW.

Our method recovers sparse visual factors directly from frozen OVD representations without predefined attributes or unknown-class annotations. The post-hoc audit identifies coherent visual patterns in 119 of 200 factors, and latent removal-and-recovery experiments establish their functional relevance to unknown detection.

Together, these results provide an interpretable connection between internal visual representations and unknown-object discovery. This contribution complements the incremental detection results by explaining what internal evidence supports the additional objectness branch.

We would appreciate your feedback on this aspect and whether our additional evidence addresses your remaining concerns.

Reviewer_HmCDOpenReview ↗
Rating
3
Confidence
1(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SAE-OVOW, an open-world object detection framework that extracts class-agnostic objectness from frozen open-vocabulary detector features. It employs a sparse autoencoder to learn latent factors, selects transferable directions through activation reliability, distribution similarity, and decoder diversity, and combines sparse factor objectness with known-class uncertainty. Experiments across several OWOD benchmarks demonstrate improved unknown-object detection while maintaining competitive known-class performance.

Critical strengths
  1. This paper presents an intriguing perspective on how to extract class-agnostic "objectness" from the internal representations of a frozen open-vocabulary detector.

  2. The proposed framework for sparse factor learning and selection is well-motivated; it requires neither annotations for unseen classes nor fine-tuning of the detector.

  3. Extensive experiments demonstrate that this method is highly competitive for open-world object detection (OWOD) tasks. Furthermore, the paper validates its effectiveness through component ablation studies, comparisons of different representation schemes, and investigations into feature intervention.

Critical weaknesses
  1. The rationale behind JSD-based factor selection remains insufficiently validated. Since unmatched regions contain both unknown objects and background, low JSD does not necessarily indicate objectness-related factors. The authors should examine how the unknown/background composition affects factor selection and detection performance.

  2. Table 4 compares methods with different backbones and training paradigms. A comparison with the same frozen YOLO-World under identical vocabulary expansion, but without SAE, is needed.

  3. The advantage over simple object-prompt baselines is not fully established. Table 5(a) shows substantially higher U-Recall but slightly lower U-mAP and known-class mAP than the generic object-prompt baseline. Precision–recall and known–unknown trade-off comparisons under matched settings would clarify the overall benefit.

  4. Generalization across different OVD architectures remains unverified. Although Table 6 evaluates different YOLO-World scales, all experiments rely on the same detector family. Evaluation on another OVD architecture would strengthen the claimed general applicability.

Questions for authors

I would appreciate further quantitative evidence addressing the key concerns raised above, particularly regarding the validity of JSD-based factor selection given the mixture of unknown objects and background, the effectiveness of SAE compared with the same frozen YOLO-World baseline without SAE, the precision–recall and known–unknown trade-offs relative to simple object-prompt baselines, and the generalizability of the proposed method across different OVD architectures. These issues are important for establishing the effectiveness and broader applicability of the proposed framework. If the authors cannot provide convincing quantitative experiments to adequately address these concerns, I may consider lowering my score.

2 comments
AuthorsOpenReview ↗

### Additional Clarification: Interpretability of Selected Factors

Additional Clarification: Interpretability of Selected Factors

We would also like to emphasize that interpretability is an important objective of SAE-OVOW, alongside unknown-object discovery.

The selected sparse factors provide analyzable internal evidence for objectness rather than an opaque detection score. Our post-hoc semantic audit identifies coherent visual patterns in 119 of 200 factors. Furthermore, the selected factors exhibit activation rates of 0.76/0.47/0.19 on unknown objects, known objects, and background, respectively, while latent removal-and-recovery experiments verify their functional relevance to unknown detection.

These analyses complement the additional JSD robustness experiments by connecting factor selection to interpretable and functionally relevant object evidence.

We would appreciate your feedback on this contribution and whether our additional experiments address your remaining concerns.

Thank you for the detailed rebuttal and additional experiments. I find this work interesting, and I appreciate the authors' efforts in addressing my concerns.

However, after considering the comments from other reviewers, particularly Reviewer 367o's concerns about the precision–recall trade-off, I have become somewhat more cautious about my assessment. While the substantial improvements in unknown recall are encouraging, I am not fully convinced that the overall contribution is strong enough to warrant a score of 3. If intermediate scores were available, I would lean toward 2.5.

Nevertheless, I will maintain my original score of 3, while lowering my confidence to 1 to reflect my uncertainty in this assessment. I hope the authors will continue to improve this promising work.

Reviewer_367oOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes SAE-OVOW, a method for detecting unknown objects using internal representations of a frozen open-vocabulary detector. A sparse autoencoder (SAE) learns latent factors from known-aligned region embeddings. Factors are selected using activation reliability, Jensen–Shannon divergence (JSD), and decoder diversity, then combined with known-class uncertainty to identify unknown objects.

Experiments on multiple open-world detection benchmarks demonstrate improvements in unknown recall, supported by component ablations and feature-level analyses.

Critical strengths
  1. An interesting approach to discovering unknown objects without unknown-class supervision.

SAE-OVOW explores class-agnostic objectness from frozen visual representations rather than relying on predefined language attributes or unknown-class annotations.

The method achieves strong unknown-recall results across M-OWODB and S-OWODB while maintaining competitive known-class performance. The reported WI and A-OSE results also provide useful evidence that the gains are not simply due to indiscriminate unknown predictions.

  1. Useful experimental and representation-level analyses.

The paper compares sparse representations with PCA, random directions, dense autoencoders, and conventional objectness branches. The activation analysis and removal-and-recovery experiments support the functional relevance of selected factors. Additional sensitivity and seed-robustness experiments strengthen the empirical study.

Critical weaknesses
  1. The methodological contribution over closely related attribute-based approaches remains insufficiently established.

SAE-OVOW shares important design principles with OW-OVD (Xi et al., 2025b), including distribution-based selection of transferable features and combining the resulting evidence with known-class uncertainty.

The principal distinction appears to be replacing language-defined attributes with SAE-learned latent factors. This is potentially useful, but the paper does not directly establish the advantages of learned sparse factors over attribute-based representations under comparable conditions.

Without a controlled comparison with OW-OVD or an equivalent attribute-based alternative, the independent value of the proposed representation remains difficult to assess.

  1. The unknown-detection gains raise concerns about precision, score competition, and factor-selection robustness.

Table 5(a) shows that SAE-OVOW increases unknown recall from 20.8% to 66.0% compared with the generic object-prompt baseline, while unknown mAP changes from 5.5% to 5.3%.

The substantial recall improvement is valuable, but it does not establish improved unknown-detection precision or ranking quality. In the same comparison, known-class mAP decreases from 73.5% to 66.6%.

Furthermore, JSD-based selection compares known-aligned regions with an unmatched pool containing an unknown mixture of objects and background. Although the paper provides encouraging activation analyses, it does not establish whether factor selection remains reliable when this mixture changes substantially.

These issues limit the strength of the claimed improvement in overall unknown-detection quality.

Questions for authors
  1. What measurable advantage do SAE-learned factors provide over the attributes used by OW-OVD?

Could the authors compare SAE-OVOW against OW-OVD under matched detector, benchmark split, proposal-generation, and post-processing settings?

If a direct reproduction is impractical, a controlled comparison replacing the SAE-derived factors with language-defined attributes within the same frozen-detector pipeline would help isolate the contribution of the learned representation.

Please report known-class mAP, unknown recall, and unknown mAP.

  1. Does the recall improvement correspond to better unknown-object detection at comparable false-positive rates?

For the generic object-prompt baseline and SAE-OVOW in Table 5(a), please provide unknown precision–recall curves and recall at matched false positives per image, or precision at matched recall.

These comparisons would clarify whether SAE-OVOW provides a better detection trade-off or primarily increases the number of unknown predictions.

  1. Why does known-class mAP differ substantially between the generic object-prompt baseline and SAE-OVOW?

Although the known-class detector is frozen, the additional unknown scores can affect the final predictions through score competition and NMS.

Please report the known-class mAP of frozen YOLO-World using only known-class prompts, and explain the generic object-prompt scoring procedure.

A comparison of known predictions before and after the final known/unknown post-processing steps would help determine whether the 73.5% versus 66.6% difference is caused by unknown-score competition.

  1. How robust is JSD-based factor selection to changes in the unmatched-region composition?

A controlled diagnostic could vary the proportion of background proposals in the unmatched pool while keeping the trained SAE and selection budget fixed.

Please report changes in selected-factor overlap and unknown detection performance, especially U-mAP. Evaluation annotations could be used for this diagnostic analysis without introducing unknown labels into the proposed training procedure.

This would directly test whether low-JSD selection remains effective when the unmatched pool becomes more background-dominated.

Additional Comments

  1. Objectness-score interpretation. Eq. (15) assigns zero when no selected factor is active and otherwise aggregates sigmoid-transformed positive activations. The resulting score has a restricted, discontinuous range and should not automatically be interpreted as a calibrated probability. Please clarify its interpretation and discuss how its scale affects the combination with known-class confidence in Eq. (16).

  2. Presentation of incremental tasks. Section 4 would benefit from a brief description of the class progression across Tasks 1–4 rather than relying entirely on Appendix A.1. The incorrect reference to Appendix A.2 for dataset splits should also be corrected.

Overall Assessment

SAE-OVOW investigates an interesting direction for extracting class-agnostic objectness from frozen detector representations. The experimental results demonstrate substantial unknown-recall gains, and the representation-level analyses are useful.

However, the independent methodological contribution relative to OW-OVD is not adequately established. More importantly, the unknown-recall improvements do not clearly translate into better unknown average precision, and the effects of unknown-score competition and unmatched-region composition remain insufficiently explained.

A matched comparison with attribute-based methods, more informative precision–recall analyses, and clarification of the known-class performance differences would be important for reconsidering this assessment.

1 comment

Thank you for the detailed responses and substantial revisions.

The additional experiments and clarifications have addressed many of my original concerns, significantly improved the clarity of the paper, and provided stronger evidence for its contributions.

However, I still have reservations about the fundamental precision–recall trade-off highlighted in Figure 4(a). While unknown-object recall improves substantially, the very low precision at higher recall levels remains an important limitation of the approach.

Overall, I appreciate the authors’ efforts and the improvements made during the rebuttal. Based on the revised manuscript and additional evidence, I have raised my overall rating from 1 (Clear Reject) to 2 (Weak Reject).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.