acceptodds
Under review as a conference paper at ICLR 2027

UniOSE: Semantics-Enriched Object Queries for Open-Ended Detection and Segmentation

Abstract

Open-ended object detection and segmentation require discovering objects and generating their category names without a predefined category vocabulary. Existing methods typically generate names from object representations produced by a detector. We find that insufficient category semantics in these representations limit naming accuracy, while enriched object semantics can further improve detection and segmentation. We therefore propose UniOSE, a unified framework that constructs semantics-enriched object queries to support both category generation and spatial prediction. Our Semantics-Enriched Object Namer combines detector features with region-level semantics from a pretrained vision–language model, providing richer semantic representations for category generation. Building on these representations, the Semantics-Guided Object Detector incorporates semantic compatibility between object queries and target categories into training-time matching, allowing supervision assignment to account for both object semantics and spatial location. The Semantically Conditioned Mask Decoder injects the same queries into mask decoding, enabling object semantics and local visual evidence to jointly inform mask prediction. Experiments demonstrate strong performance in open-ended object detection and segmentation, including a 5.0 AP gain over the prior method on LVIS minival.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
open-ended,detection,segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_Z7X8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

UniOSE addresses open-ended object detection and segmentation without a user-supplied category vocabulary. It combines detector queries and regional detector features with features from a pretrained vision–language encoder to construct semantics-enriched object queries. These queries condition category-name generation, provide semantic costs for training-time detection matching, and condition a SAM-family mask decoder. Validation includes LVIS minival detection, five cross-dataset semantic segmentation settings, component ablations, and diagnostic analyses. The method reports improvements of 5.0 and 3.6 detection AP over the corresponding Open-Det configurations, although several comparisons and diagnostics do not fully isolate the proposed mechanisms.

Critical strengths
  1. A coherent integration of naming and spatial prediction. Reusing a pre-generation object representation for naming, supervision assignment, and mask conditioning provides a concrete extension of detector-based open-ended recognition. The design preserves object correspondence and avoids requiring generated names to be re-encoded before segmentation.

  2. Promising empirical detection performance. The reported gains over Open-Det are substantial, and the VLM-branch ablation shows a large improvement within the tested configuration. Strong results on several segmentation settings further suggest that the resulting object representations transfer beyond detection. These findings make the approach worth considering despite the validation concerns.

Critical weaknesses
  1. The principal comparisons do not sufficiently isolate the contribution. The Open-Det comparison changes the backbone and introduces a separate VLM image encoder, while confidence scoring has a substantial effect on reported AP. Appendix B.2 shows that changing the fusion weight from 0.5 to 0.2 improves UniOSE by 4.23/5.39 AP, comparable to or greater than the headline gains. The weight is selected on the same LVIS minival set used for reporting, and equivalent scoring treatment of the baselines is not established. Consequently, the results demonstrate promising system performance but do not clearly attribute the gains to the proposed mechanisms.

  2. The central diagnostic evidence does not cleanly substantiate the claimed semantic benefits. Figure 4(a) compares absolute similarities in different native embedding spaces and measures positive-category alignment without assessing separation from incorrect categories. Figure 4(b) changes both the training matcher and the offline evaluation matcher, conflating prediction improvements with assignment effects. Meanwhile, semantic matching contributes only 0.8 AP with the VLM branch, and mask conditioning improves mIoU by 0.3/0.4/1.1 points without reported uncertainty. These limitations leave the central claim—that enriched naming semantics reliably improve spatial prediction—only partially supported.

Questions for authors

The following are clarification requests and requests for evidence relevant to the current claims, rather than suggestions for future work.

  1. Was the confidence-fusion weight selected using an independent validation set? Were the Open-Det baselines evaluated with the same category mapper, suppression settings, and equivalent fusion-weight tuning? Please provide comparisons under a common scoring protocol.

  2. What performance does an Open-Det-style baseline achieve with the same SAM2 backbone, EVA02 text encoder, training schedule, and evaluation settings? Does the gain persist when these factors are held constant?

  3. Does the Joint@τ improvement persist when both checkpoints are evaluated using the same semantic-independent assignment rule? Can the alignment analysis report category retrieval accuracy or positive-versus-negative margins rather than comparing absolute similarities across embedding spaces?

  4. Were bbox-only, random-query, and empty-query variants independently trained with equivalent adaptation and LoRA settings, or were queries replaced only at inference? Are the small improvements over bbox-only consistent across repeated runs?

Reviewer_SNA3OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

UniOSE builds on Open-Det, an open-ended detector that finds objects and generates their names with no category list. It adds features from a separate frozen CLIP model (EVA02-CLIP) to each object's query, giving a "semantic query" z_i. That same z_i is then used three ways: to generate the name, to decide which prediction is matched to which ground-truth object during training, and to condition a SAM2 mask decoder.

Critical strengths

The headline result is +5.0 AP over Open-Det on LVIS minival.

Critical weaknesses
  1. The paper motivates semantic conditioning by claiming that category semantics help distinguish adjacent, occluded, or visually similar objects (Sec. 1, Sec. 3.3), but no experiment isolates these cases. Moreover, for adjacent instances of the same category, the semantic cost and the semantic query carry the same category information, so the claim can only apply to neighbors of different categories.

  2. Limited novelty relative to prior work fusing SAM with vision–language models. Both SAM-related components follow established designs that the paper does not position itself against.

    • Semantics-Enriched Object Namer. The motivating observation, that SAM-family features carry insufficient category semantics, is expected given SAM's class-agnostic, label-free pretraining. It has already motivated prior work fusing SAM with CLIP for recognition (e.g., SAM-CLIP; Open-Vocabulary SAM, ECCV 2024; RegionSpot). Since UniOSE adopts a SAM2 backbone and adds EVA02-CLIP region features, the paper should position its Semantics-Enriched Object Namer relative to these works and clarify what is new beyond applying this known remedy to open-ended detection.
    • Semantically Conditioned Mask Decoder. The decoder combines detector box prompts with a semantic embedding injected into a SAM-family decoder. Two established designs are close to this:
      • prompting SAM with boxes from an open-set detector (e.g., Grounded-SAM);
      • conditioning SAM/SAM2 decoding on semantic or language embeddings through lightweight adapters (e.g., LISA, Open-Vocabulary SAM, and OpenWorldSAM, NeurIPS 2025). The paper distinguishes its conditioning signal from "a language-model state or a re-encoded category name" (Sec. 2), but it cites no method that conditions on re-encoded category names and does not compare against this alternative.
  3. Fig. 1(c) shows that Open-Det with a fine-tuned SAM2-T backbone localizes comparably to Swin-T but names objects worse. However, the LVIS AP of this same-backbone baseline is not reported. Since UniOSE uses SAM2-T plus an additional frozen EVA02-CLIP encoder, the headline +5.0 AP over Swin-T Open-Det conflates the backbone change, the added VLM, and the proposed components.

Questions for authors
  1. The paper emphasizes efficiency (fewer training epochs; a 248M naming model vs. an 8B one) but reports no inference cost. At inference, every dynamically selected candidate (300–900 under the inherited Open-Det protocol) undergoes autoregressive name generation before suppression. UniOSE additionally runs a full EVA02-CLIP encoder pass, three region-reading branches per candidate, and SAM2 mask decoding. Could the authors report end-to-end latency/FPS and peak memory?
Reviewer_AfuvOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes UniOSE, which connects open-ended category generation and spatial prediction through semantics-enriched object queries by combining detector features with region-level semantics from a pretrained VLM. It further incorporates these queries into training-time detection matching and mask prediction, enabling the same object representation to support category generation, detection, and segmentation.

Critical strengths

The idea of enriching object queries with region-level semantic information and leveraging the enriched representations for both category naming and spatial prediction seems reasonable.

Critical weaknesses
  1. In Table 1, UniOSE uses a SAM2 backbone and introduces additional components, while Open-Det uses a Swin backbone. This raises concerns about the fairness of the comparison. Could the authors provide a more controlled comparison using the same backbone to better isolate the contributions of the proposed method?
  2. UniOSE introduces additional components, which may increase computational overhead. Could the authors report the inference latency, GPU memory consumption, or computational cost (e.g., FLOPs) of the complete framework?
Questions for authors

Please see the weaknesses above.

Reviewer_saHGOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper studies open-ended object detection and segmentation, where a model must find objects and generate their category names without a category list at inference. The authors argue that object representations from existing detectors carry too little category semantics, which limits naming accuracy, and that richer object semantics could also help localization and mask prediction. They propose UniOSE, which builds on Open-Det [1] and uses SAM 2 [2] as the shared image encoder.

The method has three parts. The Semantics-Enriched Object Namer uses deformable region readers to read three feature sources around each candidate box: the detector's encoder memory, the detection features, and image features from a frozen EVA02-CLIP [3] encoder. Separate MLPs project the object query and the three region-enriched queries into a common space, and a linear layer fuses them into a semantic query z_i. A FLAN-T5-Base [8] name generator takes this query as a single visual token and generates the category name. The Semantics-Guided Object Detector keeps the Transformer detector inherited from Open-Det. During training, it adds the cosine distance between z_i and the VLM text embeddings of the ground-truth names to the Hungarian matching cost. This semantic cost is calibrated to the scale of the original semantic matching cost and is applied in all six decoder layers. The Semantically Conditioned Mask Decoder maps z_i through a small semantic adapter and adds it to the first mask token of a pretrained SAM 2 mask decoder, which also receives the predicted box as a prompt.

Training has two stages. The first stage trains detection and naming on Visual Genome while the shared image encoder and the VLM image and text encoders stay frozen. The second stage freezes this model and trains the semantic adapter, the copied mask and quality readouts, and low-rank decoder updates on 50K COCO Panoptic images.

On LVIS minival, the paper reports 32.4 AP with SAM2-T and 36.7 AP with SAM2-L, against 27.4 and 33.1 for Open-Det with Swin-T and Swin-L backbones [4]. For zero-shot semantic segmentation, it reports mIoU on A-847, PC-459, A-150, PC-59 and VOC-20 and compares against vocabulary-free baselines. Ablations study the VLM branch, semantic-query matching, and the kind of query used to condition the mask decoder. An analysis section reports alignment scores between semantic queries and ground-truth names, and a joint accuracy that requires both correct naming and correct localization.

[1] Open-Det: An Efficient Learning Framework for Open-Ended Detection.

[2] SAM 2: Segment Anything in Images and Videos.

[3] EVA-CLIP: Improved Training Techniques for CLIP at Scale.

[4] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.

[8] Scaling Instruction-Finetuned Language Models.

Critical strengths
  1. The VLM branch is simple and the ablation backs it clearly. In Table 3 it takes the SAM2-T model from 16.4 to 28.4 AP at 5 epochs, a gap far beyond what tuning or seed noise could explain.
  2. On the two ADE20K settings, the margins over vocabulary-free baselines are large: 31.8 against 18.8 on A-150, and 11.1 against 6.7 on A-847. They hold even against CRTNet [7], which reports 23.6 and 7.9. The model also returns boxes, names and masks tied to the same object, which the segmentation-only baselines do not.

[7] CLIP-Adapted Region-to-Text Learning for Generative Open-Vocabulary Semantic Segmentation.

Critical weaknesses
  1. The headline detection gain over Swin-T Open-Det [1] depends on a scoring weight tuned on the test set. Appendix B.2 sweeps the confidence-fusion weight on the full LVIS minival set, and the best points of that sweep are exactly the numbers reported in Table 1. At equal weighting, the Tiny model would be only slightly above Swin-T Open-Det, and the Large model would fall below Swin-L Open-Det. LVIS minival is also the only detection benchmark, while Open-Det and GenerateU [9] also report COCO and Objects365, so this choice is never checked on held-out data. At the Large scale, rare-category AP is lower than both Swin-L Open-Det and VL-SAM-V2 [10], and the paper does not discuss it.
  2. Table 1 does not separate the method from the backbone change and the extra encoder. UniOSE replaces the Swin backbone [4] with SAM 2 [2] and adds a full EVA02-CLIP [3] image encoder at inference, and Table 3 shows that the VLM branch accounts for most of the gain. Without a Swin-T Open-Det control that also has the VLM branch, the improvement cannot be credited to the semantic query. Figure 1 is also thin support for the motivation. It compares one checkpoint per backbone, which only shows that SAM 2 features carry less category information than Swin features in this setting. That is a much narrower point than the general finding stated in the abstract, which says that insufficient category semantics in detector representations limit naming accuracy.
  3. The novelty over Open-Det is small, and Section 3.1 does not say so. Open-Det's Vision-to-Language Distillation Module already reads backbone and encoder features around each query with deformable cross-attention, projects them with MLPs, and fuses them with a linear layer into the prompt for the language model. The Semantics-Enriched Object Namer is that module plus a VLM-feature branch. Semantic-query matching then swaps the decoder query in Open-Det's semantic matching cost for this semantic query. Projecting a VLM embedding into the SAM 2 mask decoder was already done in OpenWorldSAM [5], and DetCLIPv3 [6] already attached an object captioner to detector queries and showed that it helps detection. Neither work is cited.
  4. The segmentation claims go beyond the evidence. CRTNet [7] is a vocabulary-free method that uses the same FLAN-T5-Base [8] name generator, and it reports higher mIoU than UniOSE on PC-459, PC-59 and VOC-20, though it trains on more caption data. It is not cited, and the claim of leading results on PC-459 and PC-59 does not hold. The introduction also says that UniOSE outperforms existing methods, while Table 2 itself places it behind AutoSeg [11] and Chicken-and-egg [12] on VOC-20. The semantic conditioning of the mask decoder adds only a small mIoU gain in Table 4. There is also no direct mask metric such as LVIS mask AP, which VL-SAM [13] reports, even though the title promises open-ended segmentation.
  5. The diagnostics in Section 4.4 do not show what they claim. The alignment panel of Figure 4 compares cosine scores in two different text spaces, OpenAI CLIP [14] for Open-Det and EVA02 for UniOSE, and the appendix states that no cross-space calibration is applied. It also barely separates Base Query from the full model, even though the two differ by a large AP margin. The matching panel of Figure 4 scores each model after a query–GT assignment that uses that model's own matching cost. The semantic-query model therefore gets the queries whose semantic query is closest to the GT name, and the semantic query is exactly what the name generator reads, so part of the gain comes from the metric itself. The direct test is Table 3, where semantic-query matching gives a small AP gain in a single run and lowers rare-category AP, both with and without the VLM branch.

[1] Open-Det: An Efficient Learning Framework for Open-Ended Detection.

[2] SAM 2: Segment Anything in Images and Videos.

[3] EVA-CLIP: Improved Training Techniques for CLIP at Scale.

[4] Swin Transformer: Hierarchical Vision Transformer using Shifted Windows.

[5] OpenWorldSAM: Extending SAM2 for Universal Image Segmentation with Language Prompts.

[6] DetCLIPv3: Towards Versatile Generative Open-vocabulary Object Detection.

[7] CLIP-Adapted Region-to-Text Learning for Generative Open-Vocabulary Semantic Segmentation.

[8] Scaling Instruction-Finetuned Language Models.

[9] Generative Region-Language Pretraining for Open-Ended Object Detection.

[10] VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion.

[11] Auto-Vocabulary Semantic Segmentation.

[12] From Open-Vocabulary to Vocabulary-Free Semantic Segmentation.

[13] Training-Free Open-Ended Object Detection and Segmentation via Attention as Prompts.

[14] Learning Transferable Visual Models From Natural Language Supervision.

Questions for authors
  1. Which text encoder maps generated names to benchmark labels in Table 1, in the name-to-category similarity score of Appendix B.2, and in Table 2? The Open-Det [1] and GenerateU [9] rows match their papers, which map with CLIP [14]. If UniOSE maps with the EVA02 text encoder [3] that the semantic query is trained to align with, the protocols differ. The Table 2 baselines also use different category mappers, namely SBERT [15], LAVE [11] and manual mapping, and AutoSeg's own ablation shows that the mapper alone moves VOC mIoU from 73.9 to 87.1. How are per-object names and masks merged into the semantic segmentation map?
  2. Which EVA02-CLIP model and input size produce the VLM image features? The FLAN-T5 name generator [8] runs on all 300 to 900 dynamically selected queries before candidate suppression. Could you report total parameters, FLOPs and per-image latency against Open-Det and VL-SAM-V2 [10]? Table 2 counts BLIP [16] and CLIP with their visual encoders, but counts FLAN-T5 without EVA02.
  3. At inference you keep the mask with the highest predicted quality among the K outputs, but only the first mask token receives the semantic query and the copied mask and quality readouts. How often is that token picked? Were the bbox-only, Random query and Empty query variants in Table 4 trained separately with the same readouts and LoRA [17], or are they input swaps on the trained model? The 14-point drop of Empty query below bbox-only on VOC-20 suggests the latter.
  4. The SAM2-T baseline [2] in Figure 1 is fully fine-tuned, but UniOSE freezes the shared image encoder. What AP does the model without the VLM branch reach with a fine-tuned SAM2-T? If that closes much of the gap in Table 3, the VLM branch is mostly making up for the frozen encoder.

[1] Open-Det: An Efficient Learning Framework for Open-Ended Detection.

[2] SAM 2: Segment Anything in Images and Videos.

[3] EVA-CLIP: Improved Training Techniques for CLIP at Scale.

[8] Scaling Instruction-Finetuned Language Models.

[9] Generative Region-Language Pretraining for Open-Ended Object Detection.

[10] VL-SAM-V2: Open-World Object Detection with General and Specific Query Fusion.

[11] Auto-Vocabulary Semantic Segmentation.

[14] Learning Transferable Visual Models From Natural Language Supervision.

[15] Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks.

[16] BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation.

[17] LoRA: Low-Rank Adaptation of Large Language Models.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.