acceptodds
Under review as a conference paper at ICLR 2027

OpusDet: A Simple yet Effective Unified Framework for Open-Vocabulary Detection

Abstract

Recent unified open-vocabulary detection (OVD) methods support text queries, visual exemplars, and their combinations. However, they often rely on increasingly complex designs such as heavy cross-modal fusion, staged training, and iterative annotation pipelines. We revisit whether such complexity is necessary with stronger foundation models available. We find that unified OVD can be made much simpler with strong visual features and scalable grounding supervision. We present OpusDet (Open-vocabulary, Prompt-Unified, Simple), a unified detector supporting text, interactive visual, generic visual, and mixed prompting. OpusDet keeps the model, training, and data pipeline simple. It uses one detector for all prompt types, one-stage text–visual training, and a SAM3-based single-pass data engine. From the matched Base to the Full data setting, with the same Swin-T backbone, OpusDet gains 6.5 Visual-I AP on LVIS-minival, compared with 3.1 AP for our T-Rex2-style re-implementation. Our final DINOv3-ConvNeXt-B model achieves state-of-the-art Visual-I performance among comparable unified detectors, reaching 68.1, 69.2, and 54.7 AP on COCO, LVIS-minival, and ODinW35. It also maintains competitive Text and Visual-G performance. It further turns mixed prompting from interference into complementarity, improving over text or visual prompts alone. These results show that a simple unified detector can provide strong performance across prompting modes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary Object Detection, Visual Prompting

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jSb5OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents OpusDet, a unified open-vocabulary detection framework that supports text, visual, and mixed prompting without heavy fusion, staged training, or iterative relabeling. It uses a single detector with one-stage text–visual training and a SAM3-based single-pass data engine. The authors report state-of-the-art Visual-I performance on COCO, LVIS-minival, and ODinW35 while maintaining competitive Text and Visual-G accuracy. They also demonstrate mixed-prompt complementarity and analyze how text and visual prompt contributions change as grounding supervision expands.

Critical strengths

S.1:Leading Performance. The authors propose OpusDet, which achieves leading detection performance. It attains state-of-the-art Visual-I results on COCO, LVIS-minival, and ODinW35 while maintaining competitive Text and Visual-G accuracy.

S.2:Comprehensive and Credible Comparisons. The authors conduct comprehensive comparisons with many existing methods on widely used benchmark datasets, which strengthens the credibility of the reported results.

S.3:Extensive and Detailed Ablations. The authors perform extensive and detailed ablation studies, providing insights into how OpusDet achieves its final performance, including analyses of mixed-prompt complementarity and how the contributions of text and visual prompts change as grounding supervision expands.

Critical weaknesses

W.1: Unclear Contributions. In the abstract (Lines 012–014: “However, they often rely on increasingly complex designs such as heavy cross-modal fusion, staged training, and iterative annotation pipeline”) and the introduction (Line 037: “This flexibility often comes at the cost of a more complex system”), the authors claim to investigate whether such complexity is necessary (see abstract Lines 014–014: “We revisit whether such complexity is necessary with stronger foundation models available” and introduction Lines 043–046: “This motivates us to ... without heavy fusion, staged training, or iterative relabeling”). However, OpusDet combines the DINOv3 backbone, the RT-DETR Hybrid Encoder, the T-Rex2 Visual Prompt Encoder, and the Grounding DINO Prompt-Aware Decoder. I do not see targeted architectural improvements in the model design. In addition, OpusDet also employs fusion operations (the Grounding DINO decoder includes Image-Prompt and Prompt-Image fusion), and its One-stage Text-Visual Training has also been adopted in DETR-ViP. A clearer articulation of what unique designs OpusDet adopts to avoid building a complex system would better highlight the contribution the authors claim to make toward their stated motivation.

W.2: Insufficient Mechanistic Analysis of ICA. The authors propose Instance-level Contrastive Alignment (ICA) to align content prompts and decoder queries at the instance level, and demonstrate its significant gains in the ablation study (Table 4). However, to my understanding, LclsL_{\text{cls}} has already aligned prompts and queries to some extent, because prompt-based detection scores are computed by the similarity between prompts and queries. LclsL_{\text{cls}} optimizes the scores toward 0 (background and mismatched prompts/queries) and 1 (matched prompts/queries), thereby achieving alignment. Therefore, it is necessary to investigate how ICA differs from LclsL_{\text{cls}} in its effect. (a) In addition to the quantitative analysis in Table 4, visualizing the prompt distribution before and after applying ICA (similar to Figure 5 in T-Rex2 and Figure 4 in DETR-ViP) may help understand the mechanism of ICA. (b) The proposed ICA appears mechanistically similar to the Visual-Textual Prompt Relation Distillation Loss proposed in DETR-ViP; comparing and analyzing its effect with ICA would more strongly demonstrate the effectiveness of ICA.

W.3: Lack of Intuitive Comparison under a Fairer Setting. The authors use a data engine to expand the training data. Therefore, excluding the influence of dataset scale in comparisons would better highlight the role of architectural and optimization improvements. As far as I know, YOLOE and DETR-ViP are both trained on the OG dataset, corresponding to the Base-level training set in Table 6. Copying the results of OpusDet trained on the OG dataset from Table 6 into Table 1, or adding comparisons with YOLOE and DETR-ViP in Table 6, would more fairly reflect contributions beyond the dataset.

Questions for authors
  1. What is the difference between Pm,convP_{m,\text{con}}^v in Line 151 and Pcon,ivP_{\text{con},i}^v in Lines 196–198, and why is the order of the subscripts swapped?

  2. In Eq. (7), QiQ_i, Pcon,ivP_{\text{con},i}^v, and Pcon,jvP_{\text{con},j}^v are all obtained through learnable modules. Does ICA apply gradient truncation (stop-gradient) to any of them?

  3. Lines 206–207 state: “In practice we use the symmetric from that also contrasts content prompts against queries and averages the two directions.” In implementation, does this simply mean swapping the positions of QQ and PP in the equation? Unlike each query corresponding to only one category prompt, each prompt may correspond to multiple queries, which would result in multiple positive pairs in the denominator of Eq. (7). Would this affect optimization? Specifically, ICA would optimize the numerator to be as large as possible (increasing the similarity between QQ and PP in the numerator) and the denominator to be as small as possible (reducing the similarity between QQ and PP in the denominator).

1 comment

We thank Reviewer jSb5 for recognizing the Visual-I results and the breadth of our comparisons and ablations. We address the design rationale, ICA, and the comparison on Objects365 and GoldG below.

W1. Overall Contribution and Design Rationale

Thank you for asking us to clarify OpusDet’s contribution. We develop a unified detector around three choices: a lightweight encoder with prompt-aware decoding, one-stage text–visual training with instance-level supervision, and a single-pass annotation pipeline. Before data expansion, OpusDet already improves LVIS-minival Visual-I from 56.3 to 64.1 AP over our T-Rex2-style reimplementation with the same Swin-T backbone, Base data, and training schedule.

Architecture. We adopt the RT-DETR hybrid encoder and a GDINO–style decoder. we do not employ Image–Prompt or Prompt–Image fusion. Image features are computed independently of the prompt and can be reused across interactions. In the shared decoder, object queries attend to image and prompt features. At fixed backbone and resolution, replacing the encoder reduces computation from 208 to 164 GFLOPs. On this lighter encoder, prompt-aware decoding improves Visual-I by 5.5 AP and reduces Visual-G by 2.0 AP (Tables 4–5). These ablations quantify the efficiency and accuracy trade-off behind our architectural choices.

Training. One-stage text–visual training has precedent in DETR-ViP. We use eight text batches per visual batch and apply ICA on visual batches. ICA pairs each exemplar’s content embedding with the detection query matched to the same object. Table 11 shows the effect of the batch ratio on different prompt modes.

Data. We use source-specific prompts and a shared SAM3 pipeline to convert captioned images, classification data, and segmentation data into box–phrase supervision. This produces grounding annotations for 4.38M images in one pass using a pretrained model.

Together, these results show that this configuration is sufficient for strong interactive detection in a single model that also supports text, generic visual, and mixed prompts.

W2 & Q1–Q3. What ICA Learns and How It Is Implemented

The detection classification loss trains a query to respond to its category prompt. ICA trains an exemplar’s content embedding to match the query assigned to the same object. For two people in one image, both detection queries should respond to the person prompt. In ICA, each exemplar has its own matched query as the positive, and the other person’s query is a negative. DETR-ViP’s relation distillation uses similarities between text categories to supervise the relationships among visual category embeddings.

In Eq. (2), Pm,convP^v_{m,\mathrm{con}} collects the exemplar content embeddings of category mm. In Eq. (7), Pcon,ivP^v_{\mathrm{con},i} is one such embedding indexed by instance in the batch. Line 185 refers to the class token Pm,clsvP^v_{m,\mathrm{cls}}. Gradients flow through both the query and content embeddings. We apply cross-entropy to the rows and columns of the same similarity matrix and average the two losses. One-to-one Hungarian matching gives each row and column one diagonal positive. Off-diagonal entries remain in the denominator, including pairs of distinct instances of the same category.

Thank you for suggesting this visualization. It would help show how ICA affects category clustering. We have not run this comparison. The cited t-SNE plots examine category prompts across images. ICA matches each exemplar to its own detection query, so changes in category clustering would be indirect. Both settings use LalignL_{\mathrm{align}} for visual–text category alignment, which can be less reliable when examples are scarce. ICA adds a complementary instance-level signal, which may help explain the 3.9 AP gain on rare categories under Visual-G (Table 4).

W3. Comparison on Objects365 and GoldG

The LVIS-minival table compares models trained on Objects365 and GoldG. The two Base runs match backbone, data, and schedule. The external models use the same named data sources.

ModelVisual-IVisual-GText
T-Rex2-style reimplementation (Swin-T, Base)56.333.626.5
OpusDet (Swin-T, Base)64.134.433.9
DETR-ViP-T66.141.125.0
YOLOE-v8-L—34.235.9

DETR-ViP-T leads on both visual protocols, particularly Visual-G, where its global prompt integration during training more closely matches the evaluation setup. YOLOE-v8-L leads on Text. It trains a text detector first, then fits a visual adapter with the other modules frozen.

OpusDet targets balanced performance across prompt modes with a simple, unified detector. With Base data, it gains 7.8 Visual-I AP over our matched T-Rex2-style reimplementation. Its Text AP exceeds DETR-ViP-T’s, and it stays within 0.2 AP of YOLOE-v8-L on Visual-G and 2.0 AP on Text.

Reviewer_NDbpOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes OpusDet, a unified open-vocabulary detector supporting text, interactive visual, generic visual, and mixed prompting. It combines a DINOv3-pretrained backbone, a prompt-aware decoder, instance-level contrastive alignment (ICA), and a SAM3-based data engine. Experiments on COCO, LVIS, and ODinW demonstrate strong Visual-I performance and competitive results across multiple prompting modes.

Critical strengths

The paper presents a relatively simple and unified framework supporting multiple prompting modes. Its strongest contribution is the competitive Visual-I performance across several benchmarks, supported by extensive experiments and component-wise analyses.

Critical weaknesses
  1. Insufficient evidence for the effectiveness of the core methodological contributions. Table 4 shows that ICA improves Visual-I AP by only 0.2, whereas the SAM3-based data engine contributes 6.0 AP. This makes it difficult to distinguish the benefits of the proposed method from those of expanded training supervision.

  2. Inconsistent effectiveness across prompting modes. Despite strong Visual-I results, Visual-G performance remains relatively weak. The prompt-aware decoder even reduces Visual-G AP from 35.3 to 33.3 in Table 4, raising concerns about the claimed unified prompting capability.

Questions for authors
  1. Can the authors provide stronger evidence that the performance improvements originate from the proposed components rather than expanded training data?

  2. Why does the prompt-aware decoder degrade Visual-G performance, and how does the proposed framework address this inconsistency across prompting modes?

  3. How sensitive is ICA to false negatives from same-category instances, and would category-aware negative sampling improve its effectiveness?

1 comment

Thank you for asking us to separate component gains from data scaling and to explain the Visual-G trade-off.

W1 & Q1. Model and Training Gains at Fixed Data

Rows A–D of Table 4 all use Base data (Objects365 and GoldG). We include the additional B + ICA run so that the two decoder variants (B and C) can be compared with and without ICA at fixed backbone and data. All results are on LVIS-minival.

SettingICAVisual-I APVisual-I rare APVisual-G APVisual-G rare APText APMixed AP
A. T-Rex2-style baselineOff56.363.833.628.026.535.2
B. Backbone and encoderOff57.566.835.329.129.536.2
B + ICA (added)On58.467.634.629.330.334.6
C. Prompt-aware decoderOff63.073.933.326.132.736.5
D. C + ICAOn63.274.733.530.033.238.0

At fixed data, the changes from A to D improve Visual-I by 6.9 AP and Text by 6.7 AP. The backbone and encoder update in A→B contributes 1.2 Visual-I AP. The decoder change in B→C contributes another 5.5 Visual-I AP and 3.2 Text AP. The 0.2 Visual-I gain cited by the reviewer is the incremental effect of ICA in C→D. The added B + ICA run shows that the pattern depends on the decoder. ICA raises Visual-I by 0.9 AP and rare Visual-G by 0.2 AP with the original decoder, and by 0.2 and 3.9 AP with the prompt-aware decoder. Table 6 also evaluates the combined design at fixed Swin-T, Base data, and training schedule, where Visual-I improves from 56.3 to 64.1 AP.

The Base runs share a one-stage schedule that alternates text and visual batches. ICA adds instance-level supervision on visual batches. The data contribution is the pipeline itself, which uses SAM3 to convert captioned images, classification data, and segmentation data into box–phrase supervision for 4.38M images in one pass. The 6.0 Visual-I AP gain in D→E is observed when expanding both grounding data and the training budget.

W2 & Q2. Design Rationale and the Visual-G Trade-off

Our design keeps image encoding independent of the prompt so image features can be reused across interactions. All four modes share a decoder whose object queries attend to image and prompt features. At fixed backbone and resolution, the hybrid encoder reduces computation from 208 to 164 GFLOPs (Table 5). The prompt-aware decoder then improves Visual-I by 5.5 AP at a cost of 2.0 Visual-G AP (B→C).

Visual-prompt training and Visual-I inference use same-image exemplars. Visual-G uses cross-image category prototypes and can include categories absent from the target image. Both decoder variants face this shift. We hypothesize that the additional query–prompt attention helps the new decoder exploit same-image cues and makes it more sensitive to cross-image prompt distributions.

With ICA enabled, the overall Visual-G gap between the decoder variants narrows from 2.0 (B 35.3 vs. C 33.3) to 1.1 AP (B + ICA 34.6 vs. C + ICA 33.5), and the rare-category gap reverses from a 3.0 AP deficit (B 29.1 vs. C 26.1) to a 0.7 AP lead (B + ICA 29.3 vs. C + ICA 30.0). The training ratio can also shift performance across modes. Changing the visual-to-text ratio from 1:4 to 1:8 improves 2.0 Visual-G and 1.6 Text AP at a cost of 1.4 Visual-I AP (Table 11). With the expanded data and training schedule, the Full model reaches 38.6 Visual-G AP while retaining the same shared detector design.

Q3. Sensitivity to Same-Category Negatives

We have not measured the sensitivity to same-category negatives or the effect of masking them. In ICA, the matching unit is an object instance. Two boxes of the same class share a category prompt for detection, while each box has its own positive query–content pair in ICA. The unmatched same-category pairs compete as negatives. The category-level loss LalignL_{\mathrm{align}} supplies visual–text category supervision alongside ICA.

A category-aware version could mask same-category nonmatches in both directions, keeping the matched instance and different-category negatives. This would reduce competition between instances of the same class and could affect instance discrimination and category generalization differently. Such a comparison would isolate the effect of that choice of negatives.

Reviewer_SqkgOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

OpusDet is a unified framework for open-vocabulary object detection with text, interactive visual, generic visual, and mixed prompts. It combines a DINOv3 backbone, hybrid encoder, and prompt-aware decoder with one-stage text–visual training. A SAM3-based single-pass data engine converts multiple data sources into grounding annotations. Experiments show the strongest gains in interactive visual detection, including an increase in LVIS-minival AP from 59.4 to 70.6 over a T-Rex2-style re-implementation under the same Swin-T backbone and Full data setting.

Critical strengths
  • Meaningful Visual-I gains. Under the same Swin-T backbone and Full data setting, LVIS-minival AP improves from 59.4 to 70.6 over the T-Rex2-style re-implementation (Table 6).
  • Useful system measurements. Table 5 decomposes architecture–efficiency trade-offs, and Table 8 reports approximately 90% box accuracy from manual checks of generated annotations.
Critical weaknesses
  • The central simplification argument lacks sufficient matched validation

Showing that a simple pipeline works does not establish whether previous complex designs retain value with the same strong backbone, data, and training budget. Table 5 provides useful local controls: A→B1 upgrades the baseline backbone, and B2→B3 isolates the encoder replacement. However, the final Full setting lacks a matched strong-backbone baseline retaining the previous design. Appendix E.3 explicitly acknowledges that its staged-training comparison is not controlled, and single-pass versus iterative annotation is not quantitatively compared. These results leave the motivating question—whether stronger foundation models make the previous complexities unnecessary—insufficiently answered.

  • Data-scaling gains are confounded by increased training

Base uses 64 × 120K image inputs, whereas Full uses 128 × 400K, increasing exposure from 7.68M to 51.2M (p.5). Thus, the +6.5 versus +3.1 Visual-I AP comparison measures responses to jointly scaling data and optimization. Without a fixed-budget or Base-long-training control, it does not justify “proving” better utilization of expanded grounding supervision (p.8, lines 415–417).

  • Mixed-prompt claims overstate the evidence

The Table 4 baseline already exhibits complementarity: Mixed 35.2 exceeds Text 26.5 and Visual-G 33.6. Official T-Rex2 also shows complementarity on full LVIS (Table 3). The COCO comparison therefore does not establish that OpusDet's changes eliminate interference under matched conditions. Moreover, ΔG−ΔT = Text−Visual-G, so the relative gain changes do not independently demonstrate that text stabilizes noisy visual prototypes. Table 10 offers indirect reference-count evidence; the proposed mechanism remains a hypothesis.

Questions for authors
  • Grammatical errors.

Examples include “We uses ... and alternates ...” (p.5, lines 250–251) and “a larger Visual-I gains” (p.8, lines 415–416).

  • Why are the performance advantages concentrated in Visual-I?

In Table 1, Text AP trails CP-DETR-T on COCO/LVIS-minival (49.6/43.0 versus 52.0/47.6), and Visual-G trails DETR-ViP-T on LVIS-minival/ODinW35 (38.6/25.9 versus 41.1/31.2). The paper already discusses the Visual-I training versus Visual-G inference mismatch and the partial recovery from ICA, and Table 11 shows trade-offs across prompting modes as the training ratio changes. While the SOTA claim is appropriately limited to Visual-I, to what extent do these factors explain the remaining gaps in the final Full setting?

1 comment

Thank you for recognizing the matched Visual-I gains and for asking us to clarify what our comparisons establish.

W1. Design Rationale and Contribution

Our goal is to build a strong unified detector with a lightweight image encoder, one-stage text–visual training, and single-pass grounding annotation. We chose T-Rex2 as a reference because its late-fusion design is relatively simple and already supports all four prompt modes. With Swin-T, data, and training schedule matched within each setting, OpusDet improves LVIS-minival Visual-I over our T-Rex2-style reimplementation by 7.8 AP on Base and 11.2 AP on Full.

The component results explain the trade-offs. With DINOv3 and input resolution fixed on Base, the hybrid encoder reduces computation from 208 to 164 GFLOPs, with a 1.6 AP decrease in Visual-I and a 1.9 AP increase in Visual-G (Table 5). Our single-pass pipeline generates grounding supervision from three data sources with approximately 90% box accuracy in the manual checks (Table 8).

Measuring the added value of earlier designs under DINOv3 and Full data would require matched controls for fusion, staged training, and iterative annotation. Our results show what this simpler configuration already achieves and establish a baseline for those comparisons.

W2. Matched Comparisons and Training Exposure

Table 6 compares LVIS-minival Visual-I AP within and across data settings. Within each data setting, all three models use the same training data, global batch size, and number of training steps. The first two rows also share the Swin-T backbone.

ModelBackboneBaseFullGain
T-Rex2-style reimplementationSwin-T56.359.4+3.1
OpusDetSwin-T64.170.6+6.5
OpusDetDINOv3-ConvNeXt-B63.269.2+6.0

In the matched Swin-T comparison, OpusDet leads by 7.8 AP on Base and 11.2 AP on Full. The advantage is already present before data expansion.

As the reviewer notes, total image inputs increase from 7.68M (64 × 120K) on Base to 51.2M (128 × 400K) on Full. The +6.5 versus +3.1 AP gains quantify how the two Swin-T models respond to the same joint scaling of data and training budget. The training loss curves for both models approach a plateau toward the end of their respective Base and Full runs. We will include these loss curves and the corresponding training details in the revision.

W3. Complementarity and Interpretation of Mixed Gains

We agree that complementarity is already present in the Table 4 baseline (Mixed 35.2, Text 26.5, Visual-G 33.6) and in the published T-Rex2 results on full LVIS. On COCO, T-Rex2 [1] reports a Mixed AP of 42.4, below its Text AP of 45.8. Its limitations section states: “Visual prompts may sometimes interfere with text prompts, especially in scenarios involving common objects, as indicated by the reduced performance on the COCO benchmark when both are used together in Tab. 8.” At the default 16 references, OpusDet's Mixed AP exceeds its own stronger single-prompt score on all three reported benchmarks, by 0.3 on COCO, 2.2 on LVIS-minival, and 2.8 on full LVIS (Table 3).

In Table 4, the gain of Mixed over Text (ΔT\Delta_T) decreases from 8.7 to 2.2, while the gain over Visual-G (ΔG\Delta_G) rises from 1.6 to 6.6. We agree that ΔG−ΔT=(Mixed−Visual-G)−(Mixed−Text)=Text−Visual-G\Delta_G-\Delta_T=(\mathrm{Mixed}-\mathrm{Visual\text{-}G})-(\mathrm{Mixed}-\mathrm{Text})=\mathrm{Text}-\mathrm{Visual\text{-}G}, but this identity does not remove either trend. The two trends still describe a shift in which single-prompt baseline benefits more from mixing.

Q2. Why the Advantages Are Concentrated in Visual-I

Visual-prompt training uses same-image exemplars, directly matching Visual-I inference. Visual-G uses cross-image prototypes and its prompt set can include categories absent from the target image. In Table 1, OpusDet trails CP-DETR-T on Text on COCO and LVIS-minival (49.6 / 43.0 vs. 52.0 / 47.6). Its Visual-G scores trail DETR-ViP-T on LVIS-minival and ODinW35 (38.6 / 25.9 vs. 41.1 / 31.2), while the COCO scores are 43.4 and 43.2.

The Base ablations establish the direction of the effects within OpusDet. ICA recovers 3.9 rare-category Visual-G AP, with a smaller overall gain of 0.2 AP. Changing the visual-to-text ratio from 1:4 to 1:8 improves Visual-G and Text by 2.0 and 1.6 AP at a cost of 1.4 Visual-I AP. DETR-ViP also uses cross-image prompt integration, relation distillation, and selective fusion, which are designed to improve visual prompting. The share of the final Full gaps attributable to each of these factors is unmeasured.

Q1. Wording

Thank you for catching these errors. The correct phrases are “We use AdamW and alternate visual and text batches at a ratio of 1:8” and “a larger Visual-I gain.” We will proofread for the same errors.

Reference

[1] Jiang et al. T-Rex2: Towards Generic Object Detection via Text-Visual Prompt Synergy. ECCV 2024.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.