- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper presents OpusDet, a unified open-vocabulary detection framework that supports text, visual, and mixed prompting without heavy fusion, staged training, or iterative relabeling. It uses a single detector with one-stage text–visual training and a SAM3-based single-pass data engine. The authors report state-of-the-art Visual-I performance on COCO, LVIS-minival, and ODinW35 while maintaining competitive Text and Visual-G accuracy. They also demonstrate mixed-prompt complementarity and analyze how text and visual prompt contributions change as grounding supervision expands.
Critical strengths
S.1:Leading Performance. The authors propose OpusDet, which achieves leading detection performance. It attains state-of-the-art Visual-I results on COCO, LVIS-minival, and ODinW35 while maintaining competitive Text and Visual-G accuracy.
S.2:Comprehensive and Credible Comparisons. The authors conduct comprehensive comparisons with many existing methods on widely used benchmark datasets, which strengthens the credibility of the reported results.
S.3:Extensive and Detailed Ablations. The authors perform extensive and detailed ablation studies, providing insights into how OpusDet achieves its final performance, including analyses of mixed-prompt complementarity and how the contributions of text and visual prompts change as grounding supervision expands.
Critical weaknesses
W.1: Unclear Contributions. In the abstract (Lines 012–014: “However, they often rely on increasingly complex designs such as heavy cross-modal fusion, staged training, and iterative annotation pipeline”) and the introduction (Line 037: “This flexibility often comes at the cost of a more complex system”), the authors claim to investigate whether such complexity is necessary (see abstract Lines 014–014: “We revisit whether such complexity is necessary with stronger foundation models available” and introduction Lines 043–046: “This motivates us to ... without heavy fusion, staged training, or iterative relabeling”). However, OpusDet combines the DINOv3 backbone, the RT-DETR Hybrid Encoder, the T-Rex2 Visual Prompt Encoder, and the Grounding DINO Prompt-Aware Decoder. I do not see targeted architectural improvements in the model design. In addition, OpusDet also employs fusion operations (the Grounding DINO decoder includes Image-Prompt and Prompt-Image fusion), and its One-stage Text-Visual Training has also been adopted in DETR-ViP. A clearer articulation of what unique designs OpusDet adopts to avoid building a complex system would better highlight the contribution the authors claim to make toward their stated motivation.
W.2: Insufficient Mechanistic Analysis of ICA. The authors propose Instance-level Contrastive Alignment (ICA) to align content prompts and decoder queries at the instance level, and demonstrate its significant gains in the ablation study (Table 4). However, to my understanding, has already aligned prompts and queries to some extent, because prompt-based detection scores are computed by the similarity between prompts and queries. optimizes the scores toward 0 (background and mismatched prompts/queries) and 1 (matched prompts/queries), thereby achieving alignment. Therefore, it is necessary to investigate how ICA differs from in its effect. (a) In addition to the quantitative analysis in Table 4, visualizing the prompt distribution before and after applying ICA (similar to Figure 5 in T-Rex2 and Figure 4 in DETR-ViP) may help understand the mechanism of ICA. (b) The proposed ICA appears mechanistically similar to the Visual-Textual Prompt Relation Distillation Loss proposed in DETR-ViP; comparing and analyzing its effect with ICA would more strongly demonstrate the effectiveness of ICA.
W.3: Lack of Intuitive Comparison under a Fairer Setting. The authors use a data engine to expand the training data. Therefore, excluding the influence of dataset scale in comparisons would better highlight the role of architectural and optimization improvements. As far as I know, YOLOE and DETR-ViP are both trained on the OG dataset, corresponding to the Base-level training set in Table 6. Copying the results of OpusDet trained on the OG dataset from Table 6 into Table 1, or adding comparisons with YOLOE and DETR-ViP in Table 6, would more fairly reflect contributions beyond the dataset.
Questions for authors
-
What is the difference between in Line 151 and in Lines 196–198, and why is the order of the subscripts swapped?
-
In Eq. (7), , , and are all obtained through learnable modules. Does ICA apply gradient truncation (stop-gradient) to any of them?
-
Lines 206–207 state: “In practice we use the symmetric from that also contrasts content prompts against queries and averages the two directions.” In implementation, does this simply mean swapping the positions of and in the equation? Unlike each query corresponding to only one category prompt, each prompt may correspond to multiple queries, which would result in multiple positive pairs in the denominator of Eq. (7). Would this affect optimization? Specifically, ICA would optimize the numerator to be as large as possible (increasing the similarity between and in the numerator) and the denominator to be as small as possible (reducing the similarity between and in the denominator).
We thank Reviewer jSb5 for recognizing the Visual-I results and the breadth of our comparisons and ablations. We address the design rationale, ICA, and the comparison on Objects365 and GoldG below.
W1. Overall Contribution and Design Rationale
Thank you for asking us to clarify OpusDet’s contribution. We develop a unified detector around three choices: a lightweight encoder with prompt-aware decoding, one-stage text–visual training with instance-level supervision, and a single-pass annotation pipeline. Before data expansion, OpusDet already improves LVIS-minival Visual-I from 56.3 to 64.1 AP over our T-Rex2-style reimplementation with the same Swin-T backbone, Base data, and training schedule.
Architecture. We adopt the RT-DETR hybrid encoder and a GDINO–style decoder. we do not employ Image–Prompt or Prompt–Image fusion. Image features are computed independently of the prompt and can be reused across interactions. In the shared decoder, object queries attend to image and prompt features. At fixed backbone and resolution, replacing the encoder reduces computation from 208 to 164 GFLOPs. On this lighter encoder, prompt-aware decoding improves Visual-I by 5.5 AP and reduces Visual-G by 2.0 AP (Tables 4–5). These ablations quantify the efficiency and accuracy trade-off behind our architectural choices.
Training. One-stage text–visual training has precedent in DETR-ViP. We use eight text batches per visual batch and apply ICA on visual batches. ICA pairs each exemplar’s content embedding with the detection query matched to the same object. Table 11 shows the effect of the batch ratio on different prompt modes.
Data. We use source-specific prompts and a shared SAM3 pipeline to convert captioned images, classification data, and segmentation data into box–phrase supervision. This produces grounding annotations for 4.38M images in one pass using a pretrained model.
Together, these results show that this configuration is sufficient for strong interactive detection in a single model that also supports text, generic visual, and mixed prompts.
W2 & Q1–Q3. What ICA Learns and How It Is Implemented
The detection classification loss trains a query to respond to its category prompt. ICA trains an exemplar’s content embedding to match the query assigned to the same object. For two people in one image, both detection queries should respond to the person prompt. In ICA, each exemplar has its own matched query as the positive, and the other person’s query is a negative. DETR-ViP’s relation distillation uses similarities between text categories to supervise the relationships among visual category embeddings.
In Eq. (2), Pm,conv collects the exemplar content embeddings of category m. In Eq. (7), Pcon,iv is one such embedding indexed by instance in the batch. Line 185 refers to the class token Pm,clsv. Gradients flow through both the query and content embeddings. We apply cross-entropy to the rows and columns of the same similarity matrix and average the two losses. One-to-one Hungarian matching gives each row and column one diagonal positive. Off-diagonal entries remain in the denominator, including pairs of distinct instances of the same category.
Thank you for suggesting this visualization. It would help show how ICA affects category clustering. We have not run this comparison. The cited t-SNE plots examine category prompts across images. ICA matches each exemplar to its own detection query, so changes in category clustering would be indirect. Both settings use Lalign for visual–text category alignment, which can be less reliable when examples are scarce. ICA adds a complementary instance-level signal, which may help explain the 3.9 AP gain on rare categories under Visual-G (Table 4).
W3. Comparison on Objects365 and GoldG
The LVIS-minival table compares models trained on Objects365 and GoldG. The two Base runs match backbone, data, and schedule. The external models use the same named data sources.
DETR-ViP-T leads on both visual protocols, particularly Visual-G, where its global prompt integration during training more closely matches the evaluation setup. YOLOE-v8-L leads on Text. It trains a text detector first, then fits a visual adapter with the other modules frozen.
OpusDet targets balanced performance across prompt modes with a simple, unified detector. With Base data, it gains 7.8 Visual-I AP over our matched T-Rex2-style reimplementation. Its Text AP exceeds DETR-ViP-T’s, and it stays within 0.2 AP of YOLOE-v8-L on Visual-G and 2.0 AP on Text.