- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes RT-DETR-World, a real-time open-vocabulary object detection (OVD) framework that transfers rich language semantics into a compact detector without introducing additional inference-time overhead from the proposed training modules. The authors construct GroundingCapv2, a large-scale dataset containing 1.12M images and 8.05M regions with three levels of supervision: category names, object descriptions, and image descriptions. To leverage this supervision, the proposed Dual-Path Description Alignment (DDA) combines a lightweight MiniLM pathway with a training-only LLM2CLIP teacher, aligning object queries and global visual representations with instance-level and scene-level semantics. Relation-Aware Negative Relaxation (RNR) further modifies contrastive learning by reducing the repulsion between semantically related negative samples while preserving exact positive correspondences. The detector uses a frozen DINOv3 backbone and lightweight DETR-style architecture, while the auxiliary teacher components are removed after training. Experiments on LVIS, ODinW, and COCO-O demonstrate competitive zero-shot detection performance and strong cross-domain robustness. The largest variant achieves 40.0 Fixed AP on LVIS at 32 FPS, while the smaller variants provide faster inference with lower accuracy.
Critical strengths
-
The paper addresses a practically meaningful problem: transferring rich semantic knowledge from language models to compact open-vocabulary detectors without requiring expensive language-model inference during deployment. The separation between training-time semantic supervision and lightweight inference is well motivated. In particular, the combination of object-level and image-level descriptions provides a reasonable way to incorporate fine-grained semantic information and scene context into detector representations.
-
The experimental evaluation is relatively comprehensive and shows encouraging results across multiple benchmarks. The proposed method achieves strong performance on LVIS, ODinW, and COCO-O, demonstrating competitive large-vocabulary detection and cross-domain robustness. The ablation studies also provide useful evidence for the proposed components. On LVIS, the full description-alignment framework improves Fixed AP from 33.1 to 35.2 over the category-supervision configuration, while RNR improves performance from 33.6 to 35.2 compared with standard hard contrastive alignment. The improvements are particularly notable for rare categories, supporting the potential benefit of richer semantic supervision.
Critical weaknesses
-
The performance improvements are not sufficiently disentangled from the effects of stronger pretrained models and training data. RT-DETR-World employs DINOv3 backbones and GroundingCapv2, which provides substantially richer annotations than the training datasets used by several competing real-time OVD methods. Although Table 5 includes useful internal ablations, comparisons against YOLO-World, YOLOE, and OV-DEIM involve different visual backbones, training corpora, and model capacities. Therefore, the reported superiority cannot be attributed solely to DDA and RNR. For example, RT-DETR-World-S+ achieves 36.6 Fixed AP at 58 FPS, while OV-DEIM-L achieves 35.9 Fixed AP at 91 FPS, indicating that the accuracy improvement comes with a considerable efficiency trade-off. A more controlled comparison using matched backbones, training data, or computational budgets would substantially strengthen the empirical claims. Furthermore, the paper should report the computational cost of constructing GroundingCapv2 and precomputing LLM teacher features, since the absence of inference overhead does not imply that the overall approach is computationally inexpensive.
-
The methodological novelty and the claimed compositional generalization benefits are not fully established. Multi-level semantic distillation and similarity-aware contrastive learning have already been explored in prior work, including LLMDet, SoftCLIP, SRCL, and related knowledge-distillation-based OVD methods. DDA mainly combines existing alignment objectives at different semantic levels, while RNR modifies the contribution of semantically related negatives using teacher-derived similarities. Although the integration is reasonable, the conceptual distinction from these approaches appears relatively incremental. More importantly, the paper motivates rich descriptions through compositional generalization, arguing that attributes and object relationships learned from seen categories can facilitate recognition of unseen concepts. However, the evaluation primarily reports aggregate AP on existing detection benchmarks, without directly demonstrating that the model learns or transfers these compositional semantic factors. The authors should provide targeted evaluations on unseen attribute-object combinations, fine-grained category distinctions, or controlled compositional generalization benchmarks. Additionally, since GroundingCapv2 incorporates large-vocabulary sources such as V3Det, the overlap between training categories and evaluation vocabularies should be explicitly analyzed to clarify the extent of genuine unseen-category recognition.
Questions for authors
-
Can the authors provide a more controlled comparison that separates the contribution of DDA and RNR from the effects of DINOv3 pretraining and GroundingCapv2? In particular, how does a strong detector trained with the same visual backbone and dataset but without the proposed semantic-alignment mechanisms compare against the complete framework?
-
The paper emphasizes compositional zero-shot generalization through reusable semantic factors such as attributes, materials, and object parts. Can the authors provide quantitative evidence that the proposed semantic distillation specifically improves the recognition of unseen attribute-object combinations or other compositional concepts, rather than simply improving general-purpose detection representations?
-
How much category overlap exists between GroundingCapv2 and the LVIS/ODinW evaluation vocabularies? Would the authors be able to report results separately for categories that are absent from the training category vocabulary? This would help clarify the interpretation of the reported zero-shot generalization performance.
-
What is the total computational cost of the GroundingAgent-Opt annotation pipeline and offline LLM feature extraction? Additionally, has the quality of the automatically generated object descriptions and normalized category labels been quantitatively verified through a manually audited subset? These details would help assess the scalability and reliability of the proposed training framework.
4 comments
R2. Description supervision and compositional grounding
We added two referring expression grounding evaluations, PR-Bench and FineCops-Ref. RT-DETR-World-S and S+ rank in the top two on the overall metric of both benchmarks. PR-Bench tests whether models can use attributes, positions, and object relations to localize targets. S+ achieves 28.68 overall mAcc, 2.84 percentage points above the strongest baseline (Table 5). On FineCops-Ref, S achieves the highest overall top-1 accuracy at 50.60. Level 3 requires combining multiple cues to distinguish instances of the same category. S+ achieves 26.75 P@1 on this subset, outperforming the strongest baseline by 3.82 percentage points, the largest margin across the three difficulty levels (Table 6). In the paired image-order distractor evaluation, where the referring expression is held fixed, S achieves 33.88 R@1, 4.24 percentage points above the strongest baseline (Table 14).
| Model | PR-Bench Overall mAcc (%) ↑ | FineCops-Ref Overall P@1 (%) ↑ |
|---|---|---|
| YOLO-Worldv2.1-L | 24.93 | 46.17 |
| YOLOEv8-L | 25.02 | 48.79 |
| YOLOEv11-L | 25.10 | 47.81 |
| OV-DEIM-L | 25.84 | 49.11 |
| RT-DETR-World-S | 28.43 | 50.60 |
| RT-DETR-World-S+ | 28.68 | 50.29 |
The gains from description supervision also extend to attribute and relation grounding. B0 and B1 use the same detector architecture and GroundingCap-1M. With description-based teacher supervision, B1 raises overall PR-Bench mAcc from 25.33 to 28.40, with gains of 7.09 and 3.67 percentage points on attribute and relation mAcc, respectively. FineCops-Ref Level 3 P@1 also rises from 28.66 to 29.51 (Tables 11–12). On GroundingCapv2, C5 adds direct teacher alignment and outperforms C2 by 3.32 and 2.27 percentage points on attribute and relation mAcc, respectively (Table 13). Results on both datasets suggest that description-based teacher supervision helps the detector distinguish target instances using fine-grained semantic cues.
R3. Training vocabulary overlap and detection beyond the training vocabulary
Common-88 consists of 88 LVIS minival categories whose primary names match neither the GroundingCapv2 nor the OG training vocabulary. When official LVIS synonyms are included, 52 categories remain unmatched in both vocabularies, forming Common-52. For YOLO-Worldv2.1-L, the audit covers OG only. Its additional CC-LiteV2 training data remains unaudited.
| Model | Common-52 Fixed AP (%) ↑ | Common-88 Fixed AP (%) ↑ |
|---|---|---|
| YOLO-Worldv2.1-L | 29.28 | 28.82 |
| YOLOEv8-L | 29.19 | 29.10 |
| YOLOEv11-L | 26.92 | 28.12 |
| OV-DEIM-L | 30.31 | 30.70 |
| RT-DETR-World-S | 27.58 | 27.96 |
| RT-DETR-World-S+ | 30.02 | 30.74 |
| RT-DETR-World-B | 30.65 | 33.27 |
We use each model’s full-vocabulary predictions and average per-category Fixed AP over the same subsets. RT-DETR-World-B ranks first on both subsets, achieving 30.65 Fixed AP on Common-52 and 33.27 on Common-88, outperforming the strongest baseline, OV-DEIM-L, by 0.34 and 2.57 percentage points, respectively (Table 7).
R4. Offline computational cost and annotation reliability
The GroundingAgent-Opt annotation pipeline ran for about 5 days on two RTX PRO 6000 GPUs (96 GB each), totaling approximately 240 GPU-hours. Extracting LLM2CLIP features for all image-level and object-level descriptions across the five data sources takes approximately 16.6 hours on two RTX 3090 GPUs, equivalent to 33.2 GPU-hours. GroundingCapv2 reuses the image-level descriptions from GroundingCap-1M. The new annotations and cached features are generated once offline and can be reused across detector sizes and training runs.
Ten reviewers manually audited a sample of 50,000 images using the original images and annotated boxes, with GPT assisting the review. Before correction, the manual audit found accuracies of 97.3% for normalized category labels, 95.6% for object descriptions, and 93.2% for relation statements. We then corrected the annotation errors confirmed during the audit.
Thank you for the thorough rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.
The controlled ablations clarify the contributions of DDA and RNR, while the additional grounding evaluations, vocabulary-overlap analysis, and annotation details strengthen the empirical evidence.
My remaining concern is that, while the new evaluations demonstrate improved compositional grounding, generalization to unseen attribute–object combinations remains unverified. I encourage the authors to clarify this distinction and appropriately qualify their claims in the revised paper.
Overall, the rebuttal has satisfactorily addressed my main concerns. I will maintain my rating of 3.
Thank you for your positive assessment of our additional experiments and responses, and for your further suggestions on the wording of the paper.
In the revised manuscript, we will clearly distinguish compositional grounding from generalization to unseen attribute–object combinations. The additional referring expression grounding evaluations and description-supervision comparisons support the model’s improved ability to distinguish target instances using attributes, relations, and multiple semantic cues. The Common-52/88 results demonstrate its detection capability beyond the audited training category vocabularies. We will accordingly revise statements concerning “compositional zero-shot generalization” to align our claims with the capabilities verified by these experiments.
Thank you for this suggestion, which helps us present the benefits of our method and our experimental conclusions more precisely.
We thank the reviewer for the careful reading and constructive suggestions. We have revised the manuscript accordingly.
R1. Performance gains, training conditions, and inference efficiency
1) Comparison with the same backbone and training data. We added C0, a category-supervised baseline for RT-DETR-World-S. C0 and the full model C5 share the same detector architecture, frozen DINOv3-S/16 backbone, and GroundingCapv2 annotations. C0 uses only category names and bounding boxes for positive supervision, with all description alignment and RNR disabled. On LVIS minival, C5 achieves 35.2 Fixed AP, 6.3 percentage points above C0’s 28.9 (Table 8).
2) Architecture and training conditions. Table 2 compares detectors under their original training recipes. RT-DETR-World-S and OV-DEIM-L both use DINOv3-S/16. OV-DEIM-L fine-tunes its backbone on OG and uses text features from MobileCLIP pretrained on image–text pairs. We freeze DINOv3 to retain its dense spatial features and use GroundingCapv2 object and image descriptions to supervise region queries and global visual representations, respectively. Each detector’s backbone, training data, and objectives are tied to its design. Matching the backbone and how it is updated, or using the same training data and annotations, often requires adapting feature interfaces or supervision objectives. This makes it difficult to isolate a single variable across architectures while preserving each method’s core design.
3) Design and benefits of DDA and RNR. DDA transfers description semantics directly to the detector’s visual representations. LLMDet trains detection representations by generating region phrases and image descriptions. DDA uses precomputed LLM2CLIP embeddings to align object descriptions with queries matched to annotated boxes and image descriptions with global visual features. The MiniLM path supervises the same queries with object descriptions and shares its text encoder with the category matching used at inference. The detector thus learns instance details and scene context during training and uses lightweight query–category matching at inference. On the same compact detector, standard contrastive alignment achieves 33.6 Fixed AP, 5.1 points above the adapted LLMDet-style generation objective (Table 9). Adding MiniLM object description alignment raises Fixed AP from 33.1 to 35.2 (C1 → C5), and adding direct teacher alignment at both levels raises it from 32.2 to 35.2 (C2 → C5, Table 8).
RNR reduces repulsion between negatives that share attributes while preserving instance discrimination. SoftCLIP incorporates semantic similarity into soft targets. RNR preserves the box-level positive pairs established by Hungarian matching. SRCL progressively incorporates the student model’s cross-modal similarities and normalizes negative weights to a mean of 1. RNR uses frozen teacher embeddings to independently downweight each related negative by a bounded amount. When instances in the same image share attributes, each matched query retains its assigned box description as the positive target. Related negatives receive less weight while still helping distinguish instances, and unrelated negatives retain their original weights. With teacher features and alignment targets fixed, RNR raises Fixed AP from 33.6 to 35.2 and rare-category Fixed AP from 27.6 to 32.0 (Table 10(b)). Under the same auxiliary alignment setup, RNR outperforms the adapted SoftCLIP and SRCL variants by 1.4 and 1.3 points, respectively (Table 9).
4) Accuracy and inference efficiency. In the original submission, we compiled and ran the visual network and decoder separately. For the revision, we exported the full detection graph to ONNX, compiled it as a single graph with TensorRT, and reran the benchmark on an NVIDIA T4. S+ increased from 58 to 71.1 FPS (Table 2). S and OV-DEIM-L have similar detector sizes and speeds (36.5M parameters at 90.4 FPS vs. 36M at 91 FPS), with LVIS Fixed AP scores of 35.2 and 35.9, respectively. S achieves better grounding, with 28.43 overall mAcc on PR-Bench and 50.60 overall top-1 localization accuracy on FineCops-Ref, leading OV-DEIM-L by 2.59 and 1.49 points (Tables 2, 5, and 6). GroundingCapv2 contains about 19% fewer detector training samples than OG (1.12M vs. 1.38M) and fewer unique category names (72,660 vs. an estimated 100,812). At higher inference budgets, S+ achieves 36.6 Fixed AP with 43.6M detector parameters at 71.1 FPS, 0.7 points above OV-DEIM-L. B achieves 40.0 Fixed AP with 120.6M detector parameters at 41.4 FPS, 4.1 points above OV-DEIM-L and the highest among the real-time open-vocabulary detectors in Table 2.