acceptodds
Under review as a conference paper at ICLR 2027

RT-DETR-World: Transferring Rich LLM Semantics to Real-Time Open-Vocabulary Detection

Abstract

Open-vocabulary detection (OVD) recognizes categories unseen during training through textual category queries, yet achieving strong generalization with real-time efficiency remains challenging. Beyond vocabulary scaling, zero-shot generalization may benefit from reusable visual–semantic cues learned from seen data, including attributes, actions, states, and contextual relations. Existing real-time OVD methods primarily emphasize vocabulary coverage and efficient region/query–text matching; under strict efficiency constraints, compact detectors may struggle to absorb rich instance semantics and scene context. We propose RT-DETR-World, a compact DETR-style detector that transfers the rich semantics conveyed by descriptions during training while retaining lightweight query–text matching at inference. We construct GroundingCapv2 with three levels of supervision: category names for standard OVD, object descriptions conveying instance-level semantics, and image descriptions conveying object relations and scene context. These descriptions serve only as training-time semantic supervision. To help the compact detector absorb these semantics, we propose Dual-Path Description Alignment (DDA), combining a deployment-consistent MiniLM pathway with a training-only LLM teacher. MiniLM provides query–category supervision and object-description alignment, while offline teacher features supervise matched queries and global visual representations at the object and image levels, respectively. All teacher features are precomputed, and the teacher-side modules are removed after training. We further propose Relation-Aware Negative Relaxation (RNR), which uses teacher-derived semantic similarities to relax related negatives while preserving exact positives. Experiments demonstrate competitive zero-shot accuracy and a favorable accuracy–efficiency trade-off.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary, Object Detection, LLM

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_yHCJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes RT-DETR-World, a real-time open-vocabulary object detection (OVD) framework that transfers rich language semantics into a compact detector without introducing additional inference-time overhead from the proposed training modules. The authors construct GroundingCapv2, a large-scale dataset containing 1.12M images and 8.05M regions with three levels of supervision: category names, object descriptions, and image descriptions. To leverage this supervision, the proposed Dual-Path Description Alignment (DDA) combines a lightweight MiniLM pathway with a training-only LLM2CLIP teacher, aligning object queries and global visual representations with instance-level and scene-level semantics. Relation-Aware Negative Relaxation (RNR) further modifies contrastive learning by reducing the repulsion between semantically related negative samples while preserving exact positive correspondences. The detector uses a frozen DINOv3 backbone and lightweight DETR-style architecture, while the auxiliary teacher components are removed after training. Experiments on LVIS, ODinW, and COCO-O demonstrate competitive zero-shot detection performance and strong cross-domain robustness. The largest variant achieves 40.0 Fixed AP on LVIS at 32 FPS, while the smaller variants provide faster inference with lower accuracy.

Critical strengths
  • The paper addresses a practically meaningful problem: transferring rich semantic knowledge from language models to compact open-vocabulary detectors without requiring expensive language-model inference during deployment. The separation between training-time semantic supervision and lightweight inference is well motivated. In particular, the combination of object-level and image-level descriptions provides a reasonable way to incorporate fine-grained semantic information and scene context into detector representations.

  • The experimental evaluation is relatively comprehensive and shows encouraging results across multiple benchmarks. The proposed method achieves strong performance on LVIS, ODinW, and COCO-O, demonstrating competitive large-vocabulary detection and cross-domain robustness. The ablation studies also provide useful evidence for the proposed components. On LVIS, the full description-alignment framework improves Fixed AP from 33.1 to 35.2 over the category-supervision configuration, while RNR improves performance from 33.6 to 35.2 compared with standard hard contrastive alignment. The improvements are particularly notable for rare categories, supporting the potential benefit of richer semantic supervision.

Critical weaknesses
  • The performance improvements are not sufficiently disentangled from the effects of stronger pretrained models and training data. RT-DETR-World employs DINOv3 backbones and GroundingCapv2, which provides substantially richer annotations than the training datasets used by several competing real-time OVD methods. Although Table 5 includes useful internal ablations, comparisons against YOLO-World, YOLOE, and OV-DEIM involve different visual backbones, training corpora, and model capacities. Therefore, the reported superiority cannot be attributed solely to DDA and RNR. For example, RT-DETR-World-S+ achieves 36.6 Fixed AP at 58 FPS, while OV-DEIM-L achieves 35.9 Fixed AP at 91 FPS, indicating that the accuracy improvement comes with a considerable efficiency trade-off. A more controlled comparison using matched backbones, training data, or computational budgets would substantially strengthen the empirical claims. Furthermore, the paper should report the computational cost of constructing GroundingCapv2 and precomputing LLM teacher features, since the absence of inference overhead does not imply that the overall approach is computationally inexpensive.

  • The methodological novelty and the claimed compositional generalization benefits are not fully established. Multi-level semantic distillation and similarity-aware contrastive learning have already been explored in prior work, including LLMDet, SoftCLIP, SRCL, and related knowledge-distillation-based OVD methods. DDA mainly combines existing alignment objectives at different semantic levels, while RNR modifies the contribution of semantically related negatives using teacher-derived similarities. Although the integration is reasonable, the conceptual distinction from these approaches appears relatively incremental. More importantly, the paper motivates rich descriptions through compositional generalization, arguing that attributes and object relationships learned from seen categories can facilitate recognition of unseen concepts. However, the evaluation primarily reports aggregate AP on existing detection benchmarks, without directly demonstrating that the model learns or transfers these compositional semantic factors. The authors should provide targeted evaluations on unseen attribute-object combinations, fine-grained category distinctions, or controlled compositional generalization benchmarks. Additionally, since GroundingCapv2 incorporates large-vocabulary sources such as V3Det, the overlap between training categories and evaluation vocabularies should be explicitly analyzed to clarify the extent of genuine unseen-category recognition.

Questions for authors
  1. Can the authors provide a more controlled comparison that separates the contribution of DDA and RNR from the effects of DINOv3 pretraining and GroundingCapv2? In particular, how does a strong detector trained with the same visual backbone and dataset but without the proposed semantic-alignment mechanisms compare against the complete framework?

  2. The paper emphasizes compositional zero-shot generalization through reusable semantic factors such as attributes, materials, and object parts. Can the authors provide quantitative evidence that the proposed semantic distillation specifically improves the recognition of unseen attribute-object combinations or other compositional concepts, rather than simply improving general-purpose detection representations?

  3. How much category overlap exists between GroundingCapv2 and the LVIS/ODinW evaluation vocabularies? Would the authors be able to report results separately for categories that are absent from the training category vocabulary? This would help clarify the interpretation of the reported zero-shot generalization performance.

  4. What is the total computational cost of the GroundingAgent-Opt annotation pipeline and offline LLM feature extraction? Additionally, has the quality of the automatically generated object descriptions and normalized category labels been quantitatively verified through a manually audited subset? These details would help assess the scalability and reliability of the proposed training framework.

4 comments

We thank the reviewer for the careful reading and constructive suggestions. We have revised the manuscript accordingly.

R1. Performance gains, training conditions, and inference efficiency

1) Comparison with the same backbone and training data. We added C0, a category-supervised baseline for RT-DETR-World-S. C0 and the full model C5 share the same detector architecture, frozen DINOv3-S/16 backbone, and GroundingCapv2 annotations. C0 uses only category names and bounding boxes for positive supervision, with all description alignment and RNR disabled. On LVIS minival, C5 achieves 35.2 Fixed AP, 6.3 percentage points above C0’s 28.9 (Table 8).

2) Architecture and training conditions. Table 2 compares detectors under their original training recipes. RT-DETR-World-S and OV-DEIM-L both use DINOv3-S/16. OV-DEIM-L fine-tunes its backbone on OG and uses text features from MobileCLIP pretrained on image–text pairs. We freeze DINOv3 to retain its dense spatial features and use GroundingCapv2 object and image descriptions to supervise region queries and global visual representations, respectively. Each detector’s backbone, training data, and objectives are tied to its design. Matching the backbone and how it is updated, or using the same training data and annotations, often requires adapting feature interfaces or supervision objectives. This makes it difficult to isolate a single variable across architectures while preserving each method’s core design.

3) Design and benefits of DDA and RNR. DDA transfers description semantics directly to the detector’s visual representations. LLMDet trains detection representations by generating region phrases and image descriptions. DDA uses precomputed LLM2CLIP embeddings to align object descriptions with queries matched to annotated boxes and image descriptions with global visual features. The MiniLM path supervises the same queries with object descriptions and shares its text encoder with the category matching used at inference. The detector thus learns instance details and scene context during training and uses lightweight query–category matching at inference. On the same compact detector, standard contrastive alignment achieves 33.6 Fixed AP, 5.1 points above the adapted LLMDet-style generation objective (Table 9). Adding MiniLM object description alignment raises Fixed AP from 33.1 to 35.2 (C1 → C5), and adding direct teacher alignment at both levels raises it from 32.2 to 35.2 (C2 → C5, Table 8).

RNR reduces repulsion between negatives that share attributes while preserving instance discrimination. SoftCLIP incorporates semantic similarity into soft targets. RNR preserves the box-level positive pairs established by Hungarian matching. SRCL progressively incorporates the student model’s cross-modal similarities and normalizes negative weights to a mean of 1. RNR uses frozen teacher embeddings to independently downweight each related negative by a bounded amount. When instances in the same image share attributes, each matched query retains its assigned box description as the positive target. Related negatives receive less weight while still helping distinguish instances, and unrelated negatives retain their original weights. With teacher features and alignment targets fixed, RNR raises Fixed AP from 33.6 to 35.2 and rare-category Fixed AP from 27.6 to 32.0 (Table 10(b)). Under the same auxiliary alignment setup, RNR outperforms the adapted SoftCLIP and SRCL variants by 1.4 and 1.3 points, respectively (Table 9).

4) Accuracy and inference efficiency. In the original submission, we compiled and ran the visual network and decoder separately. For the revision, we exported the full detection graph to ONNX, compiled it as a single graph with TensorRT, and reran the benchmark on an NVIDIA T4. S+ increased from 58 to 71.1 FPS (Table 2). S and OV-DEIM-L have similar detector sizes and speeds (36.5M parameters at 90.4 FPS vs. 36M at 91 FPS), with LVIS Fixed AP scores of 35.2 and 35.9, respectively. S achieves better grounding, with 28.43 overall mAcc on PR-Bench and 50.60 overall top-1 localization accuracy on FineCops-Ref, leading OV-DEIM-L by 2.59 and 1.49 points (Tables 2, 5, and 6). GroundingCapv2 contains about 19% fewer detector training samples than OG (1.12M vs. 1.38M) and fewer unique category names (72,660 vs. an estimated 100,812). At higher inference budgets, S+ achieves 36.6 Fixed AP with 43.6M detector parameters at 71.1 FPS, 0.7 points above OV-DEIM-L. B achieves 40.0 Fixed AP with 120.6M detector parameters at 41.4 FPS, 4.1 points above OV-DEIM-L and the highest among the real-time open-vocabulary detectors in Table 2.

R2. Description supervision and compositional grounding

We added two referring expression grounding evaluations, PR-Bench and FineCops-Ref. RT-DETR-World-S and S+ rank in the top two on the overall metric of both benchmarks. PR-Bench tests whether models can use attributes, positions, and object relations to localize targets. S+ achieves 28.68 overall mAcc, 2.84 percentage points above the strongest baseline (Table 5). On FineCops-Ref, S achieves the highest overall top-1 accuracy at 50.60. Level 3 requires combining multiple cues to distinguish instances of the same category. S+ achieves 26.75 P@1 on this subset, outperforming the strongest baseline by 3.82 percentage points, the largest margin across the three difficulty levels (Table 6). In the paired image-order distractor evaluation, where the referring expression is held fixed, S achieves 33.88 R@1, 4.24 percentage points above the strongest baseline (Table 14).

ModelPR-Bench Overall mAcc (%) ↑FineCops-Ref Overall P@1 (%) ↑
YOLO-Worldv2.1-L24.9346.17
YOLOEv8-L25.0248.79
YOLOEv11-L25.1047.81
OV-DEIM-L25.8449.11
RT-DETR-World-S28.4350.60
RT-DETR-World-S+28.6850.29

The gains from description supervision also extend to attribute and relation grounding. B0 and B1 use the same detector architecture and GroundingCap-1M. With description-based teacher supervision, B1 raises overall PR-Bench mAcc from 25.33 to 28.40, with gains of 7.09 and 3.67 percentage points on attribute and relation mAcc, respectively. FineCops-Ref Level 3 P@1 also rises from 28.66 to 29.51 (Tables 11–12). On GroundingCapv2, C5 adds direct teacher alignment and outperforms C2 by 3.32 and 2.27 percentage points on attribute and relation mAcc, respectively (Table 13). Results on both datasets suggest that description-based teacher supervision helps the detector distinguish target instances using fine-grained semantic cues.

R3. Training vocabulary overlap and detection beyond the training vocabulary

Common-88 consists of 88 LVIS minival categories whose primary names match neither the GroundingCapv2 nor the OG training vocabulary. When official LVIS synonyms are included, 52 categories remain unmatched in both vocabularies, forming Common-52. For YOLO-Worldv2.1-L, the audit covers OG only. Its additional CC-LiteV2 training data remains unaudited.

ModelCommon-52 Fixed AP (%) ↑Common-88 Fixed AP (%) ↑
YOLO-Worldv2.1-L29.2828.82
YOLOEv8-L29.1929.10
YOLOEv11-L26.9228.12
OV-DEIM-L30.3130.70
RT-DETR-World-S27.5827.96
RT-DETR-World-S+30.0230.74
RT-DETR-World-B30.6533.27

We use each model’s full-vocabulary predictions and average per-category Fixed AP over the same subsets. RT-DETR-World-B ranks first on both subsets, achieving 30.65 Fixed AP on Common-52 and 33.27 on Common-88, outperforming the strongest baseline, OV-DEIM-L, by 0.34 and 2.57 percentage points, respectively (Table 7).

R4. Offline computational cost and annotation reliability

The GroundingAgent-Opt annotation pipeline ran for about 5 days on two RTX PRO 6000 GPUs (96 GB each), totaling approximately 240 GPU-hours. Extracting LLM2CLIP features for all image-level and object-level descriptions across the five data sources takes approximately 16.6 hours on two RTX 3090 GPUs, equivalent to 33.2 GPU-hours. GroundingCapv2 reuses the image-level descriptions from GroundingCap-1M. The new annotations and cached features are generated once offline and can be reused across detector sizes and training runs.

Ten reviewers manually audited a sample of 50,000 images using the original images and annotated boxes, with GPT assisting the review. Before correction, the manual audit found accuracies of 97.3% for normalized category labels, 95.6% for object descriptions, and 93.2% for relation statements. We then corrected the annotation errors confirmed during the audit.

Thank you for the thorough rebuttal and additional experiments. I appreciate the authors' efforts to address my concerns.

The controlled ablations clarify the contributions of DDA and RNR, while the additional grounding evaluations, vocabulary-overlap analysis, and annotation details strengthen the empirical evidence.

My remaining concern is that, while the new evaluations demonstrate improved compositional grounding, generalization to unseen attribute–object combinations remains unverified. I encourage the authors to clarify this distinction and appropriately qualify their claims in the revised paper.

Overall, the rebuttal has satisfactorily addressed my main concerns. I will maintain my rating of 3.

Thank you for your positive assessment of our additional experiments and responses, and for your further suggestions on the wording of the paper.

In the revised manuscript, we will clearly distinguish compositional grounding from generalization to unseen attribute–object combinations. The additional referring expression grounding evaluations and description-supervision comparisons support the model’s improved ability to distinguish target instances using attributes, relations, and multiple semantic cues. The Common-52/88 results demonstrate its detection capability beyond the audited training category vocabularies. We will accordingly revise statements concerning “compositional zero-shot generalization” to align our claims with the capabilities verified by these experiments.

Thank you for this suggestion, which helps us present the benefits of our method and our experimental conclusions more precisely.

Reviewer_tBJQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces RT-DETR-World, an open-vocabulary object detection model. During the training phase, the model effectively leverages rich semantic supervision derived from object and image descriptions, while maintaining a lightweight query-text matching mechanism during inference. The framework achieves exceptional zero-shot detection performance and strikes an optimal balance between accuracy and efficiency.

Critical strengths
  1. By incorporating supervision from a large language model during the training phase, the method effectively transfers rich semantic knowledge without incurring any additional computational cost during inference.

  2. Through a confidence-based annotation filtering process, the approach provides well-structured semantic supervision characterized by multi-granularity features, which is both logical and highly fine-grained.

  3. Evaluations on the LVIS, ODinW, and COCO-O datasets demonstrate that this method achieves a balance between inference speed and the accuracy of the inference results.

Critical weaknesses
  1. The experiments do not directly establish whether reusable description semantics improve unseen-concept generalization. The authors should analyze training–evaluation vocabulary overlap and evaluate unseen semantic compositions with and without description supervision.

  2. Table 5 changes both annotation formulations and supervision objectives when comparing GroundingCap-1M and GroundingCapv2, making the dataset's independent contribution unclear. A c ontrolled comparison with matched training samples and identical supervision objectives is needed.

  3. Although Table 5 demonstrates the benefits of LLM teacher alignment, the specific contribution of LLM-derived semantics remains unclear. Comparisons with conventional text encoders (e.g., CLIP or SigLIP) as alignment targets and RNR similarity estimators are needed.

  4. The paper lacks direct evidence of annotation correctness and the benefits of the additional Qwen3-VL-32B adjudication. The authors should assess annotation quality and compare the full pipeline with directly falling back on uncertain samples without 32B adjudication.

Questions for authors

I would appreciate further quantitative evidence addressing the key concerns raised above, particularly regarding (1) whether description supervision genuinely improves generalization to unseen semantic concepts, (2) the independent contribution of GroundingCapv2 under controlled settings, (3) the advantages of LLM-derived semantic alignment over conventional text encoders such as CLIP or SigLIP, and (4) the annotation quality and necessity of Qwen3-VL-32B adjudication. These issues are central to validating the paper's claimed contributions, and qualitative explanations alone may not be sufficient. If the authors cannot provide convincing quantitative experiments to address these concerns, I may consider lowering my score.

4 comments

We thank the reviewer for the careful reading and constructive suggestions. We have revised the manuscript accordingly.

R1. Detection beyond the training vocabulary and description supervision

Detection beyond the training vocabulary. We matched LVIS minival categories against the audited training vocabularies of GroundingCapv2 and OG. Matching primary category names leaves 88 categories unmatched in both vocabularies (Common-88). Including official LVIS synonyms leaves 52 categories unmatched in both (Common-52). For YOLO-Worldv2.1-L, the audit covers OG. Its additional CC-LiteV2 training data remains unaudited. RT-DETR-World-B ranks first on both subsets, outperforming the strongest baseline by 0.34 and 2.57 percentage points, respectively (Table 7).

PR-Bench tests whether models can localize targets using cues such as attributes, positions, and object relations. FineCops-Ref Level 3 requires models to combine multiple cues to identify a target among instances of the same category. RT-DETR-World-S+ outperforms the strongest baseline by 2.84 and 3.82 percentage points on these two evaluations, respectively (Tables 5–6).

ModelCommon-52 Fixed AP (%) ↑Common-88 Fixed AP (%) ↑PR-Bench Overall mAcc (%) ↑FineCops-Ref L3 P@1 (%) ↑
YOLO-Worldv2.1-L29.2828.8224.9317.41
YOLOEv8-L29.1929.1025.0221.87
YOLOEv11-L26.9228.1225.1022.51
OV-DEIM-L30.3130.7025.8422.93
RT-DETR-World-S27.5827.9628.4323.99
RT-DETR-World-S+30.0230.7428.6826.75
RT-DETR-World-B30.6533.2726.8025.69

Contribution of description supervision. B0 and B1 use the same detector architecture and GroundingCap-1M. With description-based teacher supervision, B1 raises overall PR-Bench mAcc from 25.33 to 28.40, with gains of 7.09 and 3.67 percentage points on attribute and relation grounding, respectively. FineCops-Ref Level 3 P@1 rises from 28.66 to 29.51. LVIS Fixed AP rises from 20.66 to 25.74 on Common-52 and from 23.68 to 27.69 on Common-88 (Tables 11–12).

On GroundingCapv2, both C2 and C5 retain MiniLM description alignment. C5 adds direct image- and object-level teacher alignment and outperforms C2 by 3.32 and 2.27 percentage points on the PR-Bench attribute and relation subsets, respectively, and by 2.48 and 1.65 points on Common-52 and Common-88, respectively (Table 13). Both comparisons show gains in attribute and relation grounding and in detection beyond the audited training vocabularies with richer description supervision.

R2. Contribution of GroundingCapv2 annotations

To distinguish the contributions of base annotations and description supervision, we added the category-supervised baseline C0 to Table 8. GroundingCapv2 uses the same source samples as GroundingCap-1M. Both B0 and C0 use RT-DETR-World-S with a frozen DINOv3-S/16 backbone and train the detector using only bounding boxes and their associated text labels. B0 uses source category labels or region phrases, while C0 uses normalized category names. On LVIS minival, C0 achieves 28.9 Fixed AP versus 27.9 for B0, a difference of 1.0 percentage point.

Using the same GroundingCapv2 annotations, adding MiniLM object description alignment while retaining both levels of teacher alignment raises LVIS minival Fixed AP from 33.1 to 35.2 (C1 → C5). Adding direct teacher alignment at both levels while retaining MiniLM description alignment raises Fixed AP from 32.2 to 35.2 (C2 → C5, Table 8).

R3. Choice of teacher encoder

LLM2CLIP supports long texts and was contrastively fine-tuned on image captions. The same precomputed description embeddings serve as visual alignment targets and provide description similarities for RNR. In our training data, MobileCLIP-B(LT)’s 77-token window would truncate 98.7% of image-level descriptions. SigLIP Base has a 64-token window. With these shorter-window encoders, long descriptions would be truncated before being used for image-level alignment and RNR in that branch. LLM2CLIP lets both objectives use more of the scene context and object relations, and its features are extracted once offline.

With MiniLM description alignment and its RNR weights retained, C5 adds direct teacher alignment at the image and object levels and achieves 35.2 LVIS Fixed AP, compared with 32.2 for C2 (Table 8). With teacher features and alignment targets held fixed, enabling RNR raises Fixed AP from 33.6 to 35.2 and rare-category Fixed AP from 27.6 to 32.0 (Table 10(b)).

C2 and C5 both use GroundingCapv2 and MiniLM description alignment. With direct teacher alignment, C5 outperforms C2 by 3.32 and 2.27 percentage points on the PR-Bench attribute and relation subsets, respectively. On LVIS Common-52 and Common-88, C5 achieves 27.58 and 27.96 Fixed AP, compared with 25.10 and 26.31 for C2 (Table 13).

R4. Annotation quality and the benefit of 32B adjudication

Using the annotation logs, we simulated skipping Qwen3-VL-32B adjudication by reverting uncertain regions sent for adjudication to their original category names. Compared with this direct fallback, the full pipeline retained object descriptions for an additional 83,034 regions in COCO and 11,751 in V3Det. Coverage of generated object descriptions rose from 78.74% to 85.47% on COCO and from 85.88% to 87.60% on V3Det. Thus, 32B adjudication enabled description alignment for 94,785 regions that would otherwise have received only category-name supervision.

Ten reviewers manually audited a sample of 50,000 images using the original images and target boxes, with GPT assisting the review. Before correction, the accuracy of normalized category labels, object descriptions, and the object relations stated in those descriptions was 97.3%, 95.6%, and 93.2%, respectively. We then corrected the errors confirmed during the audit.

Thank you for your responses, and they have addressed my concerns. I keep the week accept decision.

Thank you for following up. We are glad that our responses addressed your concerns. We will release the GroundingCapv2 dataset, our training and evaluation code, and the trained model checkpoints so that others can reproduce our results and build on this work.

Reviewer_4KJNOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to transfer rich language semantics to open-vocabulary object detectors while maintaining real-time inference efficiency. The authors propose RT-DETR-World and reorganize GroundingCap-1M into GroundingCapv2, comprising category names, object descriptions, and image descriptions. DDA combines MiniLM description alignment with offline LLM teacher supervision to learn instance semantics and scene context. RNR further reduces the repulsion of semantically related negatives. The teacher model and auxiliary alignment modules are not used at inference. Experiments cover LVIS, ODinW, and COCO-O, with ablations on supervision configurations and alignment strategies.

Critical strengths
  1. The research problem has practical relevance. The paper seeks to improve open-vocabulary recognition while maintaining real-time inference efficiency, which is valuable for deployment.
  2. GroundingCapv2 builds on GroundingCap-1M by adding instance descriptions to part of the data and explicitly separating category names, object descriptions, and image descriptions. This supports multi-level semantic supervision.
  3. The method achieves good results on multiple standard open-vocabulary detection benchmarks and demonstrates a competitive accuracy–efficiency trade-off.
Critical weaknesses
  1. A central motivation is that reusable attribute and relational semantics can help recognize unseen concepts. However, this explanation is mainly supported by improvements in overall detection performance. The experiments do not directly show whether the model has learned these semantics or whether they contribute to recognizing unseen concepts.
  2. GroundingCapv2 is an important contribution, but its annotation quality lacks an independent, quantitative evaluation. Description acceptance rates and annotation coverage provide information about data curation, but do not sufficiently establish the accuracy of the descriptions or the reliability of their relational information.
  3. The distinction between RNR and existing negative reweighting methods needs further clarification. Its loss structure, which preserves positive terms and downweights related negatives, is close to SRCL. The current results do not clearly distinguish the gains from the reweighting design from those provided by the teacher features.
  4. The current comparisons do not fully establish the method’s advantages in accuracy and efficiency. For example, with nearly identical detector parameter counts, OV-DEIM-L achieves both higher accuracy and higher FPS than RT-DETR-World-S. The accuracy gains of larger variants should also be discussed alongside their model sizes and inference costs.
Questions for authors
  1. Could the authors provide more direct experiments showing that the model has learned the attributes and object relations expressed in the descriptions? For example, could it be compared with other models on detection tasks that require distinguishing attributes or understanding object relations?
  2. Could the authors quantify the specific differences between GroundingCapv2 and GroundingCap-1M, and report how many LVIS test categories, including their synonymous expressions, already appear in the training supervision of each dataset? I would like to understand how much of the improvement comes from transferring attribute and relational semantics, and how much comes from more extensive category coverage. Do the gains also hold for categories not covered by the training supervision?
  3. Could the authors report results for other detectors trained on GroundingCapv2? This would help assess whether the dataset benefits different models and further separate the contribution of the data improvements from that of the proposed method.
  4. Could the authors provide performance comparisons after pretraining with the main baselines using the same visual backbone, training data, and comparable training budgets? Differences in these conditions currently make it difficult to determine which components account for the improvements.
  5. Given the strength of LLM2CLIP’s text representations, have the authors considered directly distilling them into MiniLM? What additional benefits does the current dual-path alignment design offer over this more direct alternative? Furthermore, the method mainly aligns visual representations with text representations of entire descriptions, without explicitly separating the attributes and object relations within them. How does alignment at this representation level help establish fine-grained correspondences for attributes and relations? Could the authors provide further explanation or experimental evidence?
9 comments

We thank the reviewer for the careful reading and constructive suggestions. We have revised the manuscript accordingly.

R1. Fine-grained grounding and generalization beyond the training vocabulary

Dataset differences and category coverage. GroundingCapv2 retains GroundingCap-1M’s 1.12M source samples, 8.05M regions, and existing image-level descriptions. Matching positive training labels against LVIS primary category names and official synonyms yields coverage of 1,099/1,203 categories for GroundingCap-1M and 1,095/1,203 for GroundingCapv2. Among the 804 categories with ground-truth annotations in minival, the corresponding counts are 749 and 742. GroundingCapv2 adds instance description supervision while maintaining similar LVIS category coverage.

Fine-grained grounding and detection beyond the training vocabulary. PR-Bench requires models to localize targets using cues such as attributes, positions, or object relations. S+ leads the strongest baseline by 2.84 percentage points in overall mAcc. S leads by 7.81 points on position, and S+ leads by 0.57 points on relations. On FineCops-Ref Level 3, which requires combining multiple cues to distinguish instances of the same category, S+ leads the strongest baseline by 3.82 points in P@1 (Tables 5–6). LVIS Common-88 excludes categories whose primary names match the audited GroundingCapv2 or OG training vocabularies. Common-52 additionally excludes matches to official LVIS synonyms. None of these 52 categories match the audited GroundingCap-1M training labels either. B ranks first on both subsets among the compared models (Table 7). The audit excludes CC-LiteV2, which YOLO-Worldv2.1-L also uses for training.

Contribution of description supervision. Both B0 and B1 use GroundingCap-1M. With description-based teacher supervision, B1 scores 7.09 and 3.67 percentage points higher than B0 on PR-Bench attribute and relation mAcc, respectively, and raises Common-52 Fixed AP from 20.66 to 25.74 (Tables 11–12). C2 and C5 both use GroundingCapv2 and MiniLM description alignment. With direct teacher alignment at both the image and object levels, C5 scores 3.32 and 2.27 points higher than C2 on attribute and relation mAcc, respectively, and raises Common-52 Fixed AP from 25.10 to 27.58 (Table 13).

ModelCommon-52 Fixed AP (%) ↑Common-88 Fixed AP (%) ↑PR-Bench Overall mAcc (%) ↑FineCops-Ref L3 P@1 (%) ↑
YOLO-Worldv2.1-L29.2828.8224.9317.41
YOLOEv8-L29.1929.1025.0221.87
YOLOEv11-L26.9228.1225.1022.51
OV-DEIM-L30.3130.7025.8422.93
RT-DETR-World-S27.5827.9628.4323.99
RT-DETR-World-S+30.0230.7428.6826.75
RT-DETR-World-B30.6533.2726.8025.69

R2. Annotation quality of GroundingCapv2

Ten annotators manually audited a sample of 50,000 images. Using the original images and annotated boxes, they checked whether the normalized category labels matched the boxed instances, whether the object descriptions were supported by visual evidence, and whether the stated object relations held. GPT assisted with verification. Before correction, the manual audit found accuracies of 97.3%, 95.6%, and 93.2%, respectively. We then corrected the annotation errors confirmed during the audit.

R3. Cross-detector comparisons and training conditions

GroundingCapv2’s three levels of annotation are designed to support RT-DETR-World’s training objectives. Object descriptions supervise region queries, and image descriptions supervise global visual representations. Other real-time detectors would need corresponding training objectives to fully use both types of descriptions. Comparing an adapted detector with its original baseline would therefore reflect changes in both annotations and training objectives.

For example, OV-DEIM-L and RT-DETR-World-S both use DINOv3-S/16. OV-DEIM-L fine-tunes its visual backbone on OG and uses MobileCLIP text features pretrained with image–text alignment. We freeze DINOv3 to retain its dense spatial features and train the detector’s representations with description supervision at multiple levels. Each detector’s backbone, training data, and objectives are tied to its design. Matching backbones and how they are updated, or using the same training data and annotations, often requires adapting feature interfaces or supervision objectives. This makes it difficult to isolate a single variable across architectures while preserving each method’s core design.

Tables 8 and 9 use the same detector architecture, frozen visual backbone, and GroundingCapv2 annotations. Table 8 evaluates the contributions of the description alignment branches, and Table 9 compares auxiliary alignment strategies, including RNR.

R4. How RNR differs from prior methods and what it contributes

Both RNR and SRCL preserve positive terms and reweight negatives. SRCL progressively incorporates the student model’s cross-modal similarities during training and normalizes negative weights to a mean of 1. RNR uses frozen LLM2CLIP description embeddings to apply a bounded weight reduction independently to each related negative. In object detection, multiple instances in the same image may share attributes. RNR reduces repulsion between a query and similar but unmatched descriptions. Unrelated negatives retain their original weights, and related negatives retain enough weight to help distinguish instances.

SoftCLIP incorporates sample similarity into a soft target distribution. RNR preserves the box-level positive pairs established by Hungarian matching. Each matched query is supervised by the description of its assigned box, and description similarity only affects negative weights. Under the same auxiliary alignment setup, RNR achieves 35.2 Fixed AP, compared with 33.8 and 33.9 for the adapted SoftCLIP and SRCL variants, respectively (Table 9).

With teacher features and alignment supervision held fixed, increasing RNR’s α\alpha from 0 to 0.25 raises Fixed AP from 33.6 to 35.2 and rare-category Fixed AP from 27.6 to 32.0 (Table 10(b)). This isolates the gain from negative reweighting under the same teacher supervision.

C2 already uses teacher-derived similarities to weight negatives in MiniLM description alignment. Adding direct teacher alignment at the image level (C3), object level (C4), or both levels (C5) raises Fixed AP from 32.2 to 33.1, 33.6, and 35.2, respectively (Table 8). Table 10(a) also shows an increase from 32.2 to 35.2 at a fixed RNR coefficient, supporting the additional benefit of direct teacher alignment.

R5. Accuracy and inference efficiency

For the inference benchmark in the original submission, we compiled and ran the visual network and decoder separately. For the revised manuscript, we exported the full detection graph to ONNX, compiled it as a single graph with TensorRT, and reran the benchmark on an NVIDIA T4. With the updated deployment, RT-DETR-World-S, S+, and B achieved 90.4, 71.1, and 41.4 FPS, respectively (Table 2).

At around 90 FPS, RT-DETR-World-S performs better on referring expression grounding. S and OV-DEIM-L have similar detector sizes and speeds (36.5M parameters and 90.4 FPS vs. 36M and 91 FPS), with LVIS Fixed AP scores of 35.2 and 35.9, respectively. S achieves 28.43 overall mAcc on PR-Bench and 50.60 overall top-1 localization accuracy on FineCops-Ref, outperforming OV-DEIM-L by 2.59 and 1.49 percentage points, respectively (Tables 2, 5, and 6). GroundingCapv2 contains about 19% fewer detection training samples than the OG dataset used by OV-DEIM (1.12M vs. 1.38M) and fewer unique category names (72,660 vs. an estimated 100,812).

With a higher inference budget, S+ achieves 36.6 Fixed AP at 71.1 FPS with 43.6M detector parameters, exceeding OV-DEIM-L by 0.7 percentage points. B achieves 40.0 Fixed AP at 41.4 FPS with 120.6M detector parameters, exceeding OV-DEIM-L by 4.1 percentage points and ranking first among the real-time open-vocabulary detectors compared in Table 2.

R6. Dual-path semantic transfer and fine-grained grounding

LLM2CLIP provides a single embedding per description, while MiniLM produces token representations for query–text matching. Token-level distillation therefore lacks a corresponding teacher target for each MiniLM token. DDA transfers LLM2CLIP’s description semantics directly to the detector’s visual representations by aligning object description embeddings with queries matched to annotated boxes and image description embeddings with global visual features. The MiniLM path aligns object descriptions with the same queries, and RNR uses teacher-derived description similarities to adjust negative weights in this path. This allows teacher semantics to directly supervise region and scene representations and guide MiniLM’s vision–language alignment through shared queries. Using both paths achieves 35.2 Fixed AP, compared with 33.1 when removing MiniLM description alignment and 32.2 when removing direct teacher alignment (Table 8).

Each description is encoded as a single embedding and aligned with a visual representation at the corresponding level. In object-level alignment, box-level matching determines which instance each description supervises. For example, “the red cup to the left of the plate” and “the blue cup to the right of the plate” share a category name, but each query aligns with the description of its matched box, receiving distinct color and spatial relation cues. In image-level alignment, we aggregate the detector’s multiscale visual features into a global representation and align it with the description of the same image. Object co-occurrence, relations, and scene context in the description provide scene-level supervision for the shared visual features that region queries rely on.

Adding only image-level teacher alignment to C2 yields C3 and raises LVIS Fixed AP from 32.2 to 33.1 (Table 8). C5, which uses both object- and image-level teacher alignment, outperforms C2 by 3.32 and 2.27 percentage points in grounding mAcc on the PR-Bench attribute and relation subsets, respectively (Table 13).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.