- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper proposes ProtoSet-DETR, a DETR-based framework for open-vocabulary object detection that represents each category using multiple learned text prototypes rather than a single class-name embedding or a fixed average of textual descriptions.
The method starts from 8 offline LLM-generated descriptions for each category and uses a shared Text Prototype Aggregator (TPA) to produce a small prototype bank. The detector then performs region-dependent matching between visual features and the prototypes for encoder proposal scoring, language-guided query initialization, and final classification.
The paper also introduces Adaptive Prototype Regularization (APR), which promotes prototype diversity and balanced slot usage. It also proposes a training-only Region–Prototype Semantic Alignment (RPSA) loss that aligns clustered encoder features with the text prototype bank. Experiments on OV-LVIS report improvements over LaMI-DETR with the same ConvNeXt-L/DINO backbone, and the paper further evaluates transfer from LVIS to COCO and Objects365 without fine-tuning.
Critical strengths
-
Well-motivated category representation.
The paper identifies a meaningful limitation of representing each category with a single class-name embedding or an averaged description embedding. Constructing multiple text prototypes allows different visual regions to match different semantic aspects of a category, which is particularly suitable for open-vocabulary detection. -
Category-shared prototype construction for novel classes.
The Text Prototype Aggregator is shared across categories, allowing the same learned construction rule to generate prototypes for novel categories at inference time. -
Comprehensive integration into the detector.
The prototype bank is used consistently for encoder proposal scoring, decoder query initialization, and final classification, rather than being introduced only as a replacement classifier. This makes the proposed representation tightly coupled with the detection pipeline.
Critical weaknesses
-
External knowledge injection for novel categories.
This method employs Claude to generate eight types of descriptions, covering definitions, states/uses, attributes/contexts, and paraphrases to construct prototypes (Section 3.2.1; Appendix A.1). These descriptions introduce a wealth of external semantic knowledge. In this sense, the approach functions much like an extension of Retrieval-Augmented Generation (RAG). The observed performance gains may stem not only from the proposed multi-prototype architecture but also from the richer prior information provided by the Claude. To precisely isolate the specific contributions of multi-prototype learning, it would be beneficial to compare the proposed method against "single-prototype" or "fixed-mean" baselines while ensuring the use of an identical set of descriptors. -
Limited reproducibility and possible benchmark-specific information.
Appendix A.1 states that the descriptions were generated using Claude through interactive sessions, but that “the generation instruction was not archived.” This makes it difficult to assess whether the generation process included benchmark-specific information, such as LVIS taxonomy, category definitions, hierarchical relations, or metadata. While this is not necessarily direct test-set leakage, it may make comparisons with class-name-only baselines less fair. The authors should release the complete prompt and generation procedure, and explicitly confirm that no validation/test annotations, images, category statistics, or performance-driven prompt revisions were used. -
High methodological complexity with limited sensitivity analysis.
The method combines TPA, language-guided query fusion, APR, PCGrad-style gradient routing, RPSA, soft clustering, confidence filtering, and CLIP score ensembling (Secs. 3.2--3.5; Appendix B.1). However, the main sensitivity studies only vary the number of textual prototypes and visual centers (Sec. 4.4 and Fig. 3). Additional sensitivity or robustness analysis for key hyperparameters---e.g., APR/RPSA loss weights, slot-prior strength, routing duration, confidence threshold, and CLIP ensemble coefficients---would improve confidence in the proposed design.
Questions for authors
1.Appendix A.1 states that the description-generation instruction was not archived. Can the authors clarify whether the generation process used any LVIS-specific taxonomy, category definitions, metadata, validation/test information, or iterative prompt refinement based on experimental results?
2.The method contains several components and hyperparameters beyond and , including APR/RPSA weights, slot priors, gradient routing, and CLIP score ensembling. Which of these hyperparameters are most influential, and how robust is the final performance to their variation?
Thank you for carefully addressing my questions and providing additional experiments and analysis.
I see potential in this approach. The fully trained baselines, comparisons with a fixed selection of descriptions, and results without CLIP score fusion address several of my earlier questions and provide clearer evidence of the method’s effectiveness.
The additional results do, however, raise a question about how much the learned aggregation mechanism contributes on its own. With auxiliary training disabled, TPA improves over directly using all eight projected descriptions by 0.6 APr and 0.3 AP, and over using five fixed descriptions by 0.8 APr and 0.5 AP. These are positive but modest gains. The full system achieves larger improvements while also introducing prototype regularization, gradient projection, and region alignment. I therefore still have reservations about the benefit of learned aggregation over simpler description representations, and whether these gains justify the added training complexity.
The method combines shared description aggregation, multi-prototype matching, and auxiliary training objectives within an existing detection framework to produce an effective system. In my view, however, the contribution remains largely an incremental system improvement, and I am not yet convinced that the methodological novelty is sufficient. This remains my main concern after reading the response.
Regarding generalization, I appreciate the semantic-distance analysis, which provides encouraging evidence for novel-category performance under the standard LVIS split. The authors could also consider evaluating the method in settings with larger semantic or visual distribution shifts, such as V3Det or ODinW13/35, while analyzing the overlap between the evaluation categories and the training categories. This would help clarify how well the shared description aggregation rule generalizes and where it is most useful.
Overall, the additional experiments strengthen the evidence for the method’s effectiveness. My concerns about its novelty and the incremental benefit of learned aggregation remain, so I will keep my original rating for now. Thank you again for the effort you have put into the discussion. I look forward to seeing the work developed further.