acceptodds
Withdrawn from ICLR 2027

Categories as Prototype Sets: Constructing Text Prototypes from Descriptions for Open-Vocabulary Object Detection

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Open-vocabulary object detection uses text representations to recognize categories that have no bounding-box annotations during training. Existing approaches often represent each category with a class-name embedding or a fixed average of description embeddings, limiting how descriptions contribute to region-level recognition. We propose ProtoSet-DETR, a DETR-based detector that learns multiple text prototypes per category from a fixed set of offline-generated descriptions. A shared Text Prototype Aggregator constructs the prototypes from frozen CLIP description embeddings using attention and a learned value projection. The detector uses this bank for proposal scoring, decoder query initialization, and classification, weighting prototype contributions according to each visual region. Adaptive Prototype Regularization encourages diversity and balanced usage, while a training-only region–prototype alignment objective connects the bank to clustered encoder features. The same aggregator constructs prototypes for novel categories. Our diagnostics show concentrated description attention and reduced redundancy among the projected embeddings. On LVIS, ProtoSet-DETR achieves 46.0 rare-category AP (APr) and 46.1 AP, improving over LaMI-DETR with the same backbone by 2.6 APr and 4.8 AP. Without fine-tuning, the LVIS-trained model transfers to COCO and Objects365 with 46.3 and 25.4 AP. These results support learning how category descriptions are represented and matched to regions. Code will be released here.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open Vocabulary Object Detection, Instruct LLM, Text Prompt

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_zUJzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes ProtoSet-DETR, a DETR-based framework for open-vocabulary object detection that represents each category using multiple learned text prototypes rather than a single class-name embedding or a fixed average of textual descriptions.

The method starts from 8 offline LLM-generated descriptions for each category and uses a shared Text Prototype Aggregator (TPA) to produce a small prototype bank. The detector then performs region-dependent matching between visual features and the prototypes for encoder proposal scoring, language-guided query initialization, and final classification.

The paper also introduces Adaptive Prototype Regularization (APR), which promotes prototype diversity and balanced slot usage. It also proposes a training-only Region–Prototype Semantic Alignment (RPSA) loss that aligns clustered encoder features with the text prototype bank. Experiments on OV-LVIS report improvements over LaMI-DETR with the same ConvNeXt-L/DINO backbone, and the paper further evaluates transfer from LVIS to COCO and Objects365 without fine-tuning.

Critical strengths
  • Well-motivated category representation.
    The paper identifies a meaningful limitation of representing each category with a single class-name embedding or an averaged description embedding. Constructing multiple text prototypes allows different visual regions to match different semantic aspects of a category, which is particularly suitable for open-vocabulary detection.

  • Category-shared prototype construction for novel classes.
    The Text Prototype Aggregator is shared across categories, allowing the same learned construction rule to generate prototypes for novel categories at inference time.

  • Comprehensive integration into the detector.
    The prototype bank is used consistently for encoder proposal scoring, decoder query initialization, and final classification, rather than being introduced only as a replacement classifier. This makes the proposed representation tightly coupled with the detection pipeline.

Critical weaknesses
  • External knowledge injection for novel categories.
    This method employs Claude to generate eight types of descriptions, covering definitions, states/uses, attributes/contexts, and paraphrases to construct prototypes (Section 3.2.1; Appendix A.1). These descriptions introduce a wealth of external semantic knowledge. In this sense, the approach functions much like an extension of Retrieval-Augmented Generation (RAG). The observed performance gains may stem not only from the proposed multi-prototype architecture but also from the richer prior information provided by the Claude. To precisely isolate the specific contributions of multi-prototype learning, it would be beneficial to compare the proposed method against "single-prototype" or "fixed-mean" baselines while ensuring the use of an identical set of descriptors.

  • Limited reproducibility and possible benchmark-specific information.
    Appendix A.1 states that the descriptions were generated using Claude through interactive sessions, but that “the generation instruction was not archived.” This makes it difficult to assess whether the generation process included benchmark-specific information, such as LVIS taxonomy, category definitions, hierarchical relations, or metadata. While this is not necessarily direct test-set leakage, it may make comparisons with class-name-only baselines less fair. The authors should release the complete prompt and generation procedure, and explicitly confirm that no validation/test annotations, images, category statistics, or performance-driven prompt revisions were used.

  • High methodological complexity with limited sensitivity analysis.
    The method combines TPA, language-guided query fusion, APR, PCGrad-style gradient routing, RPSA, soft clustering, confidence filtering, and CLIP score ensembling (Secs. 3.2--3.5; Appendix B.1). However, the main sensitivity studies only vary the number of textual prototypes KpK_p and visual centers KrK_r (Sec. 4.4 and Fig. 3). Additional sensitivity or robustness analysis for key hyperparameters---e.g., APR/RPSA loss weights, slot-prior strength, routing duration, confidence threshold, and CLIP ensemble coefficients---would improve confidence in the proposed design.

Questions for authors

1.Appendix A.1 states that the description-generation instruction was not archived. Can the authors clarify whether the generation process used any LVIS-specific taxonomy, category definitions, metadata, validation/test information, or iterative prompt refinement based on experimental results?

2.The method contains several components and hyperparameters beyond KpK_p and KrK_r, including APR/RPSA weights, slot priors, gradient routing, and CLIP score ensembling. Which of these hyperparameters are most influential, and how robust is the final performance to their variation?

Reviewer_AHGEOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ProtoSet-DETR, which represents each category using multiple learnable text prototypes instead of a single embedding. The method introduces a Text Prototype Aggregator (TPA), Adaptive Prototype Regularization (APR), and Region–Prototype Semantic Alignment (RPSA) to improve prototype diversity and visual-text alignment. Experiments on LVIS, COCO, and Objects365 demonstrate promising improvements over existing methods, particularly LaMI-DETR.

Critical strengths
  1. Clear motivation and reasonable design. The paper addresses the limitations of single-vector category representations. Using multiple text prototypes for region-adaptive matching is intuitive and meaningful.

  2. Promising experimental results. The method achieves consistent improvements over LaMI-DETR on LVIS, COCO, and Objects365. Component ablations and prototype analyses further support its effectiveness.

Critical weaknesses
  1. Insufficient comparisons with simpler baselines. The paper lacks a controlled comparison against averaging CLIP description embeddings with a learnable projection. The inference-only comparison in Table 7 shows relatively modest improvements over directly using eight projected descriptions (+0.63 AP and +0.47 APr), raising questions about the necessity of the proposed complexity.

  2. Limited generalization analysis. The aggregator is trained on 866 LVIS base categories, which cover many common concepts. The semantic overlap with COCO and Objects365 is not analyzed, making it difficult to assess generalization to substantially different unseen categories.

Questions for authors
  1. Could the authors compare against simpler, fully trained baselines, such as averaging eight CLIP descriptions with a learnable projection or directly using multiple projected descriptions?

  2. How does the aggregator perform when trained with fewer base categories or evaluated on semantically distinct unseen categories? Could the authors also analyze category overlap across datasets?

  3. Given the highly concentrated attention weights, how does learnable prototype selection compare with simply selecting five fixed descriptions?

  4. Could the authors report detector-only results without CLIP score ensembling to clarify the contribution of the proposed modules?

1 comment

Thank you for carefully addressing my questions and providing additional experiments and analysis.

I see potential in this approach. The fully trained baselines, comparisons with a fixed selection of descriptions, and results without CLIP score fusion address several of my earlier questions and provide clearer evidence of the method’s effectiveness.

The additional results do, however, raise a question about how much the learned aggregation mechanism contributes on its own. With auxiliary training disabled, TPA improves over directly using all eight projected descriptions by 0.6 APr and 0.3 AP, and over using five fixed descriptions by 0.8 APr and 0.5 AP. These are positive but modest gains. The full system achieves larger improvements while also introducing prototype regularization, gradient projection, and region alignment. I therefore still have reservations about the benefit of learned aggregation over simpler description representations, and whether these gains justify the added training complexity.

The method combines shared description aggregation, multi-prototype matching, and auxiliary training objectives within an existing detection framework to produce an effective system. In my view, however, the contribution remains largely an incremental system improvement, and I am not yet convinced that the methodological novelty is sufficient. This remains my main concern after reading the response.

Regarding generalization, I appreciate the semantic-distance analysis, which provides encouraging evidence for novel-category performance under the standard LVIS split. The authors could also consider evaluating the method in settings with larger semantic or visual distribution shifts, such as V3Det or ODinW13/35, while analyzing the overlap between the evaluation categories and the training categories. This would help clarify how well the shared description aggregation rule generalizes and where it is most useful.

Overall, the additional experiments strengthen the evidence for the method’s effectiveness. My concerns about its novelty and the incremental benefit of learned aggregation remain, so I will keep my original rating for now. Thank you again for the effort you have put into the discussion. I look forward to seeing the work developed further.

Reviewer_nx1EOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProtoSet-DETR is a description-based multiple-prototype method for open-vocabulary object detection. A category-shared Text Prototype Aggregator constructs multiple learnable text prototypes from frozen CLIP description embeddings and uses them for proposal scoring, query initialization, and classification. Adaptive Prototype Regularization promotes prototype diversity and balanced usage, while Region–Prototype Semantic Alignment aligns clustered visual features with the prototypes during training. Experiments achieve 46.0 rare-category AP and 46.1 overall AP on LVIS, improving over LaMI-DETR with the same backbone by 2.6 and 4.8 points, respectively, with transfer AP of 46.3 on COCO and 25.4 on Objects365.

Critical strengths
  • Internal comparisons support the effectiveness of the complete multiple-prototype approach

  • Shared prototype construction supports novel categories and cross-dataset transfer

Critical weaknesses

Insufficient evaluation of competitiveness and generality

  1. Incomplete comparison with recent methods. ExDet also uses LVIS-base for downstream training and achieves 26.2 AP on Objects365, exceeding the paper's 25.4, but is omitted from Table 2. Backbone and pretraining differences should be acknowledged.
  2. Limited evaluation under large-scale pretraining. The current experiments do not establish the method's accuracy under a large-scale detection pretraining protocol. Please train the proposed method under an established large-scale pretraining protocol and report its detection AP, specifying the training data and evaluation setting so that its competitiveness relative to methods such as DetCLIPv3 and OWLv2 can be assessed.
  3. Unverified generality across detectors. Experiments only use DINO with ConvNeXt-L, leaving applicability to other detection architectures unclear.
Questions for authors
  • Does learned selection outperform a structured fixed description set or all descriptions?

The value of learning to select five descriptions over fixed rules and projection has not been isolated. Appendix A.1 (p. 13) initializes the five slots to prefer positions 1, 3, 5, 6, and 8; after training, 55.1% still select their original positions (Table 6, p. 18). Please compare end-to-end training with these five fixed descriptions, all eight descriptions, and five learned prototypes, using the same WV projection, region scoring, query fusion, and training budget. Replacing the bank in a fixed checkpoint, as in Table 7, tests inference sensitivity and cannot replace this comparison. Please provide matched results if available.

  • Can ablations isolate the effects of prototype count, APR, and early gradient projection?

APR has an overall ablation, but its interaction with prototype count and the final benefit of early gradient projection remain unresolved. Table 3 (p. 8) reports 43.4 APr for TPA without APR and 44.3 with APR and early gradient routing, both without RPSA. Appendix B.1 also states that the sensitivity sweeps vary only the parameter under study. However, Section 3.3 (p. 5) sets both APR terms to zero at Kp=1, so the prototype-count sweep does not maintain the same effective regularization throughout. The 50-update rank comparison in Appendix A.2 supports early collapse prevention but does not replace a comparison of final detection performance. Please add the following ablation experiments and report final AP/APr:

  1. With Kp fixed at 5, compare APR with and without early gradient projection, keeping the remaining settings unchanged.
  2. With APR and its gradient projection disabled, compare Kp=1 and Kp=5 to isolate the gain from increasing the prototype count.
  • Can the prototype bank's contributions at the three detection stages be separated?

The prototypes simultaneously affect proposal ranking, query content, and classification, leaving the main source of improvement unclear. Section 3.2 (pp. 4–5) reuses the bank at all three stages, while Table 7 (p. 18) also replaces classification and query initialization together. Under a fixed training protocol, please provide a classification-only variant followed by variants enabling prototype query fusion and proposal scoring, or equivalent existing evidence. At each stage where the proposed prototype interface is disabled, retain the corresponding LaMI-DETR design. If classification accounts for most of the gains, this would support a simpler characterization of the contribution. If proposal or query improvements are essential, this would clarify the added value over LaMI-DETR.

  • How large are the detector-only gains on novel categories after full training?

The main novel-category AP combines the new detector with frozen CLIP region scores, so the source of the gains needs clarification. Pages 7 and 16 fully specify the ensemble: it is applied only to rare categories on LVIS and to all target categories on COCO and Objects365, with the LVIS descriptions inherited from LaMI-DETR. The concern is therefore not a missing formula. Please report AP/APr for both ProtoSet-DETR and LaMI-DETR with and without the same frozen CLIP score ensemble, using their respective 12-epoch checkpoints. Keep the CLIP model, text descriptions, and fusion rule identical between the two detectors. If detector-only gains are small but increase substantially after ensembling, please explain whether this comes from box quality, score calibration, or semantic matching, and qualify the claim of improved novel-category representations accordingly.

1 comment

Thank you for the detailed responses and additional experiments. The new results address several of my questions and strengthen the evidence for the method’s effectiveness. However, I see cross-architecture generality as an important part of its potential value, and experiments on DINO / ConvNeXt-L alone do not establish this. I will therefore maintain my original rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.