acceptodds
Withdrawn from ICLR 2027

TFAG-OVD: Training-Free Attribute-Guided Open-Vocabulary Detection via LLM Semantic Composition

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Currently, open-vocabulary detectors recognize novel concepts primarily by aligning region features with category-name embeddings. However, as the vocabulary expands to thousands of categories, semantically related names crowd the alignment space, making the top match uncertain even at high similarity and disproportionately harming rare or weakly grounded concepts. Therefore, we rethink recognition as attribute-guided object understanding, representing ambiguous regions through transferable, class-agnostic attributes whose compositions can describe objects across datasets and vocabularies. A framework termed Training-Free Attribute-Guided Open-Vocabulary Detection via LLM Semantic Composition (TFAG-OVD) is proposed around a reusable pool of 3,549 class-agnostic attributes. Specifically, a frozen detector produces grounded proposals, and distribution-aware verification preserves clear predictions while routing ambiguous ones for attribute-guided review. For each routed region, a frozen vision-language recognizer retrieves shape, material, part, surface, and state cues; a text-only large language model then composes them within a task-relevant category range to revise the label. This selective attribute interface converts rigid region-to-name matching into evidence-based reasoning while keeping all models frozen. On LVIS minival, TFAG-OVD improves AP from 41.4 to 43.8 and rare-category AP from 34.1 to 37.8 over the frozen detector baseline. Zero-shot results on LVIS val, referring-expression benchmarks, ODinW, COCO, and COCO-O further demonstrate transfer across taxonomies and domains without parameter updates.

Primary area:
causal reasoning
Keywords:
open-vocabulary detection, zero-shot object detection, attribute-guided object understanding, vision-language models, large language models, semantic composition, long-tailed recognition, training-free inference

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_DsWyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TFAG-OVD, a training-free framework for open-vocabulary object detection that addresses ambiguous category predictions through attribute-guided semantic reasoning. The method builds on a frozen MM-GroundingDINO detector and introduces a distribution-aware verification mechanism that combines detector scores with CLIP similarities to identify uncertain predictions. For ambiguous regions, a frozen OpenCLIP model retrieves relevant visual attributes from a predefined pool of 3,549 class-agnostic attributes, including shape, material, parts, surface, and state. A text-only LLM then reasons over these attributes and a restricted set of candidate categories to revise the predicted label. All model parameters remain frozen, and the framework supports different inference vocabularies without additional training. Experiments on LVIS, referring-expression benchmarks, ODinW, COCO, and COCO-O demonstrate improvements over the MM-GroundingDINO baseline. In particular, TFAG-OVD improves LVIS minival AP from 41.4 to 43.8 and rare-category AP from 34.1 to 37.8, with additional gains across several transfer benchmarks.

Critical strengths
  • The paper addresses a meaningful limitation of conventional open-vocabulary detection: confusion among semantically related categories in large vocabularies. The motivation for using transferable visual attributes to distinguish ambiguous category predictions is intuitive and practically relevant, particularly for rare and fine-grained categories.

  • The training-free formulation is attractive. By combining a frozen detector, visual attribute recognition, and language-based category reasoning, the method improves recognition without requiring additional detector training. The selective routing mechanism is also conceptually reasonable because it attempts to preserve reliable predictions while applying additional semantic reasoning only to ambiguous cases.

  • The empirical results are encouraging across multiple benchmarks. The improvements on LVIS are particularly notable for rare categories, with APr increasing from 34.1 to 37.8 on minival and from 23.6 to 32.0 on val. The evaluation also covers referring-expression localization, cross-domain detection, and distribution shifts, providing useful evidence that the framework can operate across different category vocabularies.

  • The paper includes useful component ablations and hyperparameter analyses. The results demonstrate complementary contributions from CLIP verification and attribute-guided language reasoning. The experiments on score fusion, ambiguity thresholds, and candidate-set sizes also provide insights into the sensitivity of the overall framework.

Critical weaknesses
  • The technical novelty appears relatively limited, as the proposed framework primarily integrates existing pretrained components and established post-processing techniques. The system combines MM-GroundingDINO for object detection, CLIP for category verification, OpenCLIP for attribute retrieval, and a frozen LLM for category selection. Attribute-based recognition and compositional attribute modeling have already been studied in prior work, including OVAD and ComCa, while LLM-assisted classification and semantic reasoning are also well-established approaches. The distribution-aware routing mechanism provides an additional heuristic for selecting ambiguous proposals, but its formulation is relatively straightforward. Although the overall integration is reasonable, it remains unclear whether the paper introduces a sufficiently novel methodological contribution beyond assembling existing components into a sequential inference pipeline. Stronger comparisons with simpler alternatives, such as direct CLIP-based re-ranking, attribute-category similarity matching without an LLM, and direct LLM classification using candidate labels, would help establish the necessity of the proposed design.

  • The ablation studies do not sufficiently isolate the contribution of attribute-guided semantic composition. In Table 5, adding CLIP verification and proposal consolidation already improves LVIS AP from 41.4 to 43.3, whereas the complete framework reaches 43.8. Thus, the additional gain from the attribute–LLM branch under this configuration is only 0.5 AP, despite being presented as the central contribution. Moreover, the paper evaluates attribute recognition and LLM reasoning as a coupled component, making it impossible to determine whether the improvement comes from meaningful attribute composition or simply the LLM's prior knowledge about the candidate categories. More informative ablations would include LLM reasoning without attributes, attribute-based classification without an LLM, direct image-crop classification using a VLM, and random or shuffled attribute evidence. These experiments are particularly important for validating the claimed advantage of compositional attribute reasoning.

  • The computational cost and practical scalability of the training-free approach are insufficiently evaluated. Although the method requires no parameter updates, its inference pipeline involves multiple substantial models, including MM-GroundingDINO, CLIP ViT-L/14, OpenCLIP ViT-B/32, and Qwen3.5-9B. Ambiguous proposals additionally require attribute retrieval over 3,549 candidates and LLM-based category reasoning. This potentially introduces significant inference latency and memory overhead, especially for crowded scenes or large vocabularies where many proposals may be routed for review. The paper does not report FPS, wall-clock inference time, GPU memory, average LLM calls per image, or the percentage of proposals requiring additional reasoning. Without these measurements, it is difficult to judge whether the relatively modest AP improvements justify the computational overhead. The authors should provide an end-to-end efficiency comparison against the frozen detector and analyze the trade-off between accuracy, routing frequency, and inference cost.

  • The experimental evidence for the proposed ambiguity detection and category revision mechanisms is incomplete. The paper motivates distribution-aware verification by arguing that semantically related categories produce crowded similarity distributions, but does not directly evaluate whether its ambiguity score reliably identifies incorrect predictions. For instance, it would be useful to measure the error-detection precision and recall of the routing mechanism, as well as the relationship between ambiguity scores and actual classification errors. Similarly, the paper does not report how frequently the LLM changes the detector's predictions, how often these changes are correct, or how many initially correct predictions become incorrect after revision. Since the LLM is restricted to a detector-derived top-k category set, an analysis of candidate-set recall and failure cases where the correct category is absent would also be valuable. Such evaluations would provide stronger evidence for the claimed semantic reasoning mechanism rather than relying primarily on downstream AP improvements.

  • The evaluation protocol and generalization claims require further clarification. The framework uses category definitions, candidate vocabulary restrictions, and a rare-category calibration factor, but the paper does not sufficiently explain how rare categories are identified for calibration without relying on benchmark-specific category-frequency information. This is important for evaluating the claimed transferability to arbitrary vocabularies. Furthermore, some improvements on the additional benchmarks are relatively modest: ODinW13 improves by 0.6 AP, ODinW35 by 1.4 AP, COCO by 0.1 AP, and COCO-O by 0.7 AP over MM-GroundingDINO. These results support the method's applicability across benchmarks, but provide weaker evidence for substantial improvements outside long-tailed LVIS recognition. Since the reported results are single-run measurements, statistical variability and sensitivity to different LLM backends should also be investigated.

Questions for authors
  1. Can the authors provide an ablation that separates attribute recognition from LLM reasoning? In particular, how does the proposed method compare with direct LLM-based category selection without attributes, attribute-category similarity matching without an LLM, and direct VLM-based classification of ambiguous crops?

  2. What is the actual inference overhead of TFAG-OVD compared with the frozen MM-GroundingDINO baseline? Please report FPS, GPU memory, average number of reviewed proposals or LLM calls per image, and the breakdown of inference time across the different components.

  3. How effectively does the distribution-aware ambiguity score identify incorrect detector predictions? Can the authors provide routing precision/recall, the proportion of predictions selected for review, and statistics on correct versus incorrect LLM label revisions?

  4. How are rare categories identified for the proposed score calibration, particularly when transferring to datasets with different vocabularies? Does this procedure require benchmark-specific category-frequency metadata, and how much does rare-category calibration independently contribute to the reported LVIS improvements?

  5. How sensitive is the framework to the choice of LLM and the quality of retrieved attributes? Additionally, Appendix Table 10 reports higher AP at 20 candidates than at the selected 15 candidates, while the reproducibility note distinguishes the sensitivity sweep from the main run. Can the authors clarify these differences and explain the selection of the default operating point?

Reviewer_qB1WOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes TFAG-OVD, a training-free framework for open-vocabulary object detection that aims to resolve category ambiguity through attribute-guided reasoning. It combines a frozen MM-GroundingDINO detector, CLIP-based category verification, and a distribution-aware mechanism to identify ambiguous predictions. For these predictions, OpenCLIP retrieves visual attributes from a pool of 3,549 class-agnostic attributes, which are then processed by a frozen text-only LLM to revise the predicted category.

Experiments on LVIS, RefCOCO, ODinW, COCO, and COCO-O demonstrate improvements over the baseline detector, particularly for rare categories on LVIS.

Critical strengths
  1. Meaningful problem formulation. The paper addresses category ambiguity in large-vocabulary object detection. Using visual attributes to distinguish semantically similar categories is a reasonable and potentially useful direction.

  2. Broad experimental evaluation. The method is evaluated across multiple benchmarks without additional parameter training. The improvements on LVIS, particularly for rare categories, suggest that attribute-guided recognition can provide complementary information to conventional category-name matching.

Critical weaknesses
  1. Limited novelty and insufficient justification for the proposed architecture. The method mainly combines existing components, including CLIP-based attribute recognition, heuristic uncertainty estimation, and LLM-based classification. More importantly, the necessity of converting visual information into textual attributes before feeding it into a large language model is unclear. Existing multimodal models, such as Qwen-VL, can directly analyze cropped images and select among candidate categories. Without comparisons against such simpler alternatives, the advantage of the proposed attribute-to-LLM pipeline remains unconvincing.

  2. Insufficient evidence of effectiveness relative to computational cost. In Table 5, combining NMS and CLIP verification already achieves 43.3 AP, while the complete framework achieves 43.8 AP. Thus, the additional attribute and LLM branch provides only a modest improvement. Furthermore, the method introduces multiple large pretrained models, including a 9B-parameter LLM, but does not provide a systematic analysis of inference latency, computational overhead, or the proportion of predictions requiring LLM inference. These omissions make it difficult to assess whether the additional complexity is justified by the observed improvements.

Questions for authors
  1. Necessity of the LLM-based pipeline: Why does the proposed framework use a text-only LLM rather than directly applying a frozen multimodal model, such as Qwen-VL, to classify ambiguous image regions? Could the authors provide a comparison under similar computational budgets?

  2. Contribution of attribute-based reasoning: Could the authors compare against simpler alternatives, such as CLIP-based category re-ranking using attribute descriptions, or LLM classification using candidate category names without retrieved attributes? These comparisons would help establish whether explicit attribute reasoning is necessary.

  3. Inference efficiency: What are the inference latency, GPU memory requirements, and average number of LLM calls per image? What proportion of predictions is routed to the LLM, and how does this affect the overall computational cost?

  4. Additional experimental analysis: Could the authors evaluate the ambiguity-selection mechanism against simpler criteria, such as confidence thresholds or Top-1/Top-2 margins? Additionally, how much of the rare-category improvement comes from the attribute-guided branch versus the separate rare-category score calibration?

Reviewer_ZGxbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TFAG-OVD, a training-free open-vocabulary detection framework that uses CLIP to retrieve visual attributes and a text-only LLM to refine ambiguous category predictions. Experiments on LVIS and several other benchmarks demonstrate improvements, particularly for rare categories.

Critical strengths
  1. The paper presents an interesting training-free framework that combines attribute recognition with LLM reasoning to improve open-vocabulary detection.
  2. The method achieves encouraging improvements across multiple benchmarks, especially on rare categories in LVIS.
Critical weaknesses
  1. It is unclear whether the LLM actually relies on the retrieved visual attributes or mainly uses its prior world knowledge.
  2. The LLM only receives textual descriptions and no visual information, its reasoning is limited by the quality of CLIP retrieved attributes.
Questions for authors
  1. Why select the top-15 attributes rather than the most discriminative attributes?
  2. More direct evaluation of fine-grained attribute recognition would strengthen the paper, like FG-OVD comparisons with relevant methods such as GUIDED, SRA-Det, and HA-FGOVD.
  3. CLIP may struggle to recognize fine-grained attribute, why not use the LLM to generate the attributes for distinguishing candidate categories?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.