- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates higher-level category detection in open-vocabulary object detection (OVD), where a detector must identify diverse concrete subclasses using abstract category queries. The authors construct hierarchy-annotated benchmarks based on COCO, LVIS-mini, and Objects365, together with three evaluation protocols (SS, US, and UU). They further propose HARE, which employs hierarchy-supervised routing to select LoRA experts according to the semantic granularity of category queries. Experiments demonstrate improvements over existing OVD methods on higher-level category detection while largely preserving concrete-category detection performance.
Critical strengths
-
Meaningful problem setting and benchmark construction. The paper addresses the practical challenge of detecting visually diverse objects using higher-level semantic queries. The proposed hierarchical annotations and SS/US/UU evaluation protocols provide a useful framework for studying generalization across semantic granularities.
-
Consistent empirical improvements. HARE achieves improvements over Grounding DINO across three benchmarks, particularly under direct higher-level queries. The comparisons with different adaptation strategies and routing objectives provide supporting evidence for the effectiveness of hierarchy-supervised adaptation.
Critical weaknesses
-
Insufficient validation of benchmark reliability. The hierarchical annotations are constructed by propagating existing bounding boxes to ancestor categories. However, this may introduce incomplete annotations when relevant subclasses are not labeled in the original datasets. Furthermore, the semantic hierarchies are primarily generated and cleaned using GPT-5, without a systematic quantitative assessment of their correctness. These issues raise concerns about the reliability of the constructed benchmarks and the resulting evaluation.
-
Limited methodological novelty and insufficient analysis of generalization. HARE mainly combines existing LoRA experts, routing mechanisms, and a hierarchy-ranking loss, resulting in relatively limited methodological novelty. Although the US protocol evaluates unseen subclasses, the paper lacks a detailed analysis of how well the method generalizes to visually diverse unseen subclasses. Additionally, the impact of semantic abstraction depth on detection performance remains unexplored, despite being central to the paper's motivation.
Questions for authors
-
Benchmark reliability: How do the authors ensure annotation completeness when propagating existing bounding boxes to higher-level categories, particularly when some relevant subclasses are not annotated in the original datasets? Have the LLM-generated semantic hierarchies been systematically validated for correctness?
-
Generalization to unseen subclasses: If only cats and dogs are used to adapt the animal category, can the model reliably detect unseen subclasses such as horses and giraffes using the query animal? Could the authors provide per-subclass results and analyze how the diversity of seen subclasses affects generalization?
-
Evaluation across semantic levels: The constructed hierarchies contain multiple semantic levels (with a maximum depth of 9), but the paper lacks a systematic evaluation at different abstraction depths. Does HARE consistently improve detection across these levels? Furthermore, how does the benchmark handle existing parent–child relationships among original dataset categories (e.g., Labrador → dog) when all original categories are treated as leaf nodes?
-
Necessity of learned routing: Could the authors compare HARE with simpler hierarchy-based expert assignment strategies, such as fixed level-to-expert mappings, to better demonstrate the necessity and effectiveness of the proposed learned routing mechanism?
7 comments
W2-1: Methodological Novelty
Our contribution centers on a new research problem and an initial methodological exploration tailored to it. To the best of our knowledge, this work provides the first systematic study of higher-level category detection using only higher-level category queries, together with corresponding hierarchy-annotated benchmarks and evaluation protocols. Within this task, we also make the first attempt to use hierarchy-supervised routing of LoRA experts, incorporating an identity pathway to preserve concrete-category detection during adaptation. LoRA, routing, and ranking losses are established techniques; our methodological contribution is the hierarchy-aware conditional adaptation framework designed for this problem.
Task formulation. An open-vocabulary detector's ability to recognize a dog does not imply that it will detect the same instance when queried with animal. Our task requires localizing visually diverse concrete subclasses using their shared higher-level category as the query. We construct hierarchy-annotated benchmarks and use SS, US, and UU to distinguish category visibility during adaptation: SS uses seen concrete categories and seen ancestors; US uses unseen concrete subclasses and seen ancestors; and UU uses unseen concrete categories and unseen ancestors. Visibility is defined across both router training and detector adaptation. This task formulation motivates the design of HARE.
Hierarchy-supervised allocation of adaptation parameters. HARE uses a hierarchy-ranking objective to learn relative abstraction order and guides expert assignment through abstraction scores defined as expectations under the routing distribution. It requires neither absolute granularity labels nor predefined category-to-expert assignments. In an ablation with the same architecture and capacity, replacing the load-balancing supervision in Balance with the hierarchy-ranking objective increases average box AP under direct higher-level category queries on COCO from 20.9 to 24.3, with gains of 2.4 AP on US and 3.2 AP on UU. These results support the contribution of hierarchy supervision to expert routing.
Identity pathway. HARE includes an identity expert with , allowing each query to select either a low-rank update or the original projection. The complete HARE model improves higher-level category detection while maintaining COCO concrete-category AP at 56.9, matching Base.
W2-2: Generalization to Unseen Subclasses
Using only higher-level category queries, without auxiliary subclass names, HARE improves US AP by 10.2, 4.9, and 5.5 AP on COCO, LVIS-mini, and Objects365, respectively. Per-subclass diagnostics show recall gains of 12.7, 10.4, 4.6, and 2.6 percentage points for unseen airplane, motorcycle, cell phone, and laptop instances, respectively. These results provide evidence of positive transfer across multiple semantic categories.
W2-3: Detection across Abstraction Depths
For this additional analysis, we define structural height as the longest distance from a concept to a terminal descendant in the merged hierarchy graph. We keep the original predictions, annotations, and evaluation procedure unchanged and group results into , , and . HARE improves average US AP by 4.3, 17.1, and 32.2 AP in these groups, respectively, showing gains across multiple abstraction ranges.
SS / US detection results grouped by structural height
| Structural height | SS: Base | SS: HARE | Gain | US: Base | US: HARE | Gain |
|---|---|---|---|---|---|---|
| h=1–3 | 16.6 | 27.2 | +10.6 | 17.8 | 22.2 | +4.3 |
| h=4–6 | 26.7 | 38.1 | +11.4 | 19.1 | 36.2 | +17.1 |
| h=7–9 | 6.9 | 44.6 | +37.7 | 3.9 | 36.1 | +32.2 |
Q2: Generalization to Unseen Subclasses
(1) Per-subclass results under higher-level category queries. We evaluate recall for descendant subclasses excluded from adaptation, using only their higher-level category as the query. With animal as the query, HARE achieves recall of 38.8% for bird and 93.8% for cat, improving over Base by 3.4 and 2.1 percentage points, respectively.
(2) Transfer across multiple higher-level categories. With vehicle as the query, recall increases from 28.6% to 41.3% for airplane and from 29.2% to 39.6% for motorcycle. With electronic device as the query, recall improves by 4.6 percentage points for cell phone and 2.6 percentage points for laptop. These results show that adaptation benefits can transfer to descendant subclasses unseen during adaptation, without requiring their concrete-category names at inference.
Recall for subclasses unseen during adaptation under higher-level category queries
| Higher-level category query | Unseen subclass | GT instances | Base recall (%) | HARE recall (%) | Gain (percentage points) |
|---|---|---|---|---|---|
| animal | bird | 147 | 35.4 | 38.8 | +3.4 |
| animal | cat | 48 | 91.7 | 93.8 | +2.1 |
| vehicle | airplane | 63 | 28.6 | 41.3 | +12.7 |
| vehicle | motorcycle | 96 | 29.2 | 39.6 | +10.4 |
| electronic device | cell phone | 88 | 22.7 | 27.3 | +4.6 |
| electronic device | laptop | 117 | 63.3 | 65.8 | +2.6 |
(3) Seen-subclass diversity may broaden visual coverage. Diverse seen subclasses provide richer visual supervision for a shared ancestor concept: differences in shape, appearance, and context associate the same ancestor label with a broader range of instances. The relevant factors include the number of seen subclasses, the visual variation they cover, and how that variation relates to unseen subclasses. For example, vehicle queries yield recall gains of 12.7 and 10.4 percentage points for unseen airplane and motorcycle instances, while electronic device queries yield gains of 4.6 and 2.6 percentage points for unseen cell phone and laptop instances. These results demonstrate transfer to unseen descendants using only higher-level category queries.
Q3: Evaluation across Semantic Levels
We thank the reviewer for this suggestion. We have added detection results at individual structural heights and clarified the role of original dataset categories in hierarchy construction.
(1) Detection gains across structural heights. For this additional analysis on COCO, structural height is defined as the longest distance from a concept to a terminal descendant in the merged hierarchy graph. We evaluate , assigning each query to a unique height and averaging box AP equally over the valid query–split pairs at that height. The original predictions, annotations, and AP computation remain unchanged; this analysis groups the existing evaluation results by structural height.
HARE improves AP by 4.5–16.2 AP at and by 21.2–48.0 AP at . At , AP decreases by 1.4 AP. The gains therefore span multiple abstraction depths, with variation across individual heights.
COCO detection results at each structural height
| Structural height h | Base AP | HARE AP | Gain |
|---|---|---|---|
| 1 | 13.5 | 22.5 | +9.0 |
| 2 | 17.5 | 24.4 | +6.9 |
| 3 | 20.3 | 24.8 | +4.5 |
| 4 | 26.6 | 42.7 | +16.2 |
| 5 | 21.8 | 20.4 | −1.4 |
| 6 | 14.2 | 53.8 | +39.6 |
| 7 | 3.7 | 51.8 | +48.0 |
| 8 | 4.7 | 40.3 | +35.6 |
| 9 | 7.8 | 28.9 | +21.2 |
(2) Original categories can become internal nodes after hierarchy merging. We construct concrete-to-abstract semantic paths starting from the original dataset categories, then merge concepts with the same normalized name across paths into a single node. If an original category also appears as an ancestor on another category's path, it can become an internal node in the merged hierarchy, preserving the parent–child relation.
For example, if both shark and fish are original categories, merging their paths can produce:
shark → fish → aquatic vertebrate → vertebrate animal → animal → living organism → biological entity → physical object
Here, fish is both an original dataset category and an ancestor of shark, so it can be an internal node in the merged hierarchy.
We thank the reviewer for the constructive comments and respond point by point below.
We address this concern through three complementary checks: manual annotation correction, an edge-by-edge audit of the training hierarchy, and controlled missing-subclass annotation experiments.
(1) Manual annotation review and reevaluation. We randomly sampled 500 images from each of the SS, US, and UU splits of COCO, reviewed them image by image, and corrected incorrect and missing annotations. The overall correction rate was approximately 1.0%, indicating relatively high annotation quality in the audited sample. We then reevaluated Base and HARE on the same images using the corrected annotations.
The average box AP of Base/HARE changes from 16.9/25.6 with the original annotations to 17.1/25.7 with the corrected annotations, with HARE retaining an advantage of 8.7 AP. Thus, the corrections affect absolute AP but do not alter HARE's average performance advantage on this audited sample.
Detection results with manually corrected annotations (500 images per split)
(2) Edge-by-edge audit of the training hierarchy. We audited all 277 unique parent–child relations actually used in the Objects365 adaptation annotations, covering 121 original categories and 138 higher-level concepts. Two human annotators assessed the relations independently, supplemented by auxiliary assessments from GPT-6.1 and Doubao. Each assessment checked whether the child concept is a type of the parent concept is-a relation. All acceptance rates below use 277 relations as the denominator.
The human annotators accepted 89.5% and 91.0% of the relations, respectively, and the auxiliary model assessments yielded similar rates. These results indicate that most training hierarchy relations were accepted in the audit.
To quantify how frequently rejected relations occur in annotation propagation, we further weighted each relation by its actual number of occurrences along the hierarchy paths associated with training boxes. For each of the four assessments, we divided the total occurrence count of rejected relations by the occurrence count of all relations, then averaged the four ratios, obtaining 5.8%. This measures the relatively small share of propagation steps involving rejected relations. Together with the reevaluation above, these checks support the stability of the performance comparison.
(3) Sensitivity to controlled omissions of subclass annotations. We introduced controlled annotation omissions on the full existing test sets of COCO and LVIS-mini. Keeping the models, images, queries, and predictions fixed, we randomly retained 100.0%, 90.0%, 80.0%, 70.0%, or 60.0% of the annotated source subclasses for each higher-level category. Every retention ratio except 100.0% was evaluated with 10 random repetitions.
Evaluation with randomly omitted subclass annotations
Even when only 60.0% of source subclasses are retained, HARE achieves 15.6 AP on COCO and 9.5 AP on LVIS-mini, maintaining average advantages of 5.2 and 3.2 AP over Base, respectively. For every retention ratio and every random repetition on both datasets, HARE's average box AP across the three splits remains higher than Base's. Together with the reevaluation using manually corrected annotations, these results show that HARE's relative advantage persists under the tested conditions of reduced annotation completeness.