acceptodds
Under review as a conference paper at ICLR 2027

HARE: Hierarchy-Aware Routing of LoRA Experts for Open-Vocabulary Object Detection at Higher Semantic Levels

Abstract

Open-vocabulary object detectors perform strongly on concrete categories but often miss relevant instances when queried with higher-level concepts such as animal, vehicle, or furniture. We study higher-level category detection, which requires localizing instances from visually diverse concrete subclasses using their shared ancestor concept as the query. To support training and evaluation, we construct hierarchy-annotated versions of COCO, LVIS-mini, and Objects365 by associating each source category with a concrete-to-abstract semantic path and propagating its instance annotations to the concepts on that path without additional bounding-box annotation. We further propose Hierarchy-Aware Routing of Experts (HARE), a simple parameter-efficient adaptation framework that uses parent-child relations as explicit supervision for allocating adaptation parameters across category queries. Specifically, a ranking objective encourages parent concepts to receive higher abstraction scores than their children, guiding a text-conditioned router over lightweight LoRA experts. HARE is adapted only on Objects365 and evaluated on held-out Objects365 categories and the unseen COCO and LVIS-mini datasets. Under direct higher-level queries, HARE improves average box AP over Grounding DINO by 8.5, 5.1, and 4.2 points on COCO, LVIS-mini, and Objects365, respectively, while matching its concrete-category AP on COCO. Under identical subclass augmentation, HARE retains gains of 3.7, 2.3, and 1.1 points. These results demonstrate that explicit hierarchy supervision improves higher-level detection and transfers across datasets, with substantial gains even without auxiliary subclass cues at inference.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Open-Vocabulary Object Detection, Vision-Language Learning, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ALgsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates higher-level category detection in open-vocabulary object detection (OVD), where a detector must identify diverse concrete subclasses using abstract category queries. The authors construct hierarchy-annotated benchmarks based on COCO, LVIS-mini, and Objects365, together with three evaluation protocols (SS, US, and UU). They further propose HARE, which employs hierarchy-supervised routing to select LoRA experts according to the semantic granularity of category queries. Experiments demonstrate improvements over existing OVD methods on higher-level category detection while largely preserving concrete-category detection performance.

Critical strengths
  1. Meaningful problem setting and benchmark construction. The paper addresses the practical challenge of detecting visually diverse objects using higher-level semantic queries. The proposed hierarchical annotations and SS/US/UU evaluation protocols provide a useful framework for studying generalization across semantic granularities.

  2. Consistent empirical improvements. HARE achieves improvements over Grounding DINO across three benchmarks, particularly under direct higher-level queries. The comparisons with different adaptation strategies and routing objectives provide supporting evidence for the effectiveness of hierarchy-supervised adaptation.

Critical weaknesses
  1. Insufficient validation of benchmark reliability. The hierarchical annotations are constructed by propagating existing bounding boxes to ancestor categories. However, this may introduce incomplete annotations when relevant subclasses are not labeled in the original datasets. Furthermore, the semantic hierarchies are primarily generated and cleaned using GPT-5, without a systematic quantitative assessment of their correctness. These issues raise concerns about the reliability of the constructed benchmarks and the resulting evaluation.

  2. Limited methodological novelty and insufficient analysis of generalization. HARE mainly combines existing LoRA experts, routing mechanisms, and a hierarchy-ranking loss, resulting in relatively limited methodological novelty. Although the US protocol evaluates unseen subclasses, the paper lacks a detailed analysis of how well the method generalizes to visually diverse unseen subclasses. Additionally, the impact of semantic abstraction depth on detection performance remains unexplored, despite being central to the paper's motivation.

Questions for authors
  1. Benchmark reliability: How do the authors ensure annotation completeness when propagating existing bounding boxes to higher-level categories, particularly when some relevant subclasses are not annotated in the original datasets? Have the LLM-generated semantic hierarchies been systematically validated for correctness?

  2. Generalization to unseen subclasses: If only cats and dogs are used to adapt the animal category, can the model reliably detect unseen subclasses such as horses and giraffes using the query animal? Could the authors provide per-subclass results and analyze how the diversity of seen subclasses affects generalization?

  3. Evaluation across semantic levels: The constructed hierarchies contain multiple semantic levels (with a maximum depth of 9), but the paper lacks a systematic evaluation at different abstraction depths. Does HARE consistently improve detection across these levels? Furthermore, how does the benchmark handle existing parent–child relationships among original dataset categories (e.g., Labrador → dog) when all original categories are treated as leaf nodes?

  4. Necessity of learned routing: Could the authors compare HARE with simpler hierarchy-based expert assignment strategies, such as fixed level-to-expert mappings, to better demonstrate the necessity and effectiveness of the proposed learned routing mechanism?

7 comments

We thank the reviewer for the constructive comments and respond point by point below.

W1:&Q1 Quality of Hierarchical Annotations

We address this concern through three complementary checks: manual annotation correction, an edge-by-edge audit of the training hierarchy, and controlled missing-subclass annotation experiments.

(1) Manual annotation review and reevaluation. We randomly sampled 500 images from each of the SS, US, and UU splits of COCO, reviewed them image by image, and corrected incorrect and missing annotations. The overall correction rate was approximately 1.0%, indicating relatively high annotation quality in the audited sample. We then reevaluated Base and HARE on the same images using the corrected annotations.

The average box AP of Base/HARE changes from 16.9/25.6 with the original annotations to 17.1/25.7 with the corrected annotations, with HARE retaining an advantage of 8.7 AP. Thus, the corrections affect absolute AP but do not alter HARE's average performance advantage on this audited sample.

Detection results with manually corrected annotations (500 images per split)

MethodSS APUS APUU APAvg.
Base18.317.415.617.1
HARE32.127.717.525.7

(2) Edge-by-edge audit of the training hierarchy. We audited all 277 unique parent–child relations actually used in the Objects365 adaptation annotations, covering 121 original categories and 138 higher-level concepts. Two human annotators assessed the relations independently, supplemented by auxiliary assessments from GPT-6.1 and Doubao. Each assessment checked whether the child concept is a type of the parent concept is-a relation. All acceptance rates below use 277 relations as the denominator.

Assessment sourceAcceptedRejectedAcceptance rate
Human annotator 12482989.5%
Human annotator 22522591.0%
GPT-6.1 auxiliary assessment2473089.2%
Doubao auxiliary assessment2492889.9%

The human annotators accepted 89.5% and 91.0% of the relations, respectively, and the auxiliary model assessments yielded similar rates. These results indicate that most training hierarchy relations were accepted in the audit.

To quantify how frequently rejected relations occur in annotation propagation, we further weighted each relation by its actual number of occurrences along the hierarchy paths associated with training boxes. For each of the four assessments, we divided the total occurrence count of rejected relations by the occurrence count of all relations, then averaged the four ratios, obtaining 5.8%. This measures the relatively small share of propagation steps involving rejected relations. Together with the reevaluation above, these checks support the stability of the performance comparison.

(3) Sensitivity to controlled omissions of subclass annotations. We introduced controlled annotation omissions on the full existing test sets of COCO and LVIS-mini. Keeping the models, images, queries, and predictions fixed, we randomly retained 100.0%, 90.0%, 80.0%, 70.0%, or 60.0% of the annotated source subclasses for each higher-level category. Every retention ratio except 100.0% was evaluated with 10 random repetitions.

Evaluation with randomly omitted subclass annotations

DatasetSubclass retention ratioBase APHARE APGain (AP)
COCO100.0%15.824.3+8.6
COCO90.0%15.1±0.423.3±0.4+8.2±0.1
COCO80.0%13.8±0.420.9±0.3+7.1±0.4
COCO70.0%11.2±0.617.1±0.6+5.9±0.6
COCO60.0%10.3±0.815.6±0.9+5.2±0.7
LVIS-mini100.0%10.015.0+5.1
LVIS-mini90.0%9.3±0.314.1±0.5+4.8±0.3
LVIS-mini80.0%8.8±0.413.3±0.6+4.5±0.3
LVIS-mini70.0%7.0±0.710.4±0.5+3.4±0.4
LVIS-mini60.0%6.3±0.69.5±0.5+3.2±0.4

Even when only 60.0% of source subclasses are retained, HARE achieves 15.6 AP on COCO and 9.5 AP on LVIS-mini, maintaining average advantages of 5.2 and 3.2 AP over Base, respectively. For every retention ratio and every random repetition on both datasets, HARE's average box AP across the three splits remains higher than Base's. Together with the reevaluation using manually corrected annotations, these results show that HARE's relative advantage persists under the tested conditions of reduced annotation completeness.

W2-1: Methodological Novelty

Our contribution centers on a new research problem and an initial methodological exploration tailored to it. To the best of our knowledge, this work provides the first systematic study of higher-level category detection using only higher-level category queries, together with corresponding hierarchy-annotated benchmarks and evaluation protocols. Within this task, we also make the first attempt to use hierarchy-supervised routing of LoRA experts, incorporating an identity pathway to preserve concrete-category detection during adaptation. LoRA, routing, and ranking losses are established techniques; our methodological contribution is the hierarchy-aware conditional adaptation framework designed for this problem.

Task formulation. An open-vocabulary detector's ability to recognize a dog does not imply that it will detect the same instance when queried with animal. Our task requires localizing visually diverse concrete subclasses using their shared higher-level category as the query. We construct hierarchy-annotated benchmarks and use SS, US, and UU to distinguish category visibility during adaptation: SS uses seen concrete categories and seen ancestors; US uses unseen concrete subclasses and seen ancestors; and UU uses unseen concrete categories and unseen ancestors. Visibility is defined across both router training and detector adaptation. This task formulation motivates the design of HARE.

Hierarchy-supervised allocation of adaptation parameters. HARE uses a hierarchy-ranking objective to learn relative abstraction order and guides expert assignment through abstraction scores defined as expectations under the routing distribution. It requires neither absolute granularity labels nor predefined category-to-expert assignments. In an ablation with the same architecture and capacity, replacing the load-balancing supervision in Balance with the hierarchy-ranking objective increases average box AP under direct higher-level category queries on COCO from 20.9 to 24.3, with gains of 2.4 AP on US and 3.2 AP on UU. These results support the contribution of hierarchy supervision to expert routing.

Identity pathway. HARE includes an identity expert with ΔW0=0\Delta W_0=0, allowing each query to select either a low-rank update or the original projection. The complete HARE model improves higher-level category detection while maintaining COCO concrete-category AP at 56.9, matching Base.

W2-2: Generalization to Unseen Subclasses

Using only higher-level category queries, without auxiliary subclass names, HARE improves US AP by 10.2, 4.9, and 5.5 AP on COCO, LVIS-mini, and Objects365, respectively. Per-subclass diagnostics show recall gains of 12.7, 10.4, 4.6, and 2.6 percentage points for unseen airplane, motorcycle, cell phone, and laptop instances, respectively. These results provide evidence of positive transfer across multiple semantic categories.

W2-3: Detection across Abstraction Depths

For this additional analysis, we define structural height hh as the longest distance from a concept to a terminal descendant in the merged hierarchy graph. We keep the original predictions, annotations, and evaluation procedure unchanged and group results into h=1–3h=1\text{–}3, 4–64\text{–}6, and 7–97\text{–}9. HARE improves average US AP by 4.3, 17.1, and 32.2 AP in these groups, respectively, showing gains across multiple abstraction ranges.

SS / US detection results grouped by structural height

Structural heightSS: BaseSS: HAREGainUS: BaseUS: HAREGain
h=1–316.627.2+10.617.822.2+4.3
h=4–626.738.1+11.419.136.2+17.1
h=7–96.944.6+37.73.936.1+32.2

Q2: Generalization to Unseen Subclasses

(1) Per-subclass results under higher-level category queries. We evaluate recall for descendant subclasses excluded from adaptation, using only their higher-level category as the query. With animal as the query, HARE achieves recall of 38.8% for bird and 93.8% for cat, improving over Base by 3.4 and 2.1 percentage points, respectively.

(2) Transfer across multiple higher-level categories. With vehicle as the query, recall increases from 28.6% to 41.3% for airplane and from 29.2% to 39.6% for motorcycle. With electronic device as the query, recall improves by 4.6 percentage points for cell phone and 2.6 percentage points for laptop. These results show that adaptation benefits can transfer to descendant subclasses unseen during adaptation, without requiring their concrete-category names at inference.

Recall for subclasses unseen during adaptation under higher-level category queries

Higher-level category queryUnseen subclassGT instancesBase recall (%)HARE recall (%)Gain (percentage points)
animalbird14735.438.8+3.4
animalcat4891.793.8+2.1
vehicleairplane6328.641.3+12.7
vehiclemotorcycle9629.239.6+10.4
electronic devicecell phone8822.727.3+4.6
electronic devicelaptop11763.365.8+2.6

(3) Seen-subclass diversity may broaden visual coverage. Diverse seen subclasses provide richer visual supervision for a shared ancestor concept: differences in shape, appearance, and context associate the same ancestor label with a broader range of instances. The relevant factors include the number of seen subclasses, the visual variation they cover, and how that variation relates to unseen subclasses. For example, vehicle queries yield recall gains of 12.7 and 10.4 percentage points for unseen airplane and motorcycle instances, while electronic device queries yield gains of 4.6 and 2.6 percentage points for unseen cell phone and laptop instances. These results demonstrate transfer to unseen descendants using only higher-level category queries.

Q3: Evaluation across Semantic Levels

We thank the reviewer for this suggestion. We have added detection results at individual structural heights and clarified the role of original dataset categories in hierarchy construction.

(1) Detection gains across structural heights. For this additional analysis on COCO, structural height hh is defined as the longest distance from a concept to a terminal descendant in the merged hierarchy graph. We evaluate h=1,…,9h=1,\ldots,9, assigning each query to a unique height and averaging box AP equally over the valid query–split pairs at that height. The original predictions, annotations, and AP computation remain unchanged; this analysis groups the existing evaluation results by structural height.

HARE improves AP by 4.5–16.2 AP at h=1–4h=1\text{–}4 and by 21.2–48.0 AP at h=6–9h=6\text{–}9. At h=5h=5, AP decreases by 1.4 AP. The gains therefore span multiple abstraction depths, with variation across individual heights.

COCO detection results at each structural height

Structural height hBase APHARE APGain
113.522.5+9.0
217.524.4+6.9
320.324.8+4.5
426.642.7+16.2
521.820.4−1.4
614.253.8+39.6
73.751.8+48.0
84.740.3+35.6
97.828.9+21.2

(2) Original categories can become internal nodes after hierarchy merging. We construct concrete-to-abstract semantic paths starting from the original dataset categories, then merge concepts with the same normalized name across paths into a single node. If an original category also appears as an ancestor on another category's path, it can become an internal node in the merged hierarchy, preserving the parent–child relation.

For example, if both shark and fish are original categories, merging their paths can produce:

shark → fish → aquatic vertebrate → vertebrate animal → animal → living organism → biological entity → physical object

Here, fish is both an original dataset category and an ancestor of shark, so it can be an internal node in the merged hierarchy.

Reviewer_9jBSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies higher-level category detection in OVOD, where detectors are queried with abstract ancestor concepts to localize concrete subclasses while preserving concrete-category performance. The authors construct a taxonomic hierarchy using GPT-5 over COCO, LVIS-mini, and Objects365 vocabularies, obtaining multi-granularity bounding-box annotations via label propagation along taxonomic paths. The paper proposes HARE, trained in two stages: (1) pretraining an MLP router with a margin-based ranking loss on continuous expected abstraction scores, and (2) freezing the detector and router while adapting rank-16 LoRA experts inserted into Grounding DINO's text pathway, reserving Expert 0 as an untuned identity pathway. The method is evaluated across Seen-Seen, Unseen-Seen, and Unseen-Unseen protocols under direct and subclass-augmented settings.

Critical strengths
  1. The paper targets a realistic problem in OVOD, handling user queries across varying degrees of semantic abstraction without requiring manual re-annotation of multi-granularity bounding boxes.
  2. Allocating an untuned identity pathway (Expert 0, ΔW0=0\Delta W_0 = 0) provides a simple, clean, and parameter-efficient way to retain the detector's base capabilities on concrete categories.
Critical weaknesses
  1. Unclear Root Cause of High-Level Query Degradation and Weak Motivation for Hierarchy-Aware Routing: The paper observes an empirical performance drop when querying higher-level categories compared to concrete categories, but fails to provide a principled explanation for why OVOD models inherently struggle with abstract queries (e.g., intra-class visual variance, semantic drift in cross-modal attention, or text representation dilution). Because the underlying bottleneck is merely stated rather than analyzed, it remains theoretically unsubstantiated why routing LoRA experts based on hierarchical abstraction specifically resolves this degradation mechanism.
  2. Ambiguity in Expert Count vs. Taxonomy Depth and Missing Visual Verification: The mapping between the taxonomic depth and the number of experts remains ambiguous. The GPT-5-generated paths have category-dependent lengths (LcL_c) rather than a uniform depth, whereas the expert pool is fixed to a predetermined constant (E=10E=10). The paper does not clarify how these differing depths map to a non-one-to-one expert space. More critically, the authors provide no visualization (e.g., category assignment histograms or concept clouds) showing which specific concepts are routed to which experts. While the relative transition matrix (Figure 5b) shows ordering consistency, it does not confirm whether distinct experts have truly learned meaningful, well-separated semantic granularities, or whether certain intermediate experts suffer from allocation collapse.
Questions for authors
  1. Underlying Mechanism: Could the authors elaborate on the theoretical or representational reasons why pretrained OVOD detectors degrade on abstract concepts? What empirical evidence demonstrates that hierarchy-guided LoRA adaptation directly alleviates that specific representational bottleneck?
  2. Concept-to-Expert Occupancy Visualization: Can the authors provide an explicit concept assignment breakdown (e.g., a distribution plot or frequency table) showing which concepts and taxonomy levels are allocated to Expert 0 through Expert 9? Are concrete leaf categories exclusively absorbed by Expert 0?
  3. Saturation Point of Expert Count: Why was the evaluation of EE restricted to 6,8,106, 8, 10 in Table 7? What happens if EE is increased further (e.g., E=12,16E=12, 16)? Does performance saturate or decline once EE significantly exceeds the average taxonomy depth?
  4. Clarification on Typographical Inaccuracies: Please rectify the typo on line 747 ("Eq. equation 7").
5 comments

We thank the reviewer for the constructive comments and respond point by point below.

W1 & Q1: Query Degradation and Routing Motivation

(1) Higher-level category queries must cover visually diverse concrete subclasses. A pretrained detector may match an image region to the concrete category airplane, but the linguistic relation “an airplane is a vehicle” does not directly constrain that region's matching score with vehicle. The query vehicle must also cover visually different subclasses such as motorcycle, boat, and train. We therefore propose a testable explanation: the pretrained visual–semantic alignment for a concrete-category query may transfer to a higher-level category query to different degrees across subclasses.

(2) Visual feature analysis before multimodal fusion. We extract GT region features before multimodal fusion and compare similarities within and between subclasses that share an ancestor. For the four vehicle subclasses below, the average within-subclass similarity is 0.641, compared with 0.493 between subclasses. For the four animal subclasses, the corresponding values are 0.655 and 0.520. These measurements indicate that a higher-level category query must match regions spanning visually diverse concrete subclasses.

GT region feature similarities among vehicle subclasses

Subclassmotorcycleairplaneboattrain
motorcycle0.6290.4410.4760.539
airplane0.4410.5780.4660.518
boat0.4760.4660.6280.516
train0.5390.5180.5160.727

GT region feature similarities among animal subclasses

Subclassbirdcatelephantsheep
bird0.4860.4570.4600.495
cat0.4570.7300.5690.547
elephant0.4600.5690.7070.591
sheep0.4950.5470.5910.695

(3) Query replacement on the same instances reveals uneven subclass coverage. We replace concrete-category queries with higher-level category queries while evaluating the same GT instances. For 63 airplane instances, Grounding DINO's recall drops from 76.2% with airplane to 28.6% with vehicle. HARE improves recall with vehicle to 41.3%, while recall with airplane remains 76.2%. For 96 motorcycle instances, HARE improves recall with vehicle from 29.2% to 39.6%, while recall under concrete-category queries remains 45.8%.

The pattern varies across subclasses. Base already achieves higher recall for elephants with animal than with the concrete-category query, while HARE slightly reduces recall for trains with vehicle. Thus, the observed difficulty is uneven subclass coverage under higher-level category queries, and HARE improves coverage for several of these subclasses.

Recall with concrete-category and higher-level category queries on the same instances

Concrete subclass - Higher-level categoryGT instancesBase: concrete-category queryBase: higher-level category queryHARE: concrete-category queryHARE: higher-level category query
motorcycle - vehicle9645.8%29.2%45.8%39.6%
airplane - vehicle6376.2%28.6%76.2%41.3%
boat - vehicle14615.8%6.9%18.5%8.2%
train - vehicle4885.4%66.7%87.5%64.6%
bird - animal14749.0%35.4%40.1%38.8%
cat - animal4895.8%91.7%95.8%93.8%
elephant - animal10123.8%45.5%23.8%61.4%
sheep - animal20612.1%22.8%29.1%21.4%

(4) Controlled comparison of hierarchy-supervised routing. Propagating instance labels along each concrete category's fixed semantic path provides higher-level category supervision from multiple concrete subclasses. To assess the additional contribution of hierarchy-aware routing, we compare Balance and HARE using the same propagated instance annotations, router architecture, LoRA expert configuration, and detection adaptation procedure. Only the router training objective differs: Balance uses load-balancing supervision, whereas HARE uses the hierarchy-ranking objective to encourage higher abstraction scores for parent categories than for their children.

HARE increases adjacent-pair consistency from 66.0% to 99.7% relative to Balance (Figure 5a). Under higher-level category queries on COCO, average box AP increases from 20.9 to 24.3 (+3.4 AP), with gains of 2.4 AP on US and 3.2 AP on UU. This comparison shows that parent–child supervision guides expert allocation according to relative abstraction order and improves generalization under the same detection supervision: US evaluates unseen concrete subclasses under seen ancestors, while UU evaluates cases where both the concrete category and its queried ancestor are unseen during adaptation.

W2 & Q2: Depth-to-Expert Mapping and Concept Assignments

(1) Routing is learned from relative abstraction order. HARE does not use a fixed mapping from hierarchy levels to expert indices. LcL_c denotes the number of upward steps along a category's fixed semantic path, while EE denotes the total number of routing experts, including the identity pathway, and determines the range of abstraction scores. Parent–child supervision constrains the continuous abstraction score, defined as an expectation under the routing distribution:

a(c)=∑e=0E−1(e+1)pc,e.a(c)=\sum_{e=0}^{E-1}(e+1)p_{c,e}.

During both detection adaptation and inference, Top-1 routing selects the expert with the highest routing probability:

e∗(c)=arg⁡max⁡epc,e.e^*(c)=\arg\max_{e}p_{c,e}.

Multiple hierarchy levels can therefore share an expert, and a semantic path can skip expert indices. Ordering the continuous abstraction scores does not directly guarantee strictly monotonic discrete expert assignments. For example, the 9-step path for dog yields the expert sequence 1 → 2 → 2 → 3 → 3 → 4 → 4 → 6 → 7 → 8: its 10 nodes use 7 distinct experts, illustrating that hierarchy levels and experts need not have a one-to-one correspondence.

(2) Concept assignments for all 259 adaptation training concepts. We encode each concept independently once and record its Top-1 expert assignment. The table reports counts of unique concepts assigned to each expert; these counts are distinct from expert selection frequencies over training or inference inputs.

ExpertConcrete categoriesHigher-level categoriesExample concepts
Expert 0 (identity)452elephant, aircraft
Expert 1705dog, bus
Expert 2646electric drill, domesticated mammal
Expert 3038mammal, road vehicle
Expert 407animal, living organism
Expert 5014food, vehicle
Expert 6022biological entity, tool
Expert 702animal-derived product, physical object
Expert 802tangible entity
Expert 900—

Concrete categories are assigned to Experts 0, 1, and 2, with 45, 70, and 6 categories, respectively. Thus, they are not assigned exclusively to the identity expert. Experts 0–8 each receive concepts, while Expert 9 receives none in this inventory.

(3) Expert assignments correlate with relative abstraction order. In this additional analysis, concept structural height in the merged hierarchy and expert index exhibit a strong positive correlation (Spearman coefficient: 0.878). Concepts with greater structural height tend to be assigned to experts with larger indices, supporting an association between expert allocation and the relative abstraction order of concepts.

Q3: Effect of Increasing the Number of Routing Experts

(1) Extended expert-count evaluation. The original ablation compares E=6E=6, 88, and 1010, which is insufficient to establish the effect of further increasing the expert count. We have therefore added experiments with E=12E=12 and E=14E=14. Here, EE is the total number of routing experts, including the identity pathway.

(2) Average performance peaks at E=10E=10 among the tested configurations. Increasing EE from 6 to 8 and then to 10 improves average box AP from 21.9 to 23.0 to 24.3 on COCO and from 12.7 to 13.2 to 15.0 on LVIS-mini. With E=12E=12, average box AP decreases to 22.0 and 13.6, respectively, falling 2.3 and 1.4 AP below E=10E=10. With E=14E=14, the corresponding scores are 22.1 and 13.3, still below E=10E=10. These results show that increasing the number of experts does not yield continuous improvements; larger expert pools reduce average performance in the tested setting.

MethodCOCO SSCOCO USCOCO UUCOCO Avg.LVIS-mini SSLVIS-mini USLVIS-mini UULVIS-mini Avg.
Base17.716.613.115.87.412.210.29.9
E=627.925.112.721.912.114.711.312.7
E=827.425.815.923.011.514.214.013.2
HARE (E=10)31.326.814.924.315.417.112.515.0
E=1228.024.513.422.013.916.110.813.6
E=1428.825.012.622.113.116.510.413.3

(3) Possible reasons for the decline. With limited adaptation data, a larger expert pool may distribute effective supervision across more branches, leaving some experts insufficiently trained. A finer division among experts may also make routing harder to learn or introduce overlapping expert functions. These factors may increase the difficulty of expert specialization and detection adaptation.

Q4: Typographical Correction on Line 747

We thank the reviewer for pointing out this typographical error. We will replace “Eq. equation 7” with “Eq. (7)” on line 747 .

Dear Reviewer 9jBS,

Thank you again for your time and thoughtful feedback. We would appreciate hearing whether our rebuttal and additional results have helped address your concerns. If any points remain unclear or would benefit from further discussion, we would be happy to provide additional clarification.

Reviewer_pHCBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies higher-level category detection in open-vocabulary object detection. The authors construct semantic hierarchies for COCO, LVIS-mini, and Objects365, and propose HARE, which uses parent-child relationships to train a router that selects among LoRA experts. The method improves higher-level detection AP across all three datasets while preserving concrete-category performance on COCO.

Critical strengths
  1. The paper addresses a meaningful problem. Existing open-vocabulary detectors often struggle with abstract category queries, and the proposed evaluation protocols are useful for studying this issue.
  2. The experimental results are encouraging. HARE consistently improves higher-level detection AP across three datasets without degrading concrete-category AP on COCO. The comparison with load-balancing routing also supports the use of hierarchy supervision.
Critical weaknesses
  1. The necessity of the proposed routing mechanism is not fully established. It is unclear whether a learned hierarchy-aware router is better than simpler alternatives, such as assigning experts based on category depth or using the identity expert for concrete categories. The single-LoRA baseline also does not match HARE's parameter budget.
  2. The quality of the hierarchical annotations needs further validation. The hierarchy is constructed using GPT-5, and existing bounding boxes are propagated to ancestor categories. However, the paper does not provide a quantitative manual evaluation of the resulting labels. Missing subclasses or incorrect hierarchy relations could affect the reported AP.
  3. The two-stage training design is not sufficiently justified, as the router is optimized only for hierarchy ranking and does not receive detection feedback.
Questions for authors
  1. Could the authors compare HARE with a simple depth-based routing strategy using the same number of experts?
  2. How frequently is each expert selected during training and inference? Are there experts that are rarely or never activated? It would be helpful to report the expert usage distribution, including the identity expert.
  3. Why is the router trained separately and then frozen during LoRA adaptation? Have the authors considered jointly training the router and LoRA experts using both the hierarchy-ranking loss and detection loss?
4 comments

We thank the reviewer for the constructive comments and respond point by point below.

W1 & Q1: Necessity of Hierarchy-Aware Routing and Parameter-Matched Baselines

(1) Comparison with fixed-depth routing. We add a fixed-depth baseline using the same number of experts, LoRA rank, and training settings as HARE. We change only the category-to-expert assignment and retrain the LoRA experts using the fixed assignments. Since path lengths vary and the same concept may occupy different positions along different paths, path-specific level indices cannot provide consistent category-level assignments. We therefore compute a unified bottom-up structural height in the merged DAG and assign experts through a predetermined monotonic mapping. This rule assigns all 121 concrete categories (100.0%) to the identity expert (Expert 0), covering both depth-based expert assignment and identity routing for concrete categories.Compared with Base, fixed-depth routing improves average box AP by 7.5 and 3.1 AP on COCO and LVIS-mini, respectively, while HARE achieves gains of 8.6 and 5.1 AP. Under the same expert configuration, HARE further outperforms fixed-depth routing by 1.0 and 2.0 AP, respectively, demonstrating the additional benefit of learned hierarchy-aware routing.

DatasetMethodSSUSUUAvg.
COCOBase17.716.613.115.8
COCOFixed-Depth29.627.113.423.3
COCOHARE31.326.814.924.3
LVIS-miniBase7.412.210.29.9
LVIS-miniFixed-Depth12.915.910.313.0
LVIS-miniHARE15.417.112.515.0
AssignmentNumber of concrete categoriesPercentage
Expert 0 (identity)121100.0%
Other experts00.0%

(2) Comparison with parameter-matched Single LoRA. We expand Single LoRA to match HARE's total parameter increase over Base, approximately 1.7% for both methods. On COCO, the expanded Single LoRA improves average higher-level category box AP from 15.8 to 23.2 (+7.4 AP), but reduces concrete-category AP from 56.9 to 53.5 (−3.4 AP). HARE improves average higher-level category box AP to 24.3 (+8.6 AP) while maintaining concrete-category AP at 56.9. Under the same total parameter budget, these results show that HARE improves higher-level category detection while preserving concrete-category detection performance.

MethodParameter IncreaseConcrete-Category APHigher-Level Avg. AP
Base—56.915.8
Single LoRA (Expanded)Approx. +1.72%53.523.2
HAREApprox. +1.72%56.924.3

W2: Quality of Hierarchical Annotations

We address this concern through three complementary checks: manual annotation correction, an edge-by-edge audit of the training hierarchy, and controlled missing-subclass annotation experiments.

(1) Manual annotation review and reevaluation. We randomly sampled 500 images from each of the SS, US, and UU splits of COCO, reviewed them image by image, and corrected incorrect and missing annotations. The overall correction rate was approximately 1.0%, indicating relatively high annotation quality in the audited sample. We then reevaluated Base and HARE on the same images using the corrected annotations.

The average box AP of Base/HARE changes from 16.9/25.6 with the original annotations to 17.1/25.7 with the corrected annotations, with HARE retaining an advantage of 8.7 AP. Thus, the corrections affect absolute AP but do not alter HARE's average performance advantage on this audited sample.

Detection results with manually corrected annotations (500 images per split)

MethodSS APUS APUU APAvg.
Base18.317.415.617.1
HARE32.127.717.525.7

(2) Edge-by-edge audit of the training hierarchy. We audited all 277 unique parent–child relations actually used in the Objects365 adaptation annotations, covering 121 original categories and 138 higher-level concepts. Two human annotators assessed the relations independently, supplemented by auxiliary assessments from GPT-6.1 and Doubao. Each assessment checked whether the child concept is a type of the parent concept is-a relation. All acceptance rates below use 277 relations as the denominator.

Assessment sourceAcceptedRejectedAcceptance rate
Human annotator 12482989.5%
Human annotator 22522591.0%
GPT-6.1 auxiliary assessment2473089.2%
Doubao auxiliary assessment2492889.9%

The human annotators accepted 89.5% and 91.0% of the relations, respectively, and the auxiliary model assessments yielded similar rates. These results indicate that most training hierarchy relations were accepted in the audit.

To quantify how frequently rejected relations occur in annotation propagation, we further weighted each relation by its actual number of occurrences along the hierarchy paths associated with training boxes. For each of the four assessments, we divided the total occurrence count of rejected relations by the occurrence count of all relations, then averaged the four ratios, obtaining 5.8%. This measures the relatively small share of propagation steps involving rejected relations. Together with the reevaluation above, these checks support the stability of the performance comparison.

(3) Sensitivity to controlled omissions of subclass annotations. We introduced controlled annotation omissions on the full existing test sets of COCO and LVIS-mini. Keeping the models, images, queries, and predictions fixed, we randomly retained 100.0%, 90.0%, 80.0%, 70.0%, or 60.0% of the annotated source subclasses for each higher-level category. Every retention ratio except 100.0% was evaluated with 10 random repetitions.

Evaluation with randomly omitted subclass annotations

DatasetSubclass retention ratioBase APHARE APGain (AP)
COCO100.0%15.824.3+8.6
COCO90.0%15.1±0.423.3±0.4+8.2±0.1
COCO80.0%13.8±0.420.9±0.3+7.1±0.4
COCO70.0%11.2±0.617.1±0.6+5.9±0.6
COCO60.0%10.3±0.815.6±0.9+5.2±0.7
LVIS-mini100.0%10.015.0+5.1
LVIS-mini90.0%9.3±0.314.1±0.5+4.8±0.3
LVIS-mini80.0%8.8±0.413.3±0.6+4.5±0.3
LVIS-mini70.0%7.0±0.710.4±0.5+3.4±0.4
LVIS-mini60.0%6.3±0.69.5±0.5+3.2±0.4

Even when only 60.0% of source subclasses are retained, HARE achieves 15.6 AP on COCO and 9.5 AP on LVIS-mini, maintaining average advantages of 5.2 and 3.2 AP over Base, respectively. For every retention ratio and every random repetition on both datasets, HARE's average box AP across the three splits remains higher than Base's. Together with the reevaluation using manually corrected annotations, these results show that HARE's relative advantage persists under the tested conditions of reduced annotation completeness.

W3 & Q3: Two-Stage Training without Detection Feedback to the Router

(1) Joint-training baseline and results. Following the reviewer's suggestion, we added a Joint baseline that jointly optimizes the router and LoRA experts using both the hierarchy-ranking loss and the detection loss. It retains hard Top-1 expert selection in the forward pass and passes detection feedback to the router through softmax routing probabilities during backpropagation, without directly differentiating through the discrete argmax operation.

HARE achieves average box AP of 24.3 and 15.0 on COCO and LVIS-mini, respectively, compared with 23.5 and 13.7 for Joint. HARE also outperforms Joint on SS, US, and UU on both datasets. These results support the two-stage design in this comparison, including its generalization to higher-level categories unseen during adaptation.

DatasetMethodSSUSUUAvg.
COCOBase17.716.613.115.8
COCOHARE31.326.814.924.3
COCOJoint31.026.013.423.5
LVIS-miniBase7.412.210.29.9
LVIS-miniHARE15.417.112.515.0
LVIS-miniJoint12.916.711.513.7

(2) Possible reasons for the performance difference. Two factors may explain the lower performance of Joint. First, jointly updating the router and experts couples their optimization: changes in routing alter the query distribution received by each expert, while updates to the experts change the detection feedback supplied to the router. This interaction may make it harder for the experts to develop stable specialization.

Second, the hierarchy-ranking loss emphasizes hierarchical relations between concepts, whereas the detection loss primarily derives from concepts seen during adaptation. The two supervision signals may therefore favor different router updates. Even when the hierarchy-ranking loss is retained, detection supervision may bias routing toward seen concepts and affect transfer to unseen concepts.

Q2: Expert Usage during Training and Inference

(1) Selection frequencies, including the identity expert. We report the Top-1 selection frequency of every expert over all training inputs and during inference on COCO and LVIS-mini. Each query-level routing decision counts as one selection. The statistics include the identity expert, Expert 0.

Top-1 expert selection frequencies (%)

ExpertTraining inputs (%)COCO inference (%)LVIS-mini inference (%)
Expert 0 (identity)20.03.32.4
Expert 132.94.76.0
Expert 221.315.716.5
Expert 313.717.424.5
Expert 42.88.310.7
Expert 54.226.521.1
Expert 62.417.413.6
Expert 71.43.32.6
Expert 81.33.32.6
Expert 90.00.00.0

(2) Usage is distributed across multiple experts. Experts 0–8 are selected during training and on both test datasets, while Expert 9 is not selected in any of these statistics. Experts 1 and 2 receive the largest shares of training inputs. At inference, Expert 5 is selected most frequently on COCO, and Expert 3 on LVIS-mini. These distributions show that routing does not collapse to a single expert and that the model uses multiple adaptation branches.

Dear Reviewer pHCB,

Thank you again for your time and thoughtful feedback. We would appreciate hearing whether our rebuttal and additional results have helped address your concerns. If any points remain unclear or would benefit from further discussion, we would be happy to provide additional clarification.

Comments on the paper

1 comment

Dear AC, SAC, and Reviewers,

We sincerely thank the AC and SAC for coordinating the review process and the reviewers for their constructive feedback, which helped us strengthen benchmark validation, add controlled comparisons, and clarify HARE’s design and generalization.

Summary of the Original Reviews

The reviewers recognized the practical value of higher-level category detection, the usefulness of the hierarchy-annotated benchmarks and evaluation protocols, and the consistent improvements across datasets (pHCB, ALqs). They also highlighted hierarchy supervision and the identity pathway for retaining concrete-category capabilities (pHCB, 9jBS).

The main concerns involved benchmark reliability (pHCB, ALqs), the necessity of learned routing and parameter-matched comparisons (pHCB, ALqs), the motivation for hierarchy-aware adaptation and expert allocation (9jBS), two-stage training (pHCB), and methodological novelty and generalization across subclasses and abstraction levels (ALqs).

Summary of the Rebuttal and Discussion

We responded point by point with additional experiments and analyses:

  • Benchmark reliability: We manually reviewed 500 images from each COCO split. After annotation correction, HARE retained a substantial advantage over Base (25.7 vs. 17.1 AP). We also audited all 277 training hierarchy relations and tested robustness to controlled subclass-annotation omissions (pHCB, ALqs).
  • Controlled comparisons: HARE outperformed fixed-depth routing by 1.0/2.0 average AP on COCO/LVIS-mini. Against expanded Single LoRA with the same approximately 1.72% total parameter increase, HARE achieved higher COCO higher-level AP (24.3 vs. 23.2) while retaining concrete-category AP at 56.9, compared with 53.5 for Single LoRA (pHCB, ALqs).
  • Routing motivation and behavior: We added visual-similarity and same-instance query-replacement analyses, assignments for all 259 adaptation concepts, and expert usage statistics. Structural height correlated with expert index (Spearman ρ=0.878), supporting the connection between allocation and relative abstraction order (9jBS, pHCB).
  • Training and expert configuration: We added joint-training and larger expert-pool experiments. Two-stage HARE outperformed Joint on both datasets, while E=10 achieved the highest average AP among the tested configurations (pHCB, 9jBS).
  • Generalization: Added per-subclass results demonstrated transfer to subclasses unseen during adaptation. Across structural heights, HARE improved at eight of nine evaluated heights, with a 1.4 AP decrease at the remaining height (ALqs).
  • Novelty: We clarified the contribution as the new higher-level query detection problem and benchmarks, an initial exploration of hierarchy-ranking supervision for this task, and a task-specific identity expert supporting capability retention (ALqs).

Reviewers’ Follow-up

Reviewer 9jBS stated that the visual-similarity and recall analyses and concept-to-expert breakdown addressed most of their concerns, and raised their rating from Weak Rejection to Weak Accept.

Reviewer ALqs acknowledged that the unseen-subclass results and fixed-depth comparison addressed some earlier questions, while maintaining their original rating due to reservations about methodological novelty. We subsequently clarified the three contributions above and invited further discussion of any remaining novelty concerns.

Reviewer pHCB has not provided further feedback on our rebuttal as of this response. We addressed their questions point by point, including fixed-depth routing, parameter-matched Single LoRA, annotation validation, joint training, and expert usage statistics.

We thank the AC, SAC, and reviewers again for their time and thoughtful engagement. We hope these clarifications and additional results will help inform the final assessment.

Sincerely,
The Authors of Paper 14326

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.