acceptodds
Under review as a conference paper at ICLR 2027

HyperRS: Hyperbolic Embeddings for Hierarchical Aerial Image Classification

Abstract

Remote sensing scenes are naturally organized into semantic hierarchies, yet existing remote sensing foundation models typically discard this structure and learn flat representations in Euclidean space. Consequently, their predictions may be accurate without reflecting the semantic relations among categories. We introduce HyperRS, a hierarchical fine-tuning framework that explicitly aligns a pretrained hyperbolic image–text model with a remote sensing taxonomy. HyperRS applies hierarchical supervised contrastive learning at every taxonomy level and introduces a ranking loss that orders category embeddings by semantic granularity, placing fine-grained categories farther from the hyperbolic origin than their ancestors. Together with an entailment objective, these losses organize both the semantic and radial structure of the representation. Experiments on Million-AID and RSI-CB256 show that HyperRS outperforms the evaluated single-level and hierarchical models, achieving hierarchical precision of and , respectively. Detailed analysis shows that HyperRS preserves correct coarse-level predictions more often when fine-grained classification fails and recovers ancestor chains more faithfully through geometric traversal than the compared baselines. It also maintains higher hierarchical precision than baselines across three common remote sensing image corruptions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
remote sensing, hierarchical classification, vision-language models, CLIP

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_EDQcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes HyperRS to make aerial image classification reflect coarse-to-fine relationships between categories. It fine-tunes the pretrained hyperbolic image-text model MERU using a supplied label hierarchy: images and labels are aligned at multiple levels, ancestor-descendant relations are constrained, and finer categories are placed farther from the embedding origin. The experiments are mainly conducted on the Million-AID and RSI-CB datasets.

Critical strengths

A useful control for backbone and pretraining differences. CLIP and MERU use the same architecture and pretraining data (p. 6). Applying HierCon and HSC to both makes their comparison more informative than an unrestricted comparison of unrelated foundation models. The loss ablation identifies a concrete effect of radial supervision. Reporting that trade-off supports a specific geometric observation rather than claiming every objective improves everything.

Critical weaknesses

Remote sensing image classification is a well-studied task, and further accuracy gains on established benchmarks may have limited practical value. The paper should clarify which unresolved problems HyperRS addresses and why its improvements matter for real applications, beyond achieving higher benchmark scores.

The evaluation is not independent of model selection. Appendix C states “Neither dataset has a separate validation split,” and selects all loss weights using evaluation Hierarchical precision (p. 16). That makes the reported evaluation part of method development, rather than an untouched estimate of generalization. The claimed predictive increment needs independent evidence.

The paper conflates hyperbolic geometry with explicit hierarchy supervision and variable radius. The conclusion says Euclidean embeddings “essentially do not support” ancestor recovery (p. 9). However, HyperRS learns radial ordering, whereas the Euclidean controls use normalized cosine embeddings without equivalent radial supervision. Euclidean vectors can also have learned norms and ordered radii. The traversal result supports the evaluated configuration, but does not isolate negative curvature as the cause or establish the stated general limitation.

Questions for authors

See above.

6 comments
AuthorsOpenReview ↗

Reply to Reviewer EDQc (Part 1/4)

We thank the reviewer for the thoughtful review and for noting the value of the matched controls and the loss ablation.

W1. Unclear practical value of improvements on a well-studied task beyond benchmark scores.

Reply: Thank you for raising this point! We would like to clarify that, as motivated in Sec. 1, our paper addresses a task different from the well-studied flat aerial scene classification: hierarchical classification, where a model must predict a consistent label at every level of the taxonomy and should keep the correct coarse category when the fine label is wrong. This matters in remote sensing, where land-use products are defined at several granularities, and it is not measured by flat accuracy. Our evaluation therefore targets these properties, which Reviewers RkqC and c75V noted as a strength of the paper ("not only at classification accuracy" and "hierarchy beyond leaf accuracy"). We clarify the practical value of HyperRS in three aspects:

  • Errors within the taxonomy (CHA, Table 2). We examine whether the coarse labels remain correct when the finest prediction fails. HyperRS does so more often than the baselines, so its errors are more often still usable at a coarser granularity.
  • Hierarchy in the geometry (traversal, Table 3). We examine whether the taxonomy is encoded in the embedding itself. Such traversals have so far been shown qualitatively, e.g., in MERU; we score them against the ground-truth taxonomy so that this property can be compared across models.
  • New: direct applications. Practical value is best shown by applications that use the hierarchy directly, so we have added two: hierarchical retrieval and confidence-based coarse-label fallback. HyperRS outperforms CLIP in almost all settings, e.g., image-to-image mAHP 81.4 vs. 77.5 on RSI-CB and information gain 74.0 vs. 70.2 at τ=0.9\tau = 0.9 on Million-AID (full setups and tables in Parts 3/4 and 4/4 of this response and in the appendix of the revised paper).

W2. Evaluation not independent of model selection, and no independent evidence for the predictive increment.

Reply: Thank you for this important concern!

  • The weight selection barely affects the reported performance. We would like to emphasize that a validation split mainly serves hyperparameter selection, and our setting requires very little of it. We did not tune the training hyperparameters: they follow common fine-tuning practice for CLIP and MERU and are identical for all methods (Sec. 4), without any extensive search. The only selected values are the loss weights introduced by our method, and once they are in a reasonable range, they barely affect performance:

    • Level weights. In sweeps for both CLIP+HSC and MERU+HSC (Figs. 2-3 in Appendix C), hP rises sharply only at small weights and then reaches a stable plateau, within which it varies by about 0.3 points; we chose one shared setting from this plateau for all image-text methods.
    • Entailment and ranking weights. Adding both objectives changes hP only from 87.1 to 87.2 (Table 5).

    Therefore, even without a separate validation split, the evaluation remains fair.

  • Predictive increment. MERU+HSC is HyperRS without these two objectives, and HSC is itself part of our contribution. We do not claim that the full model is more accurate than MERU+HSC and will revise observation 4 in Sec. 4.1 accordingly.

  • Evidence less affected by the selection. We therefore base our predictive claims on effects larger than the selection variation: multi-level supervision over leaf-only training, and hyperbolic over Euclidean geometry under the same supervision on Million-AID (+1.2+1.2 hP), where the selected weights also lie in the best range of the CLIP+HSC curve (on RSI-CB, the two are within this variation). CHA and traversal (Tables 2-3) were not selection criteria, as all weights were chosen by hP.

AuthorsOpenReview ↗

Reply to Reviewer EDQc (Part 2/4)

W3. Hyperbolic geometry conflated with explicit hierarchy supervision and variable radius.

Reply: Thank you for this precise point! Our experiments separate the factors the reviewer mentions as far as they allow:

  • Hierarchy supervision is held fixed. All baselines in Table 3 are trained with the same three-level HierCon objective, and none uses a ranking or other radial term. Under this identical supervision, MERU recovers the whole ancestor chain for 22.6% of the images, against at most 2.9% for CLIP and RemoteCLIP. Hyperbolic embeddings, in which the distance to the origin is part of the geometry, are thus naturally suited to encoding the hierarchy.
  • Radial supervision is isolated within the hyperbolic model. With the same backbone and HSC supervision, the ranking objective raises whole-path recovery from 24.0 to 37.7 (Table 5): explicit radial supervision strengthens a structure the hyperbolic model already exhibits without it.

We agree that Euclidean embeddings can learn variable norms and ordered radii, and that our statement that they “essentially do not support” ancestor recovery is too broad. Our original controls used normalized embeddings; we have therefore added a Euclidean model with learned radii (below) and will revise this statement based on it.

Our motivation for applying radial ranking in hyperbolic space concerns how the learned ordering interacts with the surrounding geometry. At a fixed dimension, hyperbolic space exhibits asymptotically exponential volume growth with geodesic radius, providing an inductive bias suited to the increasing number of categories at deeper taxonomy levels [1]. Moreover, HyperRS combines radial ordering with hyperbolic distance and radius-dependent entailment cones [2, 3]. Thus, radial position participates in both image–text alignment and hierarchical entailment. These properties motivate our choice of geometry; they do not establish an exclusive ability to encode hierarchy or guarantee superiority over Euclidean representations with equivalent supervision.

To compare Euclidean and hyperbolic geometry, we have added CLIP + rank and Hyperbolic + rank to Table 5 of the revised paper; both use the same HierCon loss and weights and the same ranking loss. For CLIP + rank, the ranking loss acts on the label-text embeddings before L2 normalization, so that their norms can be ordered. Its traversal starts from the image direction scaled to the largest text norm, moves toward the origin, and retrieves the nearest taxonomy node by Euclidean distance to the unnormalized text embeddings, analogous to the hyperbolic traversal. Million-AID, values in %, the better value in bold:

MethodAcc(1)Acc(2)Acc(3)hPTrav. hPLCSH(d1)H(d2)H(d3)H(all)H(exact)
CLIP (w/ HierCon) + rank93.490.286.685.555.340.024.432.970.60.10.1
Hyperbolic (w/ HierCon) + rank93.889.888.087.268.161.931.082.483.720.717.0

Hyperbolic + rank predicts slightly better, and the gap widens sharply in traversal: with ordered radii, CLIP + rank still finds the leaf but rarely its ancestors, and almost never recovers the whole chain. This further supports that hyperbolic geometry is better suited to encoding the hierarchy, even when Euclidean embeddings learn ordered radii.

References

[1] M. Nickel and D. Kiela. Poincaré embeddings for learning hierarchical representations. In NeurIPS, 2017.

[2] O.-E. Ganea, G. Bécigneul, and T. Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In ICML, 2018.

[3] K. Desai, M. Nickel, T. Rajpurohit, J. Johnson, and R. Vedantam. Hyperbolic image-text representations. In ICML, 2023.

AuthorsOpenReview ↗

Reply to Reviewer EDQc (Part 3/4)

W1 (continued). New experiment 1: hierarchical retrieval.

To show the practical value of the learned hierarchy, we evaluated hierarchical retrieval on Million-AID (MA) and RSI-CB256 (RSI), comparing HyperRS with CLIP trained with HierCon. Values are in %; higher is better; the better value is in bold.

Setup. HP@k [4] is the sum of the similarities of the top- kk results divided by the best achievable sum, with similarity 1−height(LCA)/height(root)1 - \mathrm{height(LCA)}/\mathrm{height(root)}; mAHP@K is the mean of [email protected]. Image to image: image queries, macro-averaged over classes. Image to text: image queries against the leaf-class texts (mAHP over all KK leaf texts). Text to image: the text of every taxonomy node as a query, one query per node. HP@k reflects the hierarchy only once kk exceeds the size of the query's own leaf class, since before that a perfect leaf-level ranking already scores 1. We therefore set the cutoffs per dataset: on MA (median test images per class: 265.5 at level 1, 22 at level 2, 20 at the leaf), kk = 10, 50, 100, 250 and mAHP@250; on RSI (3234 at level 1, 580 at the leaf), whose classes are more than ten times larger, kk = 100, 500, 1000, 2500 and mAHP@2500. The largest cutoff is about the size of a level-1 class.

TaskMA metricMA: CLIPMA: HyperRSRSI metricRSI: CLIPRSI: HyperRS
Image to imageHP@1075.5376.61HP@10093.9894.45
HP@5067.7168.85HP@50082.9785.71
HP@10065.8467.04HP@100077.1381.23
HP@25063.0564.59HP@250068.6673.88
mAHP@25066.3467.57mAHP@250077.5081.41
Image to textHP@190.0790.91HP@195.5596.57
HP@568.3170.73HP@573.6674.38
mAHP@5188.6189.29mAHP@3590.8191.03
Text to imageHP@1091.7292.67HP@10098.5498.95
HP@5079.5680.47HP@50092.5593.37
HP@10073.8875.71HP@100085.7185.79
HP@25069.4371.42HP@250077.7577.22
mAHP@25075.1876.78mAHP@250085.8385.87

HyperRS is higher at all but one of the 26 cutoffs. The largest gains appear in image-to-image retrieval at large kk, where neighbors beyond the query's own class must be ranked by the taxonomy.

References

[4] B. Barz and J. Denzler. Hierarchy-based image embeddings for semantic image retrieval. In WACV, 2019.

Reviewer_c75VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HyperRS fine-tunes MERU for hierarchical aerial-scene classification using multi-level supervised contrastive learning, image-text entailment, and a text-radius ranking loss. It evaluates classification, coarse-label correctness on shared fine-level errors, geometric ancestor-chain retrieval, and robustness on Million-AID and RSI-CB256.

Critical strengths
  1. The paper evaluates hierarchy beyond leaf accuracy, including shared-error CHA and ancestor-chain traversal. On the HSC hyperbolic base, the full method raises whole-chain traversal coverage from 24.0% to 36.4% (Table 5), an interesting structural result.

  2. The matched CLIP/MERU pretraining setup and application of HSC to both Euclidean and hyperbolic backbones are useful controls. The loss ablations distinguish the ranking term's structural benefit from the entailment term's effect on classification.

Critical weaknesses
  1. Hyperparameter selection uses the same evaluation split on which results are reported. Sec. 4 and Appendix C explicitly choose all loss weights by evaluation hP, including the method-specific entailment and ranking weights. Sharing level weights across baselines does not remove this selection bias. A disjoint validation protocol and fresh final evaluation are necessary before the reported gains can be interpreted as held-out performance.

  2. The incremental contribution and its reproducible configuration remain unclear. Relative to MERU+HSC, Million-AID hP changes only from 87.1 to 87.2 and RSI-CB classification metrics are identical at the reported precision; the more substantial benefit is traversal. Table 3 uses HierCon baselines, while the normalized Euclidean traversal does not establish that Euclidean hierarchy-aware representations generally lack the property. In addition, the supplied README describes a sibling-normalized objective, and its recommended config imports an absent models_hier_cond module. Although HSC and ranking implementations exist in models.py, the released entry does not identify the exact configuration behind the paper.

Questions for authors
  1. Can all loss weights be selected within the training data and the comparisons rerun on a final untouched split, with uncertainty across seeds?

  2. Does the traversal improvement persist against HSC-trained baselines and a Euclidean hierarchy model allowed to learn radial structure?

  3. Which runnable configuration and objective produced each main table, and how does it relate to the different objective in the supplementary README?

3 comments
AuthorsOpenReview ↗

Reply to Reviewer c75V (Part 1/2)

We thank the reviewer for the constructive feedback and for finding the matched CLIP/MERU setup and the loss ablations useful.

W1 and Q1. Weights tuned on the evaluation split, without selection within the training data, a final untouched split, or uncertainty across seeds.

Reply: Thank you for raising this important concern!

  • The weight selection barely affects the reported performance. We would like to point out that a validation split mainly serves hyperparameter selection, and our setting requires very little of it. We did not tune the training hyperparameters: they follow common fine-tuning practice for CLIP and MERU and are identical for all methods (Sec. 4), without any extensive search. The only selected values are the loss weights introduced by our method, and once they are in a reasonable range, they barely affect performance:

    • Level weights. In sweeps for both CLIP+HSC and MERU+HSC (Figs. 2-3 in Appendix C), hP rises sharply only at small weights and then reaches a stable plateau, within which it varies by about 0.3 points; we chose one shared setting from this plateau for all image-text methods.
    • Method-specific weights. λent\lambda_{\mathrm{ent}} and λrank\lambda_{\mathrm{rank}} change hP only from 87.1 to 87.2 (Table 5).

    Therefore, even without weights selected within the training data, the comparisons remain fair.

W2. Small increment over MERU+HSC, HierCon baselines in Table 3, an overly general Euclidean claim, and a README mismatch.

Reply: Thank you for examining our work so carefully, including the code!

  • Contribution. MERU+HSC is HyperRS without the entailment and ranking objectives, and HSC is itself part of our contribution. The aim of these two objectives is not higher accuracy but a better-learned hierarchy at preserved accuracy: on the same HSC base, they raise whole-path recovery from 24.0 to 36.4 while hP stays at 87.2 vs. 87.1 (Table 5). We therefore do not claim that the full model is more accurate than MERU+HSC and will revise observation 4 in Sec. 4.1 accordingly. Our predictive claims rest on multi-level supervision over leaf-only training and on hyperbolic over Euclidean geometry under the same supervision on Million-AID (Table 1).
  • Euclidean models. In Table 3, under the same three-level HierCon supervision and without any ranking or other radial term, MERU recovers the whole ancestor chain for 22.6% of the images, against at most 2.9% for CLIP and RemoteCLIP. Hyperbolic embeddings, in which the distance to the origin is part of the geometry, are thus naturally suited to encoding the hierarchy. A new controlled experiment further supports this: with the same ranking loss added to both geometries, CLIP + rank recovers the whole ancestor chain for only 0.1% of the images, against 20.7% for Hyperbolic + rank (Q2).
  • README and configuration. We sincerely apologize for the confusion. The mismatch is limited to the documentation: the README was taken by mistake from a development branch, and the sibling-normalized objective and models_hier_cond it describes were not used in the paper. All results come from HierHyperRS and CLIPBaseline in hyperrs/models.py, with exactly the objectives and hyperparameters stated in the paper; the remaining options are disabled by default and play no role in any result, so the reported results remain reproducible. We have updated the supplementary with a corrected README and scripts needed to reproduce the results (Q3).
AuthorsOpenReview ↗

Reply to Reviewer c75V (Part 2/2)

Q2. Whether the traversal gain persists against HSC-trained baselines and a Euclidean model that learns radial structure.

Reply: Thank you for this suggestion!

  • HSC-trained baselines. On the same HSC base and backbone, the entailment and ranking objectives raise whole-path recovery from 24.0 to 36.4 (Table 5).
  • Euclidean model with radial structure. Our motivation for applying radial ranking in hyperbolic space concerns how the learned ordering interacts with the surrounding geometry. At a fixed dimension, hyperbolic space exhibits asymptotically exponential volume growth with geodesic radius, providing an inductive bias suited to the increasing number of categories at deeper taxonomy levels [1]. Moreover, HyperRS combines radial ordering with hyperbolic distance and radius-dependent entailment cones [2, 3]. Thus, radial position participates in both image–text alignment and hierarchical entailment. These properties motivate our choice of geometry; they do not establish an exclusive ability to encode hierarchy or guarantee superiority over Euclidean representations with equivalent supervision.

To compare Euclidean and hyperbolic geometry, we have added CLIP + rank and Hyperbolic + rank to Table 5 of the revised paper; both use the same HierCon loss and weights and the same ranking loss. For CLIP + rank, the ranking loss acts on the label-text embeddings before L2 normalization, so that their norms can be ordered. Its traversal starts from the image direction scaled to the largest text norm, moves toward the origin, and retrieves the nearest taxonomy node by Euclidean distance to the unnormalized text embeddings, analogous to the hyperbolic traversal. Million-AID, values in %, the better value in bold:

MethodAcc(1)Acc(2)Acc(3)hPTrav. hPLCSH(d1)H(d2)H(d3)H(all)H(exact)
CLIP (w/ HierCon) + rank93.490.286.685.555.340.024.432.970.60.10.1
Hyperbolic (w/ HierCon) + rank93.889.888.087.268.161.931.082.483.720.717.0

Hyperbolic + rank predicts slightly better, and the gap widens sharply in traversal: with ordered radii, CLIP + rank still finds the leaf but rarely its ancestors, and almost never recovers the whole chain. This further supports that hyperbolic geometry is better suited to encoding the hierarchy, even when Euclidean embeddings learn ordered radii.

Q3. Which configuration and objective produced each table, and how they relate to the README.

Reply: Thank you for the question! As described in W2, the README objective is not used in any table. Two rules hold throughout:

  • Same weights for all methods. Every image-text method trained with HierCon or HSC uses the level weights in Appendix C: (1,3,2.4)(1, 3, 2.4) on Million-AID and (1,1)(1, 1) on RSI-CB. HyperRS additionally uses entailment 3.0 per level and ranking 0.5 (margin 0.1) on Million-AID, and 0.2 and 0.2 on RSI-CB; these terms are disabled for all other methods.
  • Same default objectives. The compared models are trained with HierCon, and our models with HSC. Where a table compares other objectives, e.g., LeafCon, HierCon, and HSC in Table 1, this is stated in its caption.

We hope these responses address the reviewer's concerns.

References

[1] M. Nickel and D. Kiela. Poincaré embeddings for learning hierarchical representations. In NeurIPS, 2017.

[2] O.-E. Ganea, G. Bécigneul, and T. Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In ICML, 2018.

[3] K. Desai, M. Nickel, T. Rajpurohit, J. Johnson, and R. Vedantam. Hyperbolic image-text representations. In ICML, 2023.

Dear Reviewer c75V,

With less than 2 hours left in the discussion period, we would like to kindly follow up on our rebuttal, including the corrected README, the reproduction scripts, and the new controlled comparison with a Euclidean model (Table 5). We would greatly appreciate knowing whether these address your concerns.

Thank you again for your careful review.

Best regards,

The Authors

Reviewer_RkqCOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces HyperRS, a framework that adapts pretrained hyperbolic image–text representations for use in hierarchical remote-sensing classification. It achieves this by combining hierarchical supervised contrastive learning at various levels of the taxonomy with a radial ranking loss that positions finer categories at a greater distance from the origin and an entailment objective that matches images to their ancestor labels. The method is evaluated on Million-AID and RSI-CB256 by comparing it with Euclidean and hyperbolic baseline approaches using classification accuracy, hierarchical measures, and an analysis of the error structure. Further investigation is carried out through geometric traversal studies, loss ablation experiments, and experiments involving corrupted images to examine the learned representations. The results suggest that multi-level supervision accounts for most predictive improvements, while radial ranking substantially improves ancestor-chain recovery. The full HyperRS framework provides only marginal additional predictive gains over MERU+HSC.

Critical strengths
  1. HyperRS combines multi-level supervised contrastive learning with radial ranking and entailment constraints in order to encode semantic associations and category granularity within hyperbolic image–text representations; the various objectives play complementary roles since hierarchical supervision aligns the images with their labels at different levels, radial ranking arranges the category embeddings by their specificity, and entailment constraints relate the images to their ancestor labels.
  2. The study looks not only at classification accuracy but also investigates whether coarse-level predictions remain accurate when fine-grained ones fail and whether ancestor chains can be reconstructed by means of geometric traversal. The results of the ablation studies offer particularly clear evidence in support of the ranking objective: when it is added to MERU+HSC, whole-chain recovery on Million-AID increases from 24.0% to 37.7%, which shows that the objective is effective in making the taxonomy more directly recoverable from the embedding geometry.
Critical weaknesses
  1. Hierarchical precision on the evaluation splits is used to select the loss weights rather than relying on independent validation sets, and there is no reporting of uncertainty with respect to the training seeds or of confidence intervals. This is particularly worrying since the extra hierarchical precision obtained over MERU+HSC amounts to just 0.1 percentage points on Million-AID and is zero at the precision stated for RSI-CB256. Because there is no untouched test set and no uncertainty estimates, it is hard to assess whether these small differences are reproducible or partly reflect evaluation-set tuning.
  2. It has not yet been sufficiently established what contribution geometry makes or what practical value there is in improved ancestor-chain recovery. Even though the comparisons between CLIP and MERU control for architecture, pretraining data, and supervision, they do not isolate downstream geometry under a common initialization; furthermore, the comparisons with the full HyperRS framework alter the training objectives. Although radial ranking clearly leads to better ancestor-chain recovery, the experiments do not show whether this geometric improvement brings about any additional practical benefits beyond the classification and corruption-robustness results already reported. An application that makes direct use of the recovered hierarchy—such as hierarchical retrieval or confidence-based coarse-label fallback—would provide clearer evidence of its usefulness.
Questions for authors
  1. The loss weights mentioned in Appendix C were chosen using hP on the evaluation splits. Are the results given in Table 1 based on the same splits or has an independent held-out evaluation been carried out? Has more than one training seed been used, and if so, how variable are the differences between HyperRS and MERU+HSC? This will aid in assessing whether the small predictive improvements are reproducible and independent of the hyperparameter selection.
  2. Table 5 indicates a considerable improvement in the recovery of ancestor chains from radial ranking, although the full HyperRS framework provides only limited additional predictive advantages over MERU+HSC. Could you please cite any of the existing results that show improved geometric recovery actually contributes to prediction or to robustness, rather than mainly enhancing the traversal diagnostic? You should separate any existing evidence from those applications which have been proposed for future work.
4 comments
AuthorsOpenReview ↗

Reply to Reviewer RkqC (Part 1/3)

We sincerely thank the reviewer for the careful reading and for recognizing the complementary roles of the objectives and the evaluation beyond classification accuracy.

W1 and Q1. Loss weights selected on the evaluation splits, with no held-out evaluation or seed uncertainty to show whether the small gain over MERU+HSC is reproducible.

Reply: Thank you for your insightful comment!

  • The weight selection barely affects the reported performance. We would like to explain that a validation split mainly serves hyperparameter selection, and our setting requires very little of it. We did not tune the training hyperparameters: they follow common fine-tuning practice for CLIP and MERU and are identical for all methods (Sec. 4), without any extensive search. The only selected values are the loss weights introduced by our method, and once they are in a reasonable range, they barely affect performance:

    • Level weights. In sweeps for both CLIP+HSC and MERU+HSC (Figs. 2-3 in Appendix C), hP rises sharply only at small weights and then reaches a stable plateau, within which it varies by about 0.3 points; we chose one shared setting from this plateau for all image-text methods.
    • Entailment and ranking weights. Adding both objectives changes hP only from 87.1 to 87.2 (Table 5).

    Therefore, even without a separate validation split, the reported comparisons remain fair.

  • Difference between HyperRS and MERU+HSC. With a single training run, we agree that this difference cannot be shown to be reproducible, and we do not claim it as a gain. MERU+HSC is HyperRS without the entailment and ranking objectives, and HSC is itself part of our contribution. These two objectives aim at a better-learned hierarchy at preserved accuracy rather than at higher accuracy (see W2 and Q2), and we will revise observation 4 in Sec. 4.1 accordingly.

  • Evidence less affected by the selection. We therefore base our predictive claims on effects larger than the selection variation: multi-level supervision over leaf-only training, and hyperbolic over Euclidean geometry under the same supervision on Million-AID, where the selected weights also lie in the best range of the CLIP+HSC curve (on RSI-CB, the two are within this variation). HSC also improves over HierCon for all three backbones on Million-AID. CHA and traversal (Tables 2-3, 5) were not selection criteria: all weights were chosen by hP alone and shared by every image-text method.

W2 and Q2. Unclear contribution of geometry, and lack of evidence that improved ancestor-chain recovery helps prediction, robustness, or practical applications.

Reply: Thank you for this valuable suggestion!

  • Role of geometry. We would like to clarify that we do not claim geometry alone explains the gains; HyperRS is designed so that hyperbolic geometry and the objectives complement each other. Under identical supervision, the hyperbolic backbone is more accurate (+1.2 hP on Million-AID, Table 1) and recovers the hierarchy far better (Table 3) than its Euclidean counterpart, and the loss ablation (Table 5) shows how the objectives build on this base: ranking strengthens the hierarchy, and entailment preserves accuracy. In the revised Table 5, with the same ranking loss added to both geometries, CLIP + rank recovers the whole ancestor chain for only 0.1% of the images, against 20.7% for Hyperbolic + rank. The CLIP/MERU comparison controls architecture, pretraining data, and supervision but not initialization, and we will state this limitation.

  • Existing evidence. In our existing results, improved geometric recovery comes without a cost in overall accuracy rather than raising it: in Table 5, entailment recovers the finest-level accuracy that ranking alone lowers, while keeping most of its traversal gain. Table 6 compares HyperRS with CLIP trained with the same HSC objective, so its robustness gain under corruptions (CHA(all) higher by up to 6.9 points) comes from what HyperRS adds on top of the shared supervision: hyperbolic geometry together with entailment and ranking.

  • New experiments. We thank the reviewer for suggesting these applications and have now evaluated both, comparing HyperRS with CLIP trained with HierCon; full setups and tables are given in Parts 2/3 and 3/3 of this response.

    • Hierarchical retrieval. HyperRS ranks retrieved items closer in the taxonomy at all but one of 26 cutoffs (e.g., image-to-image mAHP 81.4 vs. 77.5 on RSI-CB).
    • Coarse-label fallback. HyperRS attains a higher information gain at every threshold and stays more specific, e.g., answering 65.0% of Million-AID images at the leaf at τ=0.9\tau = 0.9 (CLIP: 58.5%) with an information gain of 74.0 (CLIP: 70.2).

    These results suggest that the hierarchy encoded in HyperRS embeddings is useful beyond the traversal diagnostic and classification. We have added both experiments to the appendix of the revised paper.

AuthorsOpenReview ↗

Reply to Reviewer RkqC (Part 2/3)

W2 and Q2 (continued). New experiment 1: hierarchical retrieval.

As suggested by the reviewer, we evaluated hierarchical retrieval on Million-AID (MA) and RSI-CB256 (RSI), comparing HyperRS with CLIP trained with HierCon. Values are in %; higher is better; the better value is in bold.

Setup. HP@k [1] is the sum of the similarities of the top- kk results divided by the best achievable sum, with similarity 1−height(LCA)/height(root)1 - \mathrm{height(LCA)}/\mathrm{height(root)}; mAHP@K is the mean of [email protected]. Image to image: image queries, macro-averaged over classes. Image to text: image queries against the leaf-class texts (mAHP over all KK leaf texts). Text to image: the text of every taxonomy node as a query, one query per node. HP@k reflects the hierarchy only once kk exceeds the size of the query's own leaf class, since before that a perfect leaf-level ranking already scores 1. We therefore set the cutoffs per dataset: on MA (median test images per class: 265.5 at level 1, 22 at level 2, 20 at the leaf), kk = 10, 50, 100, 250 and mAHP@250; on RSI (3234 at level 1, 580 at the leaf), whose classes are more than ten times larger, kk = 100, 500, 1000, 2500 and mAHP@2500. The largest cutoff is about the size of a level-1 class.

TaskMA metricMA: CLIPMA: HyperRSRSI metricRSI: CLIPRSI: HyperRS
Image to imageHP@1075.5376.61HP@10093.9894.45
HP@5067.7168.85HP@50082.9785.71
HP@10065.8467.04HP@100077.1381.23
HP@25063.0564.59HP@250068.6673.88
mAHP@25066.3467.57mAHP@250077.5081.41
Image to textHP@190.0790.91HP@195.5596.57
HP@568.3170.73HP@573.6674.38
mAHP@5188.6189.29mAHP@3590.8191.03
Text to imageHP@1091.7292.67HP@10098.5498.95
HP@5079.5680.47HP@50092.5593.37
HP@10073.8875.71HP@100085.7185.79
HP@25069.4371.42HP@250077.7577.22
mAHP@25075.1876.78mAHP@250085.8385.87

HyperRS is higher at all but one of the 26 cutoffs. The largest gains appear in image-to-image retrieval at large kk, where neighbors beyond the query's own class must be ranked by the taxonomy.

References

[1] B. Barz and J. Denzler. Hierarchy-based image embeddings for semantic image retrieval. In WACV, 2019.

AuthorsOpenReview ↗

Reply to Reviewer RkqC (Part 3/3)

W2 and Q2 (continued). New experiment 2: confidence-based coarse-label fallback.

As suggested by the reviewer, we evaluated confidence-based coarse-label fallback on both datasets, again comparing HyperRS with CLIP trained with HierCon.

Setup. Fixed-threshold fallback [2]: from the predicted leaf, a prediction moves up to the first ancestor whose subtree probability (softmax with temperature TT over leaves) reaches τ\tau; τ=0\tau = 0 is the plain leaf prediction. TT is fitted by 2-fold cross-fitting on the evaluation split. Leaf: share of images answered at the leaf. Acc Lk\mathrm{L}k: a prediction counts as correct at level kk only if it reaches level kk and matches the true label there; a prediction that stops above level kk counts as incorrect at that level. On Million-AID, some leaves lie at depth 2, so Acc L3 averages only over the leaf classes at depth 3, whereas EMR and IG are macro-averaged over all 51 leaf classes. IG: information gain, where a correct prediction at node nn earns (log⁡2∣Y∣−log⁡2∣leaves(n)∣)/log⁡2∣Y∣(\log_2|Y| - \log_2|\mathrm{leaves}(n)|)/\log_2|Y| (1 at a leaf, 0 at the root) and a wrong one earns 0. All levels are derived from the (fallback) leaf prediction, so the τ=0\tau = 0 rows differ from the top-down predictions of Table 1. Values are in %; the better value per τ\tau is in bold.

Million-AID:

τ\tauModelLeafAcc L1Acc L2Acc L3hPWuPEMRIG
0CLIP100.0090.9690.2586.9789.9889.9887.4587.45
HyperRS100.0093.6892.0288.4690.8190.8189.0389.03
0.5CLIP93.6289.9787.9484.5890.1288.9484.5686.53
HyperRS93.4790.9888.5486.7989.7689.1386.1087.48
0.7CLIP78.1285.6980.9874.7287.4483.6374.6781.54
HyperRS82.0786.6984.1578.6488.0285.2978.6983.85
0.8CLIP71.5481.1675.3870.1184.9380.1069.3878.20
HyperRS76.2084.0281.1673.0486.1482.3473.7980.53
0.9CLIP58.4975.4066.2158.1880.3973.0757.5770.17
HyperRS65.0278.7473.7561.6582.1476.4863.5173.98

RSI-CB256:

τ\tauModelLeafAcc L1Acc L2hPWuPEMRIG
0CLIP100.0096.9794.4395.5595.5594.4394.43
HyperRS100.0097.8895.6696.5796.5795.6695.66
0.5CLIP98.4896.7093.9195.4595.2693.9194.19
HyperRS98.9197.7395.1896.5296.3495.1895.44
0.7CLIP93.2595.0890.6794.0093.0990.6791.98
HyperRS94.5196.2492.4695.1594.4092.4693.53
0.8CLIP89.5393.6387.7592.5691.0987.7589.84
HyperRS91.6995.2590.3694.2093.0190.3692.06
0.9CLIP83.1890.6182.2889.5887.2182.2885.67
HyperRS86.6392.9585.9291.6189.7585.9288.55

At every τ\tau, HyperRS attains a higher information gain, and for τ≥0.7\tau \ge 0.7 it keeps more predictions at the leaf while being more accurate at every level. These results suggest that the hierarchy encoded in the embedding is usable beyond classification. Both experiments are included in the appendix of the revised paper.

We hope these clarifications address the reviewer's concerns.

References

[2] J. Deng, J. Krause, A. C. Berg, and L. Fei-Fei. Hedging your bets: Optimizing accuracy-specificity trade-offs in large scale visual recognition. In CVPR, 2012.

Dear Reviewer RkqC,

As the discussion period closes in less than 2 hours, we would be grateful if you could review our rebuttal and let us know whether our responses have addressed your concerns. We would be more than happy to clarify any remaining questions or provide further information.

Your feedback is very important to us, and we sincerely appreciate your time and thoughtful consideration.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.