- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes HyperRS to make aerial image classification reflect coarse-to-fine relationships between categories. It fine-tunes the pretrained hyperbolic image-text model MERU using a supplied label hierarchy: images and labels are aligned at multiple levels, ancestor-descendant relations are constrained, and finer categories are placed farther from the embedding origin. The experiments are mainly conducted on the Million-AID and RSI-CB datasets.
Critical strengths
A useful control for backbone and pretraining differences. CLIP and MERU use the same architecture and pretraining data (p. 6). Applying HierCon and HSC to both makes their comparison more informative than an unrestricted comparison of unrelated foundation models. The loss ablation identifies a concrete effect of radial supervision. Reporting that trade-off supports a specific geometric observation rather than claiming every objective improves everything.
Critical weaknesses
Remote sensing image classification is a well-studied task, and further accuracy gains on established benchmarks may have limited practical value. The paper should clarify which unresolved problems HyperRS addresses and why its improvements matter for real applications, beyond achieving higher benchmark scores.
The evaluation is not independent of model selection. Appendix C states “Neither dataset has a separate validation split,” and selects all loss weights using evaluation Hierarchical precision (p. 16). That makes the reported evaluation part of method development, rather than an untouched estimate of generalization. The claimed predictive increment needs independent evidence.
The paper conflates hyperbolic geometry with explicit hierarchy supervision and variable radius. The conclusion says Euclidean embeddings “essentially do not support” ancestor recovery (p. 9). However, HyperRS learns radial ordering, whereas the Euclidean controls use normalized cosine embeddings without equivalent radial supervision. Euclidean vectors can also have learned norms and ordered radii. The traversal result supports the evaluated configuration, but does not isolate negative curvature as the cause or establish the stated general limitation.
Questions for authors
See above.
6 comments
Reply to Reviewer EDQc (Part 2/4)
W3. Hyperbolic geometry conflated with explicit hierarchy supervision and variable radius.
Reply: Thank you for this precise point! Our experiments separate the factors the reviewer mentions as far as they allow:
- Hierarchy supervision is held fixed. All baselines in Table 3 are trained with the same three-level HierCon objective, and none uses a ranking or other radial term. Under this identical supervision, MERU recovers the whole ancestor chain for 22.6% of the images, against at most 2.9% for CLIP and RemoteCLIP. Hyperbolic embeddings, in which the distance to the origin is part of the geometry, are thus naturally suited to encoding the hierarchy.
- Radial supervision is isolated within the hyperbolic model. With the same backbone and HSC supervision, the ranking objective raises whole-path recovery from 24.0 to 37.7 (Table 5): explicit radial supervision strengthens a structure the hyperbolic model already exhibits without it.
We agree that Euclidean embeddings can learn variable norms and ordered radii, and that our statement that they “essentially do not support” ancestor recovery is too broad. Our original controls used normalized embeddings; we have therefore added a Euclidean model with learned radii (below) and will revise this statement based on it.
Our motivation for applying radial ranking in hyperbolic space concerns how the learned ordering interacts with the surrounding geometry. At a fixed dimension, hyperbolic space exhibits asymptotically exponential volume growth with geodesic radius, providing an inductive bias suited to the increasing number of categories at deeper taxonomy levels [1]. Moreover, HyperRS combines radial ordering with hyperbolic distance and radius-dependent entailment cones [2, 3]. Thus, radial position participates in both image–text alignment and hierarchical entailment. These properties motivate our choice of geometry; they do not establish an exclusive ability to encode hierarchy or guarantee superiority over Euclidean representations with equivalent supervision.
To compare Euclidean and hyperbolic geometry, we have added CLIP + rank and Hyperbolic + rank to Table 5 of the revised paper; both use the same HierCon loss and weights and the same ranking loss. For CLIP + rank, the ranking loss acts on the label-text embeddings before L2 normalization, so that their norms can be ordered. Its traversal starts from the image direction scaled to the largest text norm, moves toward the origin, and retrieves the nearest taxonomy node by Euclidean distance to the unnormalized text embeddings, analogous to the hyperbolic traversal. Million-AID, values in %, the better value in bold:
| Method | Acc(1) | Acc(2) | Acc(3) | hP | Trav. hP | LCS | H(d1) | H(d2) | H(d3) | H(all) | H(exact) |
|---|---|---|---|---|---|---|---|---|---|---|---|
| CLIP (w/ HierCon) + rank | 93.4 | 90.2 | 86.6 | 85.5 | 55.3 | 40.0 | 24.4 | 32.9 | 70.6 | 0.1 | 0.1 |
| Hyperbolic (w/ HierCon) + rank | 93.8 | 89.8 | 88.0 | 87.2 | 68.1 | 61.9 | 31.0 | 82.4 | 83.7 | 20.7 | 17.0 |
Hyperbolic + rank predicts slightly better, and the gap widens sharply in traversal: with ordered radii, CLIP + rank still finds the leaf but rarely its ancestors, and almost never recovers the whole chain. This further supports that hyperbolic geometry is better suited to encoding the hierarchy, even when Euclidean embeddings learn ordered radii.
References
[1] M. Nickel and D. Kiela. Poincaré embeddings for learning hierarchical representations. In NeurIPS, 2017.
[2] O.-E. Ganea, G. Bécigneul, and T. Hofmann. Hyperbolic entailment cones for learning hierarchical embeddings. In ICML, 2018.
[3] K. Desai, M. Nickel, T. Rajpurohit, J. Johnson, and R. Vedantam. Hyperbolic image-text representations. In ICML, 2023.
Reply to Reviewer EDQc (Part 3/4)
W1 (continued). New experiment 1: hierarchical retrieval.
To show the practical value of the learned hierarchy, we evaluated hierarchical retrieval on Million-AID (MA) and RSI-CB256 (RSI), comparing HyperRS with CLIP trained with HierCon. Values are in %; higher is better; the better value is in bold.
Setup. HP@k [4] is the sum of the similarities of the top- results divided by the best achievable sum, with similarity ; mAHP@K is the mean of [email protected]. Image to image: image queries, macro-averaged over classes. Image to text: image queries against the leaf-class texts (mAHP over all leaf texts). Text to image: the text of every taxonomy node as a query, one query per node. HP@k reflects the hierarchy only once exceeds the size of the query's own leaf class, since before that a perfect leaf-level ranking already scores 1. We therefore set the cutoffs per dataset: on MA (median test images per class: 265.5 at level 1, 22 at level 2, 20 at the leaf), = 10, 50, 100, 250 and mAHP@250; on RSI (3234 at level 1, 580 at the leaf), whose classes are more than ten times larger, = 100, 500, 1000, 2500 and mAHP@2500. The largest cutoff is about the size of a level-1 class.
| Task | MA metric | MA: CLIP | MA: HyperRS | RSI metric | RSI: CLIP | RSI: HyperRS |
|---|---|---|---|---|---|---|
| Image to image | HP@10 | 75.53 | 76.61 | HP@100 | 93.98 | 94.45 |
| HP@50 | 67.71 | 68.85 | HP@500 | 82.97 | 85.71 | |
| HP@100 | 65.84 | 67.04 | HP@1000 | 77.13 | 81.23 | |
| HP@250 | 63.05 | 64.59 | HP@2500 | 68.66 | 73.88 | |
| mAHP@250 | 66.34 | 67.57 | mAHP@2500 | 77.50 | 81.41 | |
| Image to text | HP@1 | 90.07 | 90.91 | HP@1 | 95.55 | 96.57 |
| HP@5 | 68.31 | 70.73 | HP@5 | 73.66 | 74.38 | |
| mAHP@51 | 88.61 | 89.29 | mAHP@35 | 90.81 | 91.03 | |
| Text to image | HP@10 | 91.72 | 92.67 | HP@100 | 98.54 | 98.95 |
| HP@50 | 79.56 | 80.47 | HP@500 | 92.55 | 93.37 | |
| HP@100 | 73.88 | 75.71 | HP@1000 | 85.71 | 85.79 | |
| HP@250 | 69.43 | 71.42 | HP@2500 | 77.75 | 77.22 | |
| mAHP@250 | 75.18 | 76.78 | mAHP@2500 | 85.83 | 85.87 |
HyperRS is higher at all but one of the 26 cutoffs. The largest gains appear in image-to-image retrieval at large , where neighbors beyond the query's own class must be ranked by the taxonomy.
References
[4] B. Barz and J. Denzler. Hierarchy-based image embeddings for semantic image retrieval. In WACV, 2019.
Reply to Reviewer EDQc (Part 1/4)
We thank the reviewer for the thoughtful review and for noting the value of the matched controls and the loss ablation.
W1. Unclear practical value of improvements on a well-studied task beyond benchmark scores.
Reply: Thank you for raising this point! We would like to clarify that, as motivated in Sec. 1, our paper addresses a task different from the well-studied flat aerial scene classification: hierarchical classification, where a model must predict a consistent label at every level of the taxonomy and should keep the correct coarse category when the fine label is wrong. This matters in remote sensing, where land-use products are defined at several granularities, and it is not measured by flat accuracy. Our evaluation therefore targets these properties, which Reviewers RkqC and c75V noted as a strength of the paper ("not only at classification accuracy" and "hierarchy beyond leaf accuracy"). We clarify the practical value of HyperRS in three aspects:
W2. Evaluation not independent of model selection, and no independent evidence for the predictive increment.
Reply: Thank you for this important concern!
The weight selection barely affects the reported performance. We would like to emphasize that a validation split mainly serves hyperparameter selection, and our setting requires very little of it. We did not tune the training hyperparameters: they follow common fine-tuning practice for CLIP and MERU and are identical for all methods (Sec. 4), without any extensive search. The only selected values are the loss weights introduced by our method, and once they are in a reasonable range, they barely affect performance:
Therefore, even without a separate validation split, the evaluation remains fair.
Predictive increment. MERU+HSC is HyperRS without these two objectives, and HSC is itself part of our contribution. We do not claim that the full model is more accurate than MERU+HSC and will revise observation 4 in Sec. 4.1 accordingly.
Evidence less affected by the selection. We therefore base our predictive claims on effects larger than the selection variation: multi-level supervision over leaf-only training, and hyperbolic over Euclidean geometry under the same supervision on Million-AID (+1.2 hP), where the selected weights also lie in the best range of the CLIP+HSC curve (on RSI-CB, the two are within this variation). CHA and traversal (Tables 2-3) were not selection criteria, as all weights were chosen by hP.