acceptodds
Under review as a conference paper at ICLR 2027

When Labels Have Structure: Improving Image Classification with Hierarchy-Aware Cross-Entropy

Abstract

Standard cross-entropy is the default classification loss across virtually all of machine learning, yet it treats all misclassifications equally, ignoring the semantic distances that a class hierarchy encodes. We propose Hierarchy-Aware Cross-Entropy (HACE), a drop-in replacement for standard cross-entropy that incorporates a known class hierarchy directly into the loss. HACE combines two components: prediction aggregation, which propagates the model's probability mass upward through the class hierarchy so that parent nodes accumulate the confidence of their children; and ancestral label smoothing, which distributes the ground-truth signal along the path from the true class to the root. We evaluate HACE on CIFAR-100, FGVC Aircraft, and NABirds in two regimes: end-to-end training across six architectures spanning convolutional and attention-based designs, and linear probing on frozen DINOv2-Large features. In end-to-end training, HACE improves top-1 accuracy over standard cross-entropy in 15 out of 18 architecture–dataset pairs, with a mean gain of 5.40 percentage points. In linear probing, HACE outperforms all competing methods on all three datasets, with a mean improvement of 2.18 points over the next best baseline. Beyond leaf-level accuracy, HACE also makes better mistakes: its predictions are more accurate at coarser levels of the taxonomy. Evaluated with hierarchical metrics on NABirds, it achieves the best operating curves and the smallest distance to the ground truth in the taxonomy in 5 of 6 architectures.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
hierarchical classification, class hierarchy, loss function design, cross-entropy, label smoothing, soft labels, image classification, fine-grained visual classification, taxonomy-aware learning, structured label spaces

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FCWsOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Hierarchy-Aware Cross-Entropy (HACE), which expands the classifier over all hierarchy nodes, aggregates raw softmax mass upward through a reachability matrix, and distributes target mass from each true leaf toward its ancestors. The method is evaluated on CIFAR-100, FGVC Aircraft, and NABirds in end-to-end training across six architectures and in linear probing with frozen DINOv2-Large features. The paper reports improved leaf accuracy in most settings and stronger hierarchical operating metrics, with especially large gains on deeper fine-grained taxonomies.

Critical strengths
  1. HACE is a simple, broadly usable loss modification with a clear construction and negligible implementation burden. Combining ancestor targets with monotone aggregated predictions is conceptually coherent.
  2. The evaluation covers several architectures, datasets, training regimes, and hierarchical metrics, and it directly compares with relevant hierarchy-aware losses. Improvements in both leaf accuracy and mistake severity would be practically meaningful if they are robust.
Critical weaknesses
  1. The headline tables select the best result across dilution values and two learning-rate pairings, while HACE and SCE do not receive an equal number of effective model-selection opportunities (the paper states six HACE and four SCE models per architecture-dataset pair). This can bias the comparison, especially because some SCE runs diverge and the mean gain is much larger than the median.
  2. Results appear to come from single runs per configuration, without seed variability or a held-out hyperparameter-selection protocol. Consequently, the large reported average improvements cannot be separated cleanly from optimization instability and selection noise. The experiments also omit the important pretrained fine-tuning regime.
Questions for authors
  1. Under an equal tuning budget and a validation-selected dilution/learning-rate rule fixed before test evaluation, what are the paired HACE-versus-SCE results across multiple seeds? This answer could change my recommendation.
  2. How much of the gain remains when both losses are initialized from the same pretrained backbone and fine-tuned with identical schedules? The current from-scratch setting is unusually unstable for the fine-grained datasets and transformer models.
3 comments

We thank the reviewer for the constructive feedback and for recognizing the simplicity of HACE and the breadth of the experimental evaluation. We answer both questions below.

  1. Tuning budget and model selection. We believe the six-versus-four count may have given the impression that HACE received more selection opportunities in the comparison, and we clarify the design. Each HACE model is paired with exactly one SCE model trained at the matched learning rate (Appendix B.3). The SCE count is four rather than six only because, in the SCE-anchored pairing, SCE is trained at the base rate ℓ regardless of d, so a single SCE model serves as the matched counterpart for all three dilution values. The paired comparisons in Appendix Tables 8 and 9 therefore involve no selection at all: every HACE model is compared with its own SCE counterpart. HACE outperforms SCE in 17 of 18 paired linear-probing configurations (Table 9) and in 32 of 36 paired end-to-end configurations on FGVC (Table 8). The best-of results in Tables 2 and 5 are a summary; the gains do not depend on the aggregation. We are happy to provide the corresponding paired comparisons for the end-to-end results on CIFAR-100 and NABirds if the reviewer finds them useful; we initially omitted them to avoid adding several tables that convey the same message.

Regarding seeds, the original experiments used one seed per configuration. We have now run three seeds for DINOv2 linear probing on NABirds (see the table in our response to Reviewer JLqo) and for the new fine-tuning experiments described below.

  1. Pretrained fine-tuning. We fine-tuned pretrained ResNet-50 and Swin-T end-to-end on NABirds, with both losses initialized from the same backbone and trained with identical schedules (table in our response to Reviewer VTko). HACE outperforms SCE in all six matched configurations for both architectures. We plan to extend these experiments to the remaining models and datasets.

We sincerely thank the reviewer once again for their thoughtful and constructive feedback. We believe that the additional experiments and clarifications prompted by their suggestions strengthen our work. We kindly ask for further feedback, and we are happy to engage in more discussion.

Reviewer_FCWsOpenReview ↗

Response by Reviewer FCWs

Thank you for your responses. My concerns have been addressed. I will raise my socre.

Reviewer_VTkoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper asks whether a classifier should be trained differently when its labels belong to a known taxonomy—for example, when mistaking one bird species for a related species should count differently from mistaking it for an aircraft. Its proposed loss, HACE, teaches the model about both the exact leaf class and the broader categories above it: confidence assigned to a fine-grained class also supports its ancestors, while the target signal is shared along the path from the correct class to the root. The intuition is that a model should learn which branch of the hierarchy an example belongs to even when it cannot resolve the finest distinction. The authors test this idea on three hierarchical image datasets, both when training several architectures from scratch and when fitting a linear classifier to frozen DINOv2 features. The paper reports improvements in leaf accuracy and coarse-level hierarchical metrics across most tested configurations and interprets these results as evidence that an available label hierarchy can provide useful supervision rather than serving only as an evaluation tool.

Critical strengths
  1. A simple loss with modest implementation changes:HACE uses one fixed matrix product and a precomputed target table while expanding the output layer from leaves to all hierarchy nodes. Prediction aggregation enforces the hierarchical constraint that every parent score is at least as large as its descendants, and the authors sketch a mechanically defined extension to single-rooted DAGs. If the gains hold under a fair protocol, the method could be practical for moderate-sized taxonomies without changing the backbone architecture.

  2. Broad empirical scope and evaluations: The study covers six architectures, three hierarchies, two training regimes, and several competing objectives, while also evaluating coarse-level behavior and hierarchical operating curves rather than top-1 accuracy alone. This breadth identifies several settings in which HACE may be promising and provides a useful basis for a more tightly controlled evaluation.

Critical weaknesses
  1. The experimental protocol cannot establish that the reported gains come from HACE: The headline tables select the best of six HACE configurations versus four SCE configurations (Tables 2 and 5), but the paper does not state whether these choices use a validation set and reports neither repeated seeds nor uncertainty. Several hierarchy-aware baselines collapse on ConvNeXt-T (Table 3), further weakening the best-of-configuration comparisons; validation-based selection and repeated runs are therefore needed to establish the accuracy gains. Table 4's operating-curve metrics provide meaningful evidence that HACE improves coarse-to-fine hierarchical decisions. However, F1 (Leaf) and hierarchical distance are averaged over all samples and therefore do not isolate severity conditional on a misclassified leaf, and the paper does not report the standard error-conditioned mistake severity or severity distributions. This omission is notable for a paper framed around making better mistakes and leaves the narrower claim that HACE's errors “land closer” insufficiently examined, even though it does not invalidate the broader hierarchical results.

  2. Evaluation on deep hierarchies and against relevant methods is necessary to establish the claimed generality: All three evaluated taxonomies have only two to four levels below the root, but hierarchy depth directly changes HACE's objective: for ε=0\varepsilon=0, an ancestor jj edges above the leaf receives weight d(1−d)jd(1-d)^j, and a depth-kk path leaves only 1−(1−d)k1-(1-d)^k target mass on non-root terms that can produce a gradient. The current experiments therefore test only a short-path regime and cannot establish that the same dilution values, optimization behavior, or gains persist on deeper taxonomies. Established benchmarks such as iNaturalist2019 and tieredImageNet-H provide hierarchies of roughly 7 and 12 levels, respectively; the paper's generic statement that larger iNaturalist or ImageNet taxonomies are future work does not substitute for this evidence. The comparison is also incomplete: it includes HXE and soft labels from Bertinetto et al. (2020), but no experimental comparison with CRM (Karthik et al., 2021) and no discussion of HAFrame (Liang & Davis, 2023). Although these methods intervene at different stages of the pipeline, both directly address the paper's central claim of reducing mistake severity. Without deeper hierarchical benchmarks and these comparisons, HACE's behavior on realistic hierarchies and its position against relevant prior work remain unresolved.

Questions for authors
  1. Model selection and variance: Did the experiments use an unreported validation split to choose dd, the learning-rate pairing, and baseline hyperparameters, or were the best variants in Tables 2–5 selected using the reported test metrics? How many independent random seeds were run per configuration?
  2. Interpretation of the ancestral target: Prediction aggregation enforces that every parent score is at least as large as its descendant scores, whereas, without horizontal label smoothing, ancestral label smoothing assigns geometrically decreasing target mass as one moves from the ground-truth leaf toward its non-root ancestors. The resulting target and prediction vectors therefore cannot generally match coordinate by coordinate. Should the ancestral target in Equation (1) be interpreted as a probability distribution that the aggregated prediction is meant to match, or instead as a set of weights over nested log-probability terms? If the latter, would it be more precise to frame HACE as a hierarchy-aware weighted log-loss over nested events rather than as a hierarchy-aware cross-entropy between two distributions?
4 comments

We thank the reviewer for the detailed feedback and for recognizing the simplicity of HACE and the breadth of the experimental evaluation. We address the main concerns below.

1. Model selection and variability. Given the breadth of the evaluation, covering six architectures, three datasets, multiple competing objectives, and numerous training configurations, extensive hyperparameter tuning for every setting would have been computationally prohibitive. We therefore followed the architectures' recommended training recipes and evaluated all models on the last training epoch checkpoint.

Regarding the number of random seeds, the original experiments used a single seed per configuration. We have now extended the fine-tuning experiments to three independent seeds. The pretrained ResNet-50 results on NABirds are reported in our response to Reviewer JLqo, while the DINOv2 results on the NAbirds dataset are presented below. HACE outperforms or matches SCE in all matched ResNet-50 configurations and all DINOv2 configurations, with consistently small standard deviations. We plan to have 3 seeds for all model/dataset pairs in the fine tuning setting.

HACE settingHACE Top-1HACE Top-5SCE Top-1SCE Top-5
d=0.268.63 ± 0.0790.60 ± 0.0368.68 ± 0.0291.77 ± 0.01
d=0.2 adj84.93 ± 0.0297.58 ± 0.0083.45 ± 0.0197.21 ± 0.01
d=0.580.15 ± 0.0196.11 ± 0.0078.55 ± 0.0395.99 ± 0.01
d=0.5 adj84.68 ± 0.0397.51 ± 0.0183.45 ± 0.0197.21 ± 0.01
d=0.782.18 ± 0.0496.83 ± 0.0181.16 ± 0.0296.66 ± 0.00
d=0.7 adj84.32 ± 0.0397.43 ± 0.0183.45 ± 0.0197.21 ± 0.01

2. Mistake severity. We thank the reviewer for this point, and we would like to emphasize that the hierarchical evaluation goes substantially beyond mean hierarchical distance. Section 5.3 and Table 4 report several complementary metrics, including coarse-level accuracy, hierarchical precision-recall curves, correct-versus-exact operating curves, and recall at fixed correctness thresholds.

We acknowledge that the paper reports the mean hierarchical distance over all test samples (DallD_{\mathrm{all}}), rather than the error-conditioned mistake severity (DerrD_{\mathrm{err}}). The two are related exactly by $ D_{\mathrm{all}} = (1 - \text{top-1 accuracy})\,D_{\mathrm{err}}. $

We use DallD_{\mathrm{all}} because it is the expected tree-distance cost of the classifier. DerrD_{\mathrm{err}} alone does not measure classifier quality, because it ignores how many mistakes are made: a model can lower it by turning correct predictions into mild errors.

For example, a model that is correct on 99% of samples and makes distant errors (distance 6) on the remaining 1% has Derr=6D_{\mathrm{err}} = 6. A model that is correct on only 50%, with 48% sibling errors (distance 2) and 2% distant errors, has Derr≈2.2D_{\mathrm{err}} \approx 2.2. DerrD_{\mathrm{err}} ranks the second model as better, although it makes fifty times as many errors and twice as many distant ones, while DallD_{\mathrm{all}} ranks the two correctly (0.06 versus 1.08).

3. Hierarchy depth and related methods. We agree that experiments on deeper hierarchies would strengthen the evidence for generality. However, the claim that non-root supervision vanishes with depth is not correct: for a path of depth kk, its total target weight is 1−(1−d)k1-(1-d)^k, which increases with depth, while the correct leaf retains weight dd. This does not establish performance on deeper taxonomies, but it rules out the proposed vanishing-target-mass argument.

CRM (Karthik et al., 2021) is already discussed in Section 2. It is an inference-time post-processing method and is therefore complementary to HACE rather than a competing training objective. HAFrame (Liang & Davis, 2023) modifies classifier geometry and adds a representation-learning objective; we will include it in related work. HACE instead directly replaces cross-entropy without changing the backbone. The comparisons deliberately hold the architecture and training setup fixed while varying the loss, making alternative training objectives the appropriate controls for isolating HACE's contribution.

4. Interpretation of the objective. We thank the reviewer for this helpful observation. Aggregated predictions represent probabilities of nested events, not a normalized distribution over mutually exclusive classes. The ancestral targets are therefore best interpreted as weights on negative log-probabilities of nested events, rather than probabilities that aggregated predictions should match coordinate by coordinate. We will clarify that Equation (1) defines a hierarchy-aware weighted log-loss, rather than conventional cross-entropy between two normalized distributions.

We sincerely thank the reviewer once again for their thoughtful and constructive feedback. We believe that the additional experiments and clarifications prompted by their suggestions strengthen our work. We kindly ask for further feedback, and we are happy to engage in more discussion.

Thank you for the detailed and constructive response. I appreciate the clarification on the objective.

On the hierarchy depth question, my original phrasing invited a different reading. My concern is where that mass sits. Because the ancestral weights decay geometrically per level rather than being normalised over the taxonomy, making the tree deeper does not push supervision further up: most of the ancestral mass sits within the few levels above the leaf, and the share reaching the coarse levels shrinks as depth grows. What makes this more than a question of tuning the hyperparameter is that a single scalar sets both the leaf's own target mass and how many levels are reached: covering the coarse levels of a twelve-level taxonomy would require a much smaller dd than any value you report, and since the leaf-level gradient scales with dd, the same change would weaken the leaf-level training signal. Whether a workable setting exists on deeper trees — and whether the loss then still behaves as a hierarchy-aware objective rather than as cross-entropy with light ancestral smoothing — is exactly what evaluation on iNaturalist2019 and tieredImageNet-H would settle, a gap the response itself acknowledges. I am not asking for these experiments within the discussion period, but without them the generality claim is supported only at the shallow end of the depth range.

As for the mistake severity metric, I accept the authors' point that DerrD_{\text{err}} alone can be gamed by trading correct predictions for low-severity errors. I still think a severity histogram of the kind reported in Bertinetto et al. (2020,https://arxiv.org/pdf/1912.09393) and Karthik et al. (2021,https://arxiv.org/pdf/2104.00795) would give readers a complementary picture of model performance and of the influence of a hierarchy-aware approach on reducing mistake severity.

Thank you for the thoughtful follow-up. We agree that identifying the optimal way to propagate supervision through a hierarchy, particularly as its depth increases, remains an open question. Our goal here is not to claim that geometric propagation is optimal, but rather to demonstrate that even a simple mechanism for incorporating hierarchical information can substantially improve classification performance across architectures, datasets, and training regimes.

We would also like to emphasize the breadth of our experimental evaluation. We consider three datasets (CIFAR-100, FGVC Aircraft, and NABirds), six architectures spanning CNNs and transformers, and three learning settings: training from scratch, linear probing on frozen pretrained representations, and end-to-end fine-tuning of pretrained models. We additionally compare against several hierarchy-aware objectives, including HXE, structured soft labels, and Margin Softmax. While we acknowledge that evaluating deeper taxonomies such as iNaturalist2019 and tieredImageNet-H would be valuable, we believe the current evidence already demonstrates the effectiveness of the proposed approach across a broad range of settings.

Regarding mistake severity, we agree that a severity histogram would provide a useful complementary perspective to the hierarchical metrics already reported. We would be happy to include this analysis in a revised version.

Reviewer_JLqoOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper investigates how known class hierarchies can be incorporated into the cross-entropy objective for image classification. The authors argue that standard cross-entropy does not explicitly account for the hierarchical severity of different classification mistakes. They propose Hierarchy-Aware Cross-Entropy (HACE), combining prediction aggregation, which propagates confidence from descendants to ancestors, with ancestral label smoothing, which distributes supervision along the ground-truth class’s ancestral path. The method is evaluated on CIFAR-100, FGVC Aircraft, and NABirds, using six architectures in end-to-end training and linear probing on frozen DINOv2-Large features. The reported results show improvements in top-1 accuracy in most settings, together with favorable hierarchical error metrics.

Critical strengths
  1. A simple and practical training objective. HACE incorporates hierarchical information directly into a cross-entropy-style loss without requiring a complicated additional model architecture. Prediction aggregation and ancestral label smoothing have clear and complementary roles.

  2. The study considers coarse-level accuracy, mistake severity, and hierarchical operating curves in addition to conventional classification performance. Experiments span multiple network architectures, three datasets, and a frozen-feature linear-probing setting, with comparisons against existing hierarchy-aware losses.

Critical weaknesses
  1. Experimental settings limit the interpretation of results for modern pretrained classifiers. All end-to-end models are trained from scratch, and some architectures achieve weak standard-cross-entropy baselines on the smaller FGVC Aircraft and NABirds datasets. For example, Table 2 reports 30.45% accuracy for ConvNeXt-T and 15.72% for ViT-B/16 on FGVC Aircraft with standard cross-entropy. These settings may amplify differences in optimization stability or training dynamics between loss functions.

  2. The source of the accuracy improvements is not fully isolated. HACE modifies both the target distribution and the structure of predicted probabilities, while the ancestral smoothing parameter can also affect the scale of leaf-level gradients. The paper does not adequately examine whether the method remains effective when the assumed hierarchy is inaccurate or perturbed. This limits the interpretation of its central mechanism.

  3. Limited methodological novelty and insufficient isolation of contributions。The two core components of HACE, prediction aggregation and ancestral label smoothing, build on established ideas in hierarchical classification. Although their combination yields a simple and practical loss function, the paper does not sufficiently clarify its distinctive methodological contribution relative to existing hierarchy-aware losses.

  4. HACE encourages predictions to respect class hierarchies, but this may introduce a trade-off between hierarchical consistency and fine-grained discrimination. Although the paper reports top-1 accuracy and hierarchical error metrics, it does not sufficiently analyze how hierarchy-aware supervision affects classification performance at different levels, particularly the discrimination between closely related sibling classes.

Questions for authors
  1. Does HACE still improve performance when pretrained ResNet, ConvNeXt, or Vision Transformer models are fine-tuned end-to-end, especially when the standard-cross-entropy baseline is already strong?

  2. Could the authors perturb or randomly reassign the hierarchy while preserving its depth and other structural properties, or compare against gradient-scale-matched non-hierarchical regularization, to better isolate the role of meaningful hierarchical information?

5 comments

We thank the reviewer for recognizing HACE's practical value and the breadth of its evaluation.

1. Pretrained models and strong baselines. To directly address Question 1, we have completed end-to-end fine-tuning experiments on NABirds with pretrained ResNet-50 and Swin-T. The table below reports the ResNet-50 results across three seeds, showing that HACE outperforms SCE in all six matched configurations, with improvements in both top-1 and top-5 accuracy.

HACE settingHACE Top-1HACE Top-5HACE (edges deleted) Top-1HACE (edges deleted) Top-5SCE Top-1SCE Top-5
d=0.273.35 ± 0.0493.32 ± 0.0674.95 ± 0.2293.56 ± 0.1169.90 ± 0.0791.35 ± 0.02
d=0.2 adj77.82 ± 0.2094.55 ± 0.1856.67 ± 10.9878.95 ± 8.3275.40 ± 0.0892.94 ± 0.10
d=0.575.05 ± 0.0793.75 ± 0.0975.30 ± 0.1493.72 ± 0.1173.36 ± 0.1292.48 ± 0.09
d=0.5 adj76.80 ± 0.1294.07 ± 0.2077.02 ± 0.1394.06 ± 0.0875.40 ± 0.0892.94 ± 0.10
d=0.775.34 ± 0.1093.68 ± 0.0975.34 ± 0.1193.68 ± 0.1174.38 ± 0.1592.70 ± 0.06
d=0.7 adj76.21 ± 0.1493.94 ± 0.0776.31 ± 0.1693.93 ± 0.0775.40 ± 0.0892.94 ± 0.10

We observe similarly consistent gains with pretrained Swin-T, where HACE also outperforms SCE in all six matched configurations (one seed, with additional runs underway).

We would also like to highlight that the original paper evaluates pretrained DINOv2-Large features (Section 5.4, Table 5) to isolate the effect of the loss from representation learning. HACE outperforms all competing objectives across all three datasets, including when SCE already achieves high accuracy.

2. Gradient scaling and hierarchy perturbations. To directly address Question 2, we investigate whether HACE can benefit from partial hierarchical information, rather than requiring a complete hierarchy. Since SCE corresponds to the flat, leaf-only case, removing edges tests how much hierarchical information is needed to retain HACE's benefits.

Specifically, we remove 25% of the hierarchy edges on NABirds and fine-tune pretrained ResNet-50 across three seeds. Even with this incomplete hierarchy, HACE outperforms SCE in 5 of 6 matched configurations (see table above). Overall, these results suggest that even incomplete hierarchical information can provide useful supervision.

The effect of ancestral smoothing on leaf-level gradients is explicitly discussed and accounted for through two learning-rate pairings (Appendix B.3). HACE outperforms SCE in 17 of 18 matched DINOv2 linear-probing configurations (Table 9). While these pairings account for the direct leaf-level gradient contribution rather than exactly matching full gradient norms, comparisons against standard label smoothing and hierarchy-aware soft labels further distinguish the benefits of hierarchical supervision from ordinary smoothing.

3. Methodological novelty. The contribution is not prediction aggregation or label smoothing in isolation, but their combination into a hierarchy-consistent objective under leaf-only supervision. Neither component alone provides the intended hierarchical supervision, as explained in Section 3. Aggregation alone does not explicitly supervise ancestors, while ancestral smoothing alone treats parents and children as competing outcomes. HACE resolves both issues simultaneously.

Unlike approaches that redistribute supervision among leaves, HACE explicitly supervises ancestral nodes while enforcing consistency of predicted probabilities. This also distinguishes it from earlier hierarchical cross-entropy work using labels available at multiple levels.

4. Fine-grained discrimination. The concern about a potential trade-off is reasonable, but the results suggest otherwise: HACE improves leaf-level accuracy in 15 of 18 original architecture and dataset pairs, while also improving coarse-level accuracy and hierarchical error metrics. The new pretrained fine-tuning experiments further support this finding, with improvements in both top-1 and top-5 accuracy. Thus, hierarchical supervision does not generally come at the expense of fine-grained classification in these experiments. We agree that a breakdown of errors between sibling classes would complement these results.

5. Presentation. We appreciate the concern about clarity. Section 3 introduces the methodology step by step, with a visual illustration (Figure 1), mathematical definitions, and explanations of why both components are necessary. We would welcome suggestions about specific passages that remain difficult to follow and will clarify them accordingly.

Thank you for the response. The new results from fine tuning pretrained models, particularly those for ResNet50 on NABirds, partly ease my concern that the reported gains may reflect unstable optimization when training from scratch. I also appreciate the explanation of how the two components work together. However, I still have two concerns about the source of the improvements. First, deleting hierarchy edges mainly tests performance with incomplete information, rather than the effect of incorrect hierarchical relationships. Please also explain the substantial drop in accuracy and increase in variance after edge deletion in the d = 0.2 adj configuration. Second, the learning rate pairings account for the direct gradient contribution of the leaf supervision term, but do not match the overall optimization dynamics. It therefore remains difficult to distinguish gains from meaningful hierarchical information from those due to gradient scaling or regularization.

We thank the reviewer for the constructive follow-up and are glad that the pretrained fine-tuning experiments help address the concerns about weak baselines.

We appreciate the suggested experiment and have now conducted an additional ablation on CIFAR-100 using ResNet-18, randomly reassigning 10% of leaf classes while preserving the hierarchy structure. The results are reported below. Remarkably, even with incorrect hierarchical relationships, HACE outperforms SCE in all six matched configurations, in both top-1 and top-5 accuracy. Performance is generally slightly lower than with the correct hierarchy, suggesting that meaningful hierarchical relationships contribute to the gains, while the method remains effective under moderate corruption.

HACE settingHACE Top-1HACE Top-5Leaves permuted Top-1Leaves permuted Top-5SCE Top-1SCE Top-5
d=0.277.0994.0276.6793.3975.0992.13
d=0.2 adj77.6493.8377.4093.8176.6792.70
d=0.577.1693.5676.6493.1475.7892.76
d=0.5 adj76.4593.3677.0592.9376.6792.70
d=0.776.5793.1676.3693.0076.0992.73
d=0.7 adj77.5193.0976.9093.2176.6792.70

We plan to include this ablation in the revised manuscript and extend it to a vision transformer.

Regarding the earlier edge-deletion experiment, our motivation was to investigate how much hierarchical information is needed to improve upon standard cross-entropy, rather than whether the hierarchy must be perfectly accurate. Since SCE represents the flat case without ancestral supervision, removing edges provides a natural way to study the benefits of incomplete hierarchies. The instability at (d=0.2) may reflect the combination of strong ancestral supervision (80% of target weight lies above the leaf) and the fivefold learning-rate adjustment, which could destabilize optimization when the hierarchy is perturbed.

Taken together, our experiments provide consistent evidence that incorporating hierarchical information into the loss improves classification performance across training from scratch, linear probing on frozen features, and end-to-end fine-tuning, spanning six architectures and three datasets. The additional experiments show that these benefits are largely preserved even with incomplete or partially incorrect hierarchies.

Comments on the paper

1 comment

Dear Reviewers and Area Chairs,

We thank the reviewers for their careful assessments and constructive feedback. We appreciate their recognition of HACE's simple and practical formulation, broad empirical evaluation, and improvements in both classification accuracy and hierarchical metrics.

We believe HACE makes a simple but impactful contribution: a minimal modification to standard cross-entropy, requiring no changes to the backbone architecture, that consistently improves classification performance across three datasets, six architectures, and three training regimes.

During the discussion period, we substantially strengthened the experimental evidence in response to the reviewers' concerns:

  1. Pretrained fine-tuning: We introduced end-to-end fine-tuning experiments with pretrained ResNet-50 and Swin-T on NABirds. HACE outperforms SCE in all six learning-rate-matched configurations for both architectures, including improvements of +2.42 and +4.21 percentage points in top-1 accuracy, respectively, over strong SCE baselines.
  2. Hierarchy ablations: We investigated the role of hierarchical information by removing 25% of hierarchy edges and randomly reassigning 10% of leaf classes. HACE retains its improvements over SCE in nearly all configurations, while the correct hierarchy generally performs better than the corrupted one, supporting the contribution of meaningful hierarchical relationships.
  3. Multiple seeds: We extended the pretrained ResNet-50 fine-tuning and DINOv2 linear-probing experiments to three independent seeds, observing consistently small variance and confirming the robustness of the improvements.

Together with the original results, these experiments provide strong evidence for the effectiveness of HACE across training from scratch, linear probing on frozen representations, and end-to-end fine-tuning of pretrained models.

While further evaluation on deeper hierarchies remains valuable, we believe the simplicity of the method, consistency of the improvements, and substantial additional experimental evidence make a compelling case for its contribution.

We thank the reviewers once again for their constructive engagement and hope that these additions will be reflected in their final assessments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.