acceptodds
Under review as a conference paper at ICLR 2027

TreeSchism: Localizing Ambiguity in Hierarchies

Abstract

In classification, some inputs do not support fine-grained labels. For example, a blurry photo may show that a bird is a hawk, but not which type of hawk. A classifier presented with such an input must either guess its species (possibly communicating something false) or abstain (communicating nothing). A hierarchy of classes affords a third option: name the deepest node that the image justifies. Existing hierarchical predictors either assume that ground-truth labels are leaves or can only reason about two-level taxonomies. We introduce TreeSchism, which adapts evidential deep learning to localize ambiguity to a node of an arbitrarily deep taxonomy. A TreeSchism model outputs scores at every node and accumulates them upward. This parameterization yields the refinement gap, the support for a node that does not apply to any of its descendants. We train the refinement gap to peak at the deepest label each sample warrants, and use it at inference to select a prediction granularity. Across seven taxonomies, TreeSchism matches hyper-evidential networks on the two-level taxonomies to which they are confined, improves exact-node accuracy over the strongest baseline by 6 to 21 points on deeper taxonomies, and achieves these successes without using held-out calibration data.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
hierarchical classification, image classification, class hierarchy, ambiguity, deep learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_22CSOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a hierarchical classification method called TREESCHISM, which addresses the issue that ordinary classifiers can only perform precise classification but cannot make hierarchical judgments. TREESCHISM enables the model to output the intermediate nodes in the classification tree, that is, to select "the deepest level that the current evidence can support". The method is based on evidence deep learning. The model generates local evidence for each node, accumulates the descendant evidence towards the ancestors and calculates the support degree, and makes the judgment through the threshold.

Critical strengths

The method can output intermediate levels instead of only leaf nodes; In terms of the metric "determining at which layer to stop", some experiments outperformed the baseline; The refinement gap is more suitable than ordinary classification probabilities as a stopping signal.

Critical weaknesses

I found the paper did not do related work research throughly, I know at least one paper that is highly related to this method or claim. But I did not find the citation.

The evidence theory and EDL are methods with strong theoretical basis. The article makes several arbitrary changes to them in order to achieve better empirical results, but it is difficult to maintain their theoretical completeness and lacks theoretical interpretability.

The advantage of EDL lies in being able to obtain the prediction results as well as the uncertainty of the model's prediction, aka, evidence vacuity. However, the article does not utilize evidence vacuity. Instead, it designs its own indicator, which makes me question whether the author has a sufficient understanding of EDL.

There are also unfair aspects in the utilization of data. Randomly/uniformly subdividing high-level label data to low-level label data to train models will lead to semantic deviations, thereby affecting the effectiveness of the comparison model.

Questions for authors

Are there any essential differences between this method and method H-MEAN[1]? Apart from adding an additional threshold for judgment. By the way, the H-MEAN paper is not mentioned in the article either.

What is the theoretical basis for using a Dirichlet distribution at each level? If one wants to use the evidence of the same level and the evidence of its child nodes, then the idea of hyper opinions should be used, and it should support both composite sets and singletons.

Why don't use evidence vacuity as the key to decide hierarchical level?

Is it meaningful to gain better results solely at the prediction level? Each case definitely has the label to the leaf node, it's just that the accuracy of judgment is different. Precise prediction, this is exactly what the model hopes to achieve. We can represent the confidence of the decision through uncertainty and accumulate the confidence of each level instead of directly judging that the decision cannot continue at this level.

3 comments
AuthorsOpenReview ↗

Response to Reviewer 22CS

Thank you for your feedback on our paper. We will respond to each of your concerns below.

Prior work:

By H-MEAN, we think the reviewer was referring to Li et al. [1]; please let us know if we have mis-identified it. We believe this work is only tangentially related to ours. Like us, they take an approach to hierarchical classification grounded in subjective logic and similar notions. However, their contribution lies in aggregating evidence from multiple sources. Unlike our work, they have no notion of hierarchical ambiguity or anything analogous to the refinement gap, and no mechanism to predict non-leaf nodes.

Regarding the other related work you mentioned: to our knowledge, no existing knowledge models hierarchical ambiguity and localizes it to the appropriate node in taxonomies of arbitrary depth. That said, we would be grateful if you could share the specific reference you have in mind. We’d be glad to discuss it here and to cite and compare against it in a revision.

Theoretical basis, arbitrary changes:

Our method was originally developed through Dempster-Shafer theory, but we decided to present it through Dirichlet semantics to make our paper simpler and more accessible. Given reviewer interest in its theoretical grounding, however, we added a discussion of a correspondence between our method and Dempster-Shafer theory in Appendix F. Each of our primary notions (accumulated evidence, the refinement gap, etc.) has an analogue in Dempster-Shafer theory that admits the same interpretation.

Even without the Dempster-Shafer interpretation, we believe that our modifications to the standard EDL recipe are not totally arbitrary. The refinement gap does a better job of localizing ambiguity than vacuity (Table 3), our gradient analysis shows why our loss is effective in practice (Appendix E), and the construction is simple and interpretable.

Why didn’t we use vacuity?

Vacuity scales with the number of classes. As we walk up a taxonomy, the number of classes at each level decreases, so vacuity measurements from two different taxonomic levels are not directly comparable. Vacuity is therefore less well-suited to hierarchical inference than the refinement gap. We added vacuity to Table 3, confirming empirically that it cannot identify what depth to predict at as well as the refinement gap.

Unfair aspects in the utilization of data:

We believe this criticism is targeted towards our SoftCE baseline. We note that our other baselines (HENN, HRN, CHILLAX) all integrate coarse taxonomic labels during training. The fact that SoftCE outperforms these is not evidence that our evaluation was unfair, but rather an interesting discovery in its own right.

Why use separate Dirichlets per level rather than a hyper-opinion?

Simple proofs show that our method admits an equivalent formulation in terms of Dempster-Shafer theory, and the refinement gap at a node n can be interpreted as the mass on n after conditioning on membership in n’s subtree. Please see Appendix G in our revised draft for details.

Why predict interior nodes at all?

Other works have already considered the setting where each instance is understood to have a leaf label, and coarse labels in training data arise only from incomplete supervision. Two such methods (HRN, CHILLAX) served as baselines in our evaluation. We consider a different problem setting from them, in which non-leaf labels arise because any finer label is actively wrong. For example, in the BIOSCAN-5M dataset, some samples have non-leaf labels because their genera and/or species do not exist in the tree for the simple reason that no one has named their species yet.

[1] Y. Li, Y. Chen, W. Liu, J. Qu, C. Ma and X. Yue, "H-MEAN: Hierarchical Multi-view Evidence Aggregation Network for Trustworthy Medical Image Classification," 2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Wuhan, China, 2025, pp. 2423-2428, doi: 10.1109/BIBM66473.2025.11356753.

After carefully reading the authors’ response, I remain unconvinced, and my original concerns have, if anything, become more pronounced. The response does not directly address the central issues I raised, while several key points remain insufficiently discussed. My concerns are as follows:

  1. Identification at arbitrary hierarchical levels. Any EDL-based method can trace evidence upward through the hierarchy, accumulate evidence, and thereby obtain uncertainty estimates at different hierarchical levels.

  2. Motivation for modifying EDL and excluding vacuity. The authors’ explanation of these choices further reinforces my concern that the proposed modifications are primarily empirically motivated rather than derived from a clear theoretical foundation.

  3. Relationship between Dirichlet priors and evidence theory. The authors explain their motivation from the perspectives of both the Dirichlet prior distribution and evidence theory. Although this may appear reasonable at first glance, these two frameworks have fundamentally different starting points and interpretations; the fact that they lead to similar mathematical forms does not establish a substantive theoretical connection. Therefore, using this relationship to motivate the proposed method does not appear sufficiently justified.

I would appreciate it if the authors could address these concerns more directly and provide a more rigorous theoretical justification for the proposed design.

On “Any EDL method can accumulate evidence upward”:

This is true, but does not suffice for our problem setting. In leaf-only EDL with upward accumulation, internal nodes have no local evidence and the refinement gap is identically zero. It would be impossible for a sample to carry evidence at a genus that is not also evidence for some species.

Accumulating scores upwards would produce an EDL analogue to our SoftCE+HSC baseline. Our Table 3 shows that SoftCE+HSC’s stopping signal is weaker than the refinement gap for granularity prediction. In this setting, we need to place un-refinable evidence on interior nodes; it is both the right modelling choice and more effective empirically. We note that Li et al. [2] (the HENN authors, not the H-MEAN authors [1]) evaluated the EDL/upwards-accumulation approach in this setting and found that it underperformed other methods.

On “Excluding vacuity is empirically motivated”:

Vacuity and hierarchical ambiguity are different types of uncertainty. Vacuity is high when a model has little overall evidence about an input. Conversely, hierarchical ambiguity is a product of a sample rather than of a model’s knowledge. A well-trained model may hold abundant evidence that a sample belongs to a genus and be equally confident that no species label is warranted. Vacuity should be high when a model is unconfident, but a model can be confident that no species label is warranted.

Evidence accumulation does not change this. In ordinary EDL, if we accumulate evidence upward, we preserve the total evidence, so vacuity at each level is the same per-sample scalar, rescaled by the class count at that level. It communicates a model’s overall level of confidence, but cannot communicate where in the taxonomy evidence stops being refined.

The refinement gap solves this problem. Vacuity is the mass on an entire frame that cannot be refined to any proper subset. Lemma G.9 shows that the refinement gap has the same interpretation after conditioning on membership in a node’s subtree - it is exactly the mass on the entire sub-tree that cannot be refined to any proper sub-tree. Our paper speaks of localizing ambiguity to different nodes, but we could instead have said "localizing vacuity". It is precisely our formulation’s ability to quantify vacuity differently at different subtrees that allows it to determine the appropriate prediction depths. Li et al. [2] separate vagueness from vacuity for the same reason.

On “Dirichlet and evidence theory merely share a form”:

The correspondence between Dirichlets and evidence theory does not originate with us. It is used in Subjective Logic and underpins EDL. We are not creating a new correspondence, but rather specializing an existing one to a new problem setting. We are happy to discuss any questions or concerns you have about the re-interpretation of our method in evidential terms.

Other notes:

We realized that our original reply to you mis-identified some appendices, and we apologize for any confusion. The comparison between our method and Dempster-Shafer theory is in Appendix G, not F; and the gradient analysis in Appendix F, not E.

[1] Y. Li, Y. Chen, W. Liu, J. Qu, C. Ma and X. Yue, "H-MEAN: Hierarchical Multi-view Evidence Aggregation Network for Trustworthy Medical Image Classification," 2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Wuhan, China, 2025, pp. 2423-2428, doi: 10.1109/BIBM66473.2025.11356753.

[2] C. Li et al., “Hyper evidential deep learning to quantify composite classification uncertainty,” arXiv:2404.10980, 2024.

Reviewer_6zSzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

In hierarchical image classification, not all images can be classified into the deepest leaf nodes in the taxonomy tree due to ambiguity in the image. This paper aims to predict the most specific label justified by the image.

The key insight is that ambiguity may prevent an image from supporting a fine-grained label but still support a coarser label. The author proposes TreeSchism, which is a new parametrization that predicts local evidence for each node and accumulates evidence from descendants upwards. The accumulated scores can be used for classification at the level, and a refinement gap can also be defined to measure the fraction of a node’s support not inherited from its descendants, which is used to infer the stopping point.

The training objective combines per-level classification with refinement-gap supervision, encouraging the model to stop at the annotated node. At inference, the model follows the most likely child and stops when the refinement gap exceeds a fixed threshold of 0.5.

The authors evaluate TreeSchism on three two-level taxonomies with synthetically blurred images and four deeper taxonomies derived from BIOSCAN-5M. The authors design experiments to assess set-prediction quality, exact-node accuracy, prediction depth, and off-lineage errors. Analyses of stopping signals and inference-threshold sensitivity are also presented.

Critical strengths
  1. This paper tackles a clear and practical problem. Classifying an image into the right class at a higher taxonomy level is better than classifying it into the wrong class at a deeper taxonomy level. This paper generalizes this problem from only a 2-level taxonomy to a deeper taxonomy tree.

  2. The proposed parametrization of recursively accumulated evidence is neat and intuitive. This formulation decouples evidence at different levels, but still considers evidence from parent classes while providing a simple, interpretable signal for deciding where to stop in the hierarchy.

Critical weaknesses
  1. Marginal performance gain over baselines in set-prediction quality. TreeSchism explicitly models the stopping condition on arbitrarily deep taxonomy trees, but the benefits are not supported by results. In Table 1, TreeSchism is essentially on par with HENN, which only works on 2-level taxonomy trees, and SoftCE, which trains an ordinary classifier.

  2. Inconsistent gains in localization quality. In Table 2, TreeSchism achieves an impressive gain on BIOSCAN-5M in ExactNode%, but at the cost of major drop in OffLineage%. This suggests that TreeSchism can stop at the exact node more often than baselines, but when it is wrong, TreeSchism often misses the entire branch, which is a more serious miss than not getting the exact node right.

  3. The ambiguity-level annotations need stronger validation. It is hard to apply the right Gaussian blur so that an image becomes unrecognizable at the leaf level while remaining identifiable at the assigned coarse level. TreeSchism achieves larger gains on BIOSCAN-5M, but its missing fine-grained labels may reflect incomplete annotation rather than inherent visual ambiguity. The model may therefore learn patterns of label availability rather than when ambiguity genuinely warrants stopping at a coarser level.

Questions for authors
  1. How important is the Dirichlet distribution formulation in the method? The loss and stopping rule appear to use only normalized evidence scores and their ratios. Why is a Dirichlet distribution needed? Could the same method be implemented using positive node scores without a distributional interpretation, and if so, what does the Dirichlet formulation add?

  2. How robust is the stopping rule under distribution shift? Does the fixed threshold of 0.5 remain effective for unseen corruption types or changes in the proportion of ambiguous images?

  3. Can TreeSchism benefit standard hierarchical classification when all training images have leaf-level labels? The proposed evidence parameterization seems potentially useful beyond ambiguity localization. Have the authors explored its benefits for standard hierarchical classification, where all training images have leaf-level labels?

I am open to raise my final rating if the authors effectively address my main concerns.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 6zSz

Thank you for your comments. We will respond to each below.

Marginal performance gain over baselines in set prediction quality:

TreeSchism is intended to predict a taxonomic node, and thus it is on the hierarchy-aware metrics (Tables 2, 3) that it distinguishes itself. First, we note that the competitive baseline (SoftCE+HSC) was tuned on OverJS, the set-prediction metric on which it has parity with TreeSchism; conversely, TreeSchism wasn’t tuned at all. Applying HSC inference or similar on top of TreeSchism could improve it further. Second, the nearly-equal mean scores on set prediction metrics hide meaningful differences in how either method predicts, which are clear from Table 2 (i.e., HSC inference is more risk-averse, lowering its OffLineage%, while TreeSchism identifies the true node and stopping depth more reliably). TreeSchism predicts the right node more often, while SoftCE+HSC gets partial credit for its near-misses on set prediction metrics.

Inconsistent gains in localization quality.

As mentioned in the overall response, we found that our method’s worse OffLineage% was due to the fact that it had no mechanism to move to coarser nodes when it was unconfident, unlike the baselines using HSC inference. We added a paragraph to Section 4.3 and a new appendix (Appendix E) to establish that TreeSchism can typically achieve better ExactNode% and Level-\Delta than SoftCE+HSC at the same OffLineage%. The exception is when we allow very few errors (OffLineage% <= 1.5%), in which case the methods become competitive.

The ambiguity-level annotations need stronger validation.

As mentioned in our overall response, we agree with your criticism and have removed the claim that our datasets’ non-leaf labels reflect ambiguity inherent to the data. The use case of our method is to learn to predict what depth a trusted annotator pool would have predicted at. As Table 3 shows, our method does so more effectively than the baselines. This does not mean that annotator stop depths are never arbitrary, but given how strong our method performs across datasets and taxonomic levels (AUROCs of 0.861-0.973 in Table 3, usually 3-9 points ahead of the strongest baseline on BIOSCAN benchmarks), it is fair to say that they are rarely arbitrary. Since the annotators had access to information that our models do not, such as DNA barcodes in the case of BIOSCAN-5M, the ability to predict the label depths they would have chosen is both non-trivial and useful.

How important is the Dirichlet?

Our method admits a natural reinterpretation through the lens of Dempster-Shafer theory, a correspondence we discuss in a new appendix (Appendix F), so our use of the Dirichlet was not, strictly speaking, necessary. We opted to use the Dirichlet in our main text as it is simple and has nice semantics (e.g., its parameters are pseudo-counts, and the data-fit term of our loss runs through Dirichlet semantics), and we expect that it will be more familiar to most of our paper’s audience than Dempster-Shafer theory. We now explicitly state the study of other parameterizations and losses as avenues for future work.

How robust is our stopping rule to distribution shift?

A complete response to this question would require more experiments including new datasets, perhaps swapping BIOSCAN-5M images with iNaturalist images of the same species. In BIOSCAN-5M, the majority of specimens lack leaf labels, while in the two-level taxonomies the vast majority have leaf labels. Thus, our method works with very different distributions in the distributions of ambiguous labels, although whether it retains its performance under shifts in those distributions we cannot currently say.

Can TreeSchism benefit ordinary hierarchical classification?

Perhaps. Many existing hierarchy-aware training objectives have been proposed, and some of these factor probability distributions along a branch (i.e., a path from root to leaf) [1]. To the best of our knowledge, our construction differs from existing methods in that it shares evidence additively down a taxonomy, rather than multiplicatively. Thus, miscalibration errors accrue additively rather than multiplicatively, which could improve performance in deeper taxonomies. However, the refinement gap loses its significance in settings where all samples are understood to have leaf labels. It is unclear how we would supervise local evidence in interior nodes; perhaps a self-supervised loss could train a model to automatically route hard samples to coarser nodes where they can be classified with confidence.

Please let us know if you have any further concerns, we are happy to discuss them with you.

[1] Jack Valmadre. 2022. Hierarchical Classification at multiple operating points. In Proceedings of the 36th International Conference on Neural Information Processing Systems (NIPS ‘22). Curran Associates Inc., Red Hook, NY, USA, Article 1311, 18034-18045.

Thank you for your detailed explanation, which has partly addressed my concerns. My remaining concerns are:

  1. My concern about the evaluation benchmarks remains. I’m still unconvinced that either benchmark reliably identifies the most specific label the image itself supports, which matters given the paper’s motivation of localizing visual ambiguity. Could the authors evaluate TreeSchism on a dataset curated using a visual recognition model, such as ImageNet-F [1]?

  2. I think it’s important to distinguish learning where to stop at inference from learning from incomplete hierarchical labels during training. TreeSchism treats the deepest annotated node as the correct place to stop. A genus-level label therefore tells the model both “this is the genus” and “stop at genus.” If the annotation reflects what an expert can identify from the image, that supervision makes sense. But if some taxa are simply annotated less thoroughly than others, the model may learn those annotation patterns. With incomplete-label training, a genus-level annotation can supervise the genus prediction while leaving open whether the species is visually identifiable.

  3. Do you think learning when to stop requires an explicit stopping-depth annotation for each image? For example, on a dataset where every image has a leaf label, could a model learn which visual features distinguish the children of each node, perhaps by optimizing a decision tree? Could it then use the strength of that evidence to decide whether to go one level deeper on a new image?

  4. The results need clearer presentation. I understand TreeSchism might underperform due to unmatching settings. I would try to align the settings whenever possible, or explain why it is not feasible and why misaligned comparison is still meaningful.

With that said, I do see the value and contribution of this paper, and I will raise my rating to weak accept. I encourage the authors to consider these comments in future revision.

[1] Park et. al., Free-grained hierarchical visual recognition, 2026

Reviewer_RKTHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates whether a model can determine the finest level of classification justified by an image and stop there when the image does not support finer distinctions. It introduces TreeSchism, which assigns nonnegative local evidence to each non-root node and accumulates evidence upward through the taxonomy. A refinement gap measures the fraction of a node’s evidence-plus-prior contributed by its local evidence and serves as a stopping signal. Training supervises both classification and the stopping boundary, assuming annotation depth reflects inherent ambiguity. TreeSchism outperforms or performs comparably across seven taxonomies against competitors.

Critical strengths
  1. Meaningful problem: The paper addresses the practical need to predict at an appropriate granularity when an input does not support fine-grained classification.

  2. Simple and intuitive method: Evidence accumulation and the refinement gap provide a clear, trainable mechanism for selecting prediction depth in multi-level taxonomies.

  3. Strong empirical evaluation: The method improves exact-node accuracy by 6–21 percentage points on four-level taxonomies, supported by analyses of classification accuracy, stopping signals, and threshold robustness.

Critical weaknesses
  1. The assumption that coarse labels reflect inherent ambiguity is insufficiently validated.
    The method treats each annotated node as the finest label justified by the image. However, coarse annotations may also reflect incomplete labeling, limited expertise, or resource constraints. The paper does not independently establish that annotation depth reliably represents visual identifiability.
    Suggestion: Have multiple experts independently assess the finest identifiable level for a representative subset, report inter-annotator agreement, and compare their judgments with the original labels.

  2. The experiments establish improved granularity prediction, but not the interpretation of the stopping signal as inherent ambiguity.
    Table 2 convincingly demonstrates improved prediction of annotated nodes and depths, with comparable classification accuracy at the ground-truth level. Table 3 further supports the refinement gap as an effective stopping signal. However, both evaluations use annotation boundaries as ground truth and cannot distinguish learning visual ambiguity from reproducing annotation depth. The synthetic experiments also couple blurring with coarse labels.
    Suggestion: Include controls that coarsen labels while keeping images unchanged, alongside progressively degraded images with independently assessed identifiable depths. These would test whether the stopping signal follows visual information or labeling conventions.

  3. Missing direct node-prediction and stopping baselines limit attribution of the gains.
    The training labels already specify the target node, making a standard softmax classifier over all non-root nodes a natural baseline. However, the SoftCE baseline distributes internal-node targets over descendant leaves and relies on a separate inference procedure to select depth. The current comparisons therefore do not isolate the benefit of the proposed evidential formulation from that of directly supervising the prediction boundary.
    Suggestion: Compare against an all-node cross-entropy classifier and a hierarchical classifier with an explicitly supervised stop/continue head, using the same backbone and comparable tuning budgets. These comparisons would clarify what evidence accumulation and the refinement gap contribute beyond direct boundary supervision.

I am open to improve my rating given the concerns are resolved or clarified.

Questions for authors
  1. What evidence supports treating the internal-node labels in BIOSCAN-5M as boundaries of visual identifiability, rather than incomplete annotations? Were these labels independently assessed for image-based identifiability, or is this an assumption adopted from the dataset?

  2. How does TreeSchism compare with directly supervised node-prediction baselines? In particular, have the authors evaluated an all-node softmax classifier or a hierarchical classifier with a supervised stop/continue head? Such comparisons would help distinguish the benefit of the evidential formulation from that of explicit boundary supervision.

  3. Which validation objective was used to select the baselines’ operating points? Section 4.1 states that operating points maximize CompJS, whereas Appendix B states OverJS. Please clarify which was used and whether this choice affects the comparison with TreeSchism, particularly its exact-node accuracy advantage.

2 comments
AuthorsOpenReview ↗

Response to Reviewer RKTH

Thank you for your constructive comments. We will respond to each of your questions and concerns below.

“The assumption that coarse labels reflect inherent ambiguity is insufficiently validated.”

As mentioned in our overall response, we believe that this criticism is substantially true, and we have shifted our paper’s framing as a result. Settings like BIOSCAN-5M are where our method shines: a set of criteria exist that determine when each prediction depth can be justified, and we train a model to identify and apply those criteria by supplying it with labelled examples. It is also true that label noise may be present in BIOSCAN-5M. We remind the reviewer that our method is meaningfully better at selecting the annotators’ chosen prediction depth for test samples than baselines (Table 3), so even if label noise is present, TreeSchism still performs best at its revised goal of recovering the labels selected by the annotator pool.

“Experiments establish improved granularity prediction but not the interpretation of the stopping signal as inherent ambiguity.”

As mentioned above, we agree with this criticism, and have updated the paper’s definition of “ambiguity” to address it.

Missing direct node-prediction and stopping baselines:

We agree that attributing our gains to the refinement gap was not justified, and we revised the paper to retract that claim. Supervision of the prediction boundary is an integral part of our method. We see this problem setting as prediction along two axes: we must both select the right branch (i.e., the right path from root to leaf) and the right depth (i.e., the deepest justifiable node along that path). Our goal is to design a classifier for this task whose predictions along both axes must cohere with each other, and which meaningfully incorporates taxonomic relationships into decision-making.

An all-node softmax classifier treats genera and species as mutually-exclusive, which contradicts the “is-a” relationships implied by taxonomies. To restore the “is-a” semantics in this model, we must sum probabilities from descendants to ancestors, recovering our accumulated evidence construction. The results from Appendix G can be adapted to show that such a classifier is once again a Dempster-Shafer belief assignment. Since an argmax stopping rule positions genera against species as rivals, the natural principled stopping rule is our refinement gap, as explained in Appendix G.

The stop-head classifier decouples predictions along the axes, which can trigger incoherent outputs: we may descend to a child even when no children are considered likely, and we may decline to descend to a child that is considered almost certain. TreeSchism prevents such pathologies by construction. The refinement gap is computed from the same evidence scores that determine class assignments, so classifications along both axes must cohere with one another. Further, the reinterpretation of the refinement gap as uncertainty mass conditioned on membership in a subtree (Appendix G) justifies our decision to use it and a single shared threshold as the stopping signal across every taxonomic level. In a nutshell, our method offers coherence and interpretability that the proposed baseline cannot guarantee.

We acknowledge that neither proposed baseline is in the paper. We now note this in the limitations, and since our revised claims no longer rest on separating the Dirichlet parameterization from boundary supervision, we leave an empirical study of alternate losses and parameterizations to future work. The loss and parameterization we do study already show large gains over existing methods in terms of ExactNode% and Level-\Delta.

What validation objective was used to select the baselines’ operating points?

Thank you for flagging this discrepancy. The selection metric was OverJS, and we have updated Section 4 to report this accurately.

Please let us know if you have any further concerns, we would be happy to discuss them with you.

Thank you for the candid response and the clarifications.

  • Q3 has been resolved: OverJS is now reported consistently.

  • W1 and W2 are only partially resolved. The abstract and Section 3 now define the target as the depth a trusted annotator would assign, but the title, the introduction (the Empidonax example, “inherent ambiguity”) and the conclusion still frame the work as localizing visual ambiguity. The new statement that the annotation assumptions are “unfalsifiable” (L148–149) is not accurate: independent expert re-annotation on a subset would test them directly. In addition, the rebuttal’s explanation of BIOSCAN-5M’s coarse labels (DNA barcodes, unnamed species) seemed missing.

  • W3 remains, and the revision contradicts the rebuttal. The rebuttal states that the attribution of the gains to the refinement gap has been retracted. Yet, the revised text still says that “TreeSchism’s lead can be attributed to its ability to select the right prediction depths” (L421–422) and that “TreeSchism’s advantage comes from what it measures” (L492). Table 3 points the other way: even standard vacuity from a TreeSchism-trained model outperforms SoftCE’s stopping signals in most columns, which suggests the gains come mainly from depth supervision on internal nodes rather than from the refinement gap. An all-node softmax baseline would settle this question, and it remains missing.

I therefore maintain my rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Overall Response to Reviewers

General response:

We thank the reviewers for their thoughtful comments on our paper. Here are some responses to common reviewer concerns.

On the Dirichlet parameterization. Multiple reviewers asked about our decision to use the Dirichlet parameterization. We originally developed our method in terms of Dempster-Shafer theory, but opted to present an analogous Dirichlet-based approach under the assumption that most readers would find it simpler and more accessible. Given reviewer comments, we restored our discussion of the Dempster-Shafer formulation of the method in a new appendix (Appendix G). The correspondence is straightforward and grounds the main text’s notions of local evidence, accumulated evidence, and the refinement gap given in existing Dempster-Shafer concepts.

On empirical results. Reviewers also had questions about our claimed empirical improvements over the baselines. We have discussed our findings in greater detail in individual responses. Table 2 could be interpreted as suggesting that TreeSchism offers better Level-Δ\Delta and ExactNode% scores than baselines by performing worse OffLineage%. However, HSC inference lets the other methods defer to coarser nodes when unconfident, while TreeSchism had no analogous mechanism. Please see Section 4’s new “On OffLineage%” paragraph as well as Appendix E, which demonstrates that TreeSchism generally offers a better trade-off between ExactNode%/Level-Δ\Delta versus OffLineage%, and can achieve higher ExactNode% and lower Level-Δ\Delta than any baseline.

On stopping depth. Our initial submission stated that “ambiguity is a property of the image and not the annotation process.” After considering reviewer comments, we realized that it is necessary to revise that claim. The target we learn is the deepest label a competent and trusted annotation process assigns to a label, rather than a loosely-defined notion of localized ambiguity. In addition to learning which visual features identify which classes (as all vision classifiers do), we also learn which features justify finer or coarser predictions. We have updated the paper to make this clear.

The question of which features identify a given species/genus/family is non-trivial, and there are textbooks and ongoing research devoted to that task. A standard classifier trained over species alone cannot effectively integrate these non-trivial criteria to make coarse taxonomic predictions when appropriate; to do so, it would need a large volume of labelled examples to learn from. It is this setting that TreeSchism targets: in addition to distinguishing classes “horizontally” (i.e. choosing the right species), we also learn to distinguish classes “vertically” (i.e. choosing the appropriate classification depths).

To address reviewer RKTH and 6zSz’s concerns about whether coarse labels in BIOSCAN-5M simply reflect incomplete supervision, we note that BIOSCAN-5M labels are grounded both in DNA barcodes and expert morphology. A specimen labelled only to its genus therefore has no named species in the taxonomy, so coarse labels reflect an incomplete taxonomy rather than visual ambiguity. Although BIOSCAN-5M annotators used DNA barcodes and other information in addition to images, we note that our image-only model can nevertheless effectively learn to predict the appropriate stopping depths, as our Table 3 AUROC measurements show (0.861-0.973). Reproducing the annotator pool’s labels using only one part of the information they had access to is a powerful and, we think, useful capability for a model to have.

We also included some more changes targeted to specific reviewer criticisms, which we discuss in our individual replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.