- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a hierarchical classification method called TREESCHISM, which addresses the issue that ordinary classifiers can only perform precise classification but cannot make hierarchical judgments. TREESCHISM enables the model to output the intermediate nodes in the classification tree, that is, to select "the deepest level that the current evidence can support". The method is based on evidence deep learning. The model generates local evidence for each node, accumulates the descendant evidence towards the ancestors and calculates the support degree, and makes the judgment through the threshold.
Critical strengths
The method can output intermediate levels instead of only leaf nodes; In terms of the metric "determining at which layer to stop", some experiments outperformed the baseline; The refinement gap is more suitable than ordinary classification probabilities as a stopping signal.
Critical weaknesses
I found the paper did not do related work research throughly, I know at least one paper that is highly related to this method or claim. But I did not find the citation.
The evidence theory and EDL are methods with strong theoretical basis. The article makes several arbitrary changes to them in order to achieve better empirical results, but it is difficult to maintain their theoretical completeness and lacks theoretical interpretability.
The advantage of EDL lies in being able to obtain the prediction results as well as the uncertainty of the model's prediction, aka, evidence vacuity. However, the article does not utilize evidence vacuity. Instead, it designs its own indicator, which makes me question whether the author has a sufficient understanding of EDL.
There are also unfair aspects in the utilization of data. Randomly/uniformly subdividing high-level label data to low-level label data to train models will lead to semantic deviations, thereby affecting the effectiveness of the comparison model.
Questions for authors
Are there any essential differences between this method and method H-MEAN[1]? Apart from adding an additional threshold for judgment. By the way, the H-MEAN paper is not mentioned in the article either.
What is the theoretical basis for using a Dirichlet distribution at each level? If one wants to use the evidence of the same level and the evidence of its child nodes, then the idea of hyper opinions should be used, and it should support both composite sets and singletons.
Why don't use evidence vacuity as the key to decide hierarchical level?
Is it meaningful to gain better results solely at the prediction level? Each case definitely has the label to the leaf node, it's just that the accuracy of judgment is different. Precise prediction, this is exactly what the model hopes to achieve. We can represent the confidence of the decision through uncertainty and accumulate the confidence of each level instead of directly judging that the decision cannot continue at this level.
3 comments
After carefully reading the authors’ response, I remain unconvinced, and my original concerns have, if anything, become more pronounced. The response does not directly address the central issues I raised, while several key points remain insufficiently discussed. My concerns are as follows:
-
Identification at arbitrary hierarchical levels. Any EDL-based method can trace evidence upward through the hierarchy, accumulate evidence, and thereby obtain uncertainty estimates at different hierarchical levels.
-
Motivation for modifying EDL and excluding vacuity. The authors’ explanation of these choices further reinforces my concern that the proposed modifications are primarily empirically motivated rather than derived from a clear theoretical foundation.
-
Relationship between Dirichlet priors and evidence theory. The authors explain their motivation from the perspectives of both the Dirichlet prior distribution and evidence theory. Although this may appear reasonable at first glance, these two frameworks have fundamentally different starting points and interpretations; the fact that they lead to similar mathematical forms does not establish a substantive theoretical connection. Therefore, using this relationship to motivate the proposed method does not appear sufficiently justified.
I would appreciate it if the authors could address these concerns more directly and provide a more rigorous theoretical justification for the proposed design.
On “Any EDL method can accumulate evidence upward”:
This is true, but does not suffice for our problem setting. In leaf-only EDL with upward accumulation, internal nodes have no local evidence and the refinement gap is identically zero. It would be impossible for a sample to carry evidence at a genus that is not also evidence for some species.
Accumulating scores upwards would produce an EDL analogue to our SoftCE+HSC baseline. Our Table 3 shows that SoftCE+HSC’s stopping signal is weaker than the refinement gap for granularity prediction. In this setting, we need to place un-refinable evidence on interior nodes; it is both the right modelling choice and more effective empirically. We note that Li et al. [2] (the HENN authors, not the H-MEAN authors [1]) evaluated the EDL/upwards-accumulation approach in this setting and found that it underperformed other methods.
On “Excluding vacuity is empirically motivated”:
Vacuity and hierarchical ambiguity are different types of uncertainty. Vacuity is high when a model has little overall evidence about an input. Conversely, hierarchical ambiguity is a product of a sample rather than of a model’s knowledge. A well-trained model may hold abundant evidence that a sample belongs to a genus and be equally confident that no species label is warranted. Vacuity should be high when a model is unconfident, but a model can be confident that no species label is warranted.
Evidence accumulation does not change this. In ordinary EDL, if we accumulate evidence upward, we preserve the total evidence, so vacuity at each level is the same per-sample scalar, rescaled by the class count at that level. It communicates a model’s overall level of confidence, but cannot communicate where in the taxonomy evidence stops being refined.
The refinement gap solves this problem. Vacuity is the mass on an entire frame that cannot be refined to any proper subset. Lemma G.9 shows that the refinement gap has the same interpretation after conditioning on membership in a node’s subtree - it is exactly the mass on the entire sub-tree that cannot be refined to any proper sub-tree. Our paper speaks of localizing ambiguity to different nodes, but we could instead have said "localizing vacuity". It is precisely our formulation’s ability to quantify vacuity differently at different subtrees that allows it to determine the appropriate prediction depths. Li et al. [2] separate vagueness from vacuity for the same reason.
On “Dirichlet and evidence theory merely share a form”:
The correspondence between Dirichlets and evidence theory does not originate with us. It is used in Subjective Logic and underpins EDL. We are not creating a new correspondence, but rather specializing an existing one to a new problem setting. We are happy to discuss any questions or concerns you have about the re-interpretation of our method in evidential terms.
Other notes:
We realized that our original reply to you mis-identified some appendices, and we apologize for any confusion. The comparison between our method and Dempster-Shafer theory is in Appendix G, not F; and the gradient analysis in Appendix F, not E.
[1] Y. Li, Y. Chen, W. Liu, J. Qu, C. Ma and X. Yue, "H-MEAN: Hierarchical Multi-view Evidence Aggregation Network for Trustworthy Medical Image Classification," 2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Wuhan, China, 2025, pp. 2423-2428, doi: 10.1109/BIBM66473.2025.11356753.
[2] C. Li et al., “Hyper evidential deep learning to quantify composite classification uncertainty,” arXiv:2404.10980, 2024.
Response to Reviewer 22CS
Thank you for your feedback on our paper. We will respond to each of your concerns below.
Prior work:
By H-MEAN, we think the reviewer was referring to Li et al. [1]; please let us know if we have mis-identified it. We believe this work is only tangentially related to ours. Like us, they take an approach to hierarchical classification grounded in subjective logic and similar notions. However, their contribution lies in aggregating evidence from multiple sources. Unlike our work, they have no notion of hierarchical ambiguity or anything analogous to the refinement gap, and no mechanism to predict non-leaf nodes.
Regarding the other related work you mentioned: to our knowledge, no existing knowledge models hierarchical ambiguity and localizes it to the appropriate node in taxonomies of arbitrary depth. That said, we would be grateful if you could share the specific reference you have in mind. We’d be glad to discuss it here and to cite and compare against it in a revision.
Theoretical basis, arbitrary changes:
Our method was originally developed through Dempster-Shafer theory, but we decided to present it through Dirichlet semantics to make our paper simpler and more accessible. Given reviewer interest in its theoretical grounding, however, we added a discussion of a correspondence between our method and Dempster-Shafer theory in Appendix F. Each of our primary notions (accumulated evidence, the refinement gap, etc.) has an analogue in Dempster-Shafer theory that admits the same interpretation.
Even without the Dempster-Shafer interpretation, we believe that our modifications to the standard EDL recipe are not totally arbitrary. The refinement gap does a better job of localizing ambiguity than vacuity (Table 3), our gradient analysis shows why our loss is effective in practice (Appendix E), and the construction is simple and interpretable.
Why didn’t we use vacuity?
Vacuity scales with the number of classes. As we walk up a taxonomy, the number of classes at each level decreases, so vacuity measurements from two different taxonomic levels are not directly comparable. Vacuity is therefore less well-suited to hierarchical inference than the refinement gap. We added vacuity to Table 3, confirming empirically that it cannot identify what depth to predict at as well as the refinement gap.
Unfair aspects in the utilization of data:
We believe this criticism is targeted towards our SoftCE baseline. We note that our other baselines (HENN, HRN, CHILLAX) all integrate coarse taxonomic labels during training. The fact that SoftCE outperforms these is not evidence that our evaluation was unfair, but rather an interesting discovery in its own right.
Why use separate Dirichlets per level rather than a hyper-opinion?
Simple proofs show that our method admits an equivalent formulation in terms of Dempster-Shafer theory, and the refinement gap at a node n can be interpreted as the mass on n after conditioning on membership in n’s subtree. Please see Appendix G in our revised draft for details.
Why predict interior nodes at all?
Other works have already considered the setting where each instance is understood to have a leaf label, and coarse labels in training data arise only from incomplete supervision. Two such methods (HRN, CHILLAX) served as baselines in our evaluation. We consider a different problem setting from them, in which non-leaf labels arise because any finer label is actively wrong. For example, in the BIOSCAN-5M dataset, some samples have non-leaf labels because their genera and/or species do not exist in the tree for the simple reason that no one has named their species yet.
[1] Y. Li, Y. Chen, W. Liu, J. Qu, C. Ma and X. Yue, "H-MEAN: Hierarchical Multi-view Evidence Aggregation Network for Trustworthy Medical Image Classification," 2025 IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Wuhan, China, 2025, pp. 2423-2428, doi: 10.1109/BIBM66473.2025.11356753.