- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper purposes Tree-Adaptive Sets (TAPS), a method for conformal prediction in hierarchical multi-label classification. The purposed method constructs a hierarchy-aware nonconformity score that combines class probabilities with a penalty for selecting less specific ancestor nodes. A conformal calibration step adapted to tree structure is then applied to construct prediction set using the highest-level nodes satisfying the calibration threshold. The paper establishes finite-sample semantic coverage guarantees under exchangeability. Experiments on clinical coding and other hierarchical classification datasets demonstrate reductions in prediction-set size compared with the considered baselines.
Critical strengths
- The problem considered is practically relevant. Incorporating hierarchical structure into prediction sets provides a natural way to produce more compact and interpretable predictions.
- The paper presents relatively extensive experiments across multiple datasets, including comparisons with various baselines and ablation studies.
Critical weaknesses
- The purposed method mainly introduces a hierarchy-aware nonconformity score and a corresponding prediction-set construction, followed by standard split conformal calibration. The semantic coverage guarantee also largely follows from standard conformal arguments.
- A natural alternative is to first construct valid prediction sets using standard conformal methods such as LAC, APS, or RAPS, and then merge selected leaf labels into ancestor nodes. Such post-processing would preserve semantic coverage while potentially reducing prediction set size. Although the paper considers a related ablation, it does not directly compare against more carefully designed hierarchical compression procedures. This makes it difficult to assess the benefit of incorporating hierarchical penalties directly into the nonconformity score.
Questions for authors
Related to the second point in the weakness section, Table 3 in the ablation section reports that flat score + hierarchical calibration always outputs the root note. Is this because, when using the flat score, the root node is guaranteed to have a lower nonconformity score than any of its descendants, and thus cannot be pruned? If so, the author should also consider better post-processing procedures applied directly to leaf-level prediction sets, to ensure a fair comparison.
In addition, the trade-off between prediction-set size and specificity should be discussed more thoroughly. In its current form, I find it difficult to interpret the effectiveness and gains of the proposed method, since it is unclear what constitutes the best balance between set size and specificity.
We thank the reviewer for a thoughtful review, in particular for proposing a direct comparison with post-hoc compression of conformal sets. A revised paper and supplement have been uploaded; related weaknesses (W) and questions (Q) are grouped below. Status tags: [Addressed] changed in the revision; [Partly] addressed with a stated limitation; [Agreed] the point is accepted and the framing changed.
Summary of changes relevant to this review: new post-hoc lifts of flat LAC with leaf-expanded size, LCAD and a matched-target table (Appendix R); λ sweep (Appendix S); coverage intervals (Appendix T); MeSH correction (Appendix O).
W1. Novelty beyond standard conformal prediction. [Agreed] The coverage guarantee is standard split conformal prediction and dominance is immediate; dominance is now a proposition. The contribution is a hierarchy-aware nonconformity score used directly in calibration and prediction-set construction, as opposed to a separate representation transformation applied to an already constructed flat conformal set, together with the measured trade-offs. The lifts also use calibration quantiles; λ is selected on a separate split. The selection rule chooses λ = 0 on arXiv and DBpedia14, so the height penalty is inactive there. This separates the path-minimum construction from the penalty: the path-based score and output rule remain active at λ = 0, while the penalty matters on MIMIC-IV (λ = 0.10), where its removal yields about 70% root-only sets (Appendix S).
W2. Post-hoc compression of standard conformal sets. [Addressed] Lifts of the flat LAC set were added: L1 (lossless), L2 (fractional), L3 (probability-thresholded) and a more aggressive L3x (Appendix R). For a fixed flat set, replacing leaves by ancestors cannot uncover a label that is already semantically covered; our experiments additionally tune the lift parameters and the LAC level, so this fixed-set property is distinct from a finite-sample guarantee for the full tuned procedure. Three-seed means at α = 0.10 (nodes / leaf-expanded size / LCAD), MIMIC-IV: TAPS 17.08 / 36.0 / 2.83; L3 tuned for fewest nodes 9.57 / 39.3 / 3.12; lossless L1 17.63 / 33.0 / 2.07; L2 tuned for leaf-expanded size 15.08 / 34.1 / 2.44, at coverage 0.897 against 0.891 for TAPS. L2 attains numerically higher observed coverage and lower three-seed means for nodes, leaf-expanded size and LCAD; these are observed operating points, and the tuning of the lift does not establish a finite-sample guarantee for the full procedure. L1 differs: it has no tuned parameter, is applied at the nominal level and inherits the guarantee of flat LAC; it returns slightly more nodes than TAPS, with a smaller leaf-expanded set and lower LCAD. arXiv: TAPS 1.66 / 43.1 / 1.05; L3 1.98 / 42.1 / 0.97, at coverage 0.919 against 0.900. A carefully designed lift is therefore competitive; the paper no longer claims that TAPS is more compact or more specific.
Q1. Why Table 3 returns the root. [Agreed] The reviewer's explanation is correct: with a flat score the nonconformity score of a node is 1 − p̂_v, and the root probability is approximately 1 (Appendix I), so the root has the smallest score of all nodes and passes whenever any node passes; the topmost-node rule therefore returns the root. This ablation is degenerate by construction. It shows that replacing the hierarchy-aware TAPS score with a flat score under the same construction causes root collapse; the designed lifts show that other output rules can avoid this pathology.
Q2. Size-specificity balance. [Partly] The paper now reports leaf-expanded size and LCAD alongside node count, and a λ frontier for MIMIC-IV. No method is uniformly preferable: flat sets retain fine resolution, whereas hierarchical sets yield a shorter node list. No clinician study has been conducted, and no claim is made that either approach reduces coding effort.
Corrections found during our audit [identified on our side while preparing this rebuttal; not raised in the reviews]. (i) The MeSH loader inserted a second dot into tree numbers that were already dotted, so that 99.14% of MeSH labels were dropped during evaluation. MIMIC-IV, arXiv and DBpedia14 are unaffected. After correcting the loader and retraining (one seed), the corrected MeSH result is unfavourable to TAPS (coverage 0.891 with 61.7 nodes covering all 282 leaves, against 0.907 with 10.7 nodes for LAC + L3x); the claim that TAPS generalises to MeSH is withdrawn (Appendix O). (ii) Across three seeds, observed MIMIC-IV joint coverage ranges from 0.8886 to 0.8927 for TAPS and from 0.8873 to 0.8924 for flat LAC. Separate document-level bootstrap 95% intervals exclude 0.90 but may underestimate uncertainty due to repeated admissions per patient. The cause remains unresolved (Appendix T).
We hope that the revision addresses the reviewer's concerns, and we would welcome any further comments.