- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies utility calibration for multiclass predictions with a fixed semantic label tree. It shows that the realized-minus-predicted utility residual decomposes exactly into internal-node contributions, whose expectations can cancel under ordinary Utility Calibration (UC). Hierarchical Utility Calibration (HUC) audits these contributions separately while retaining the same target utility, subgroup and predicted-utility interval. Experiments include an eight-leaf synthetic problem and five real-data benchmarks.
Critical strengths
- The paper provides a clear motivation for multiclass probabilistic prediction and Utility Calibration.
- The proposed method appears to be technically sound.
- HUC has been evaluated through extensive experiments.
Critical weaknesses
- The core idea of HUC needs to be clearly stated in the introduction.
- The findings in the paper need more solid support. For example, hierarchical auditing is illustrated through two examples, but its generality needs to be further supported by some essential explanations.
- The experimental analysis needs more convincing explanations.
Questions for authors
- The paper shows that the difference between realized utility and predicted mean utility can be exactly decomposed into a sum of contributions from the internal nodes of the label tree. Hierarchical auditing serves as the foundation of the paper. However, is hierarchical auditing generally applicable in this field?
- HUC-BOOST is similar to UC-Boost. What are the specific advantages of HUC-BOOST over UC-Boost?
10 comments
2. Weakness 2 and Question 1: Generality, guarantees, and applications
Summary: Why does hierarchical auditing extend beyond the two examples, and how does it apply to hierarchical decisions?
In response, we added explanations of node contributions and calibration guarantees for aggregated utilities to Section 3.1, marked in red.
A general decomposition. Under the manuscript's assumptions, Theorem 3.1 applies to any finite rooted label tree and bounded utility, including unbalanced and multiway trees. The predictor need not be trained hierarchically: subtree and conditional branch probabilities follow from positive multiclass probabilities and the tree. Appendix K presents results on real-world datasets covering various tree structures and utilities.
Let be the predicted mean utility within subtree . For in child subtree ,
=\mu_{u,v_j}(p)-\mu_{u,v}(p),$$ and $d_v=0$ outside the parent subtree. These changes telescope from root to leaf to give $u(p,y)-\mu_u(p)$ (Eq. (9)): each contribution measures how refining one branch changes the evaluation of the same utility. For $P=f(X)$, Eq. (8) gives $$\mathbb{E}[d_v(p,Y)\mid P=p] =\pi_v^\star(p)\sum_j(q_{v,j}^\star(p)-q_{v,j}(p))\mu_{u,v_j}(p).$$ Here, $\pi_v^\star$ is the actual parent-subtree reach probability. When it is positive, $q_{v,j}^\star,q_{v,j}$ are actual and predicted conditional branch probabilities: their discrepancy is weighted by child-subtree mean utilities and reach probability. At zero reach, the conditional mean is zero. Equal child means also make the contribution zero. For a fixed tree and predictive distribution, Appendix C.3 establishes uniqueness and orthogonality under locality, child-subtree constancy, and zero predictive mean. **What HUC adds to UC.** The audit moment is the weighted average of the per-observation contribution $d_v$, with the same utility, subgroup $c$, and predicted-utility interval $I$ across nodes (Eq. (10)): $$\Gamma_{c,u,I,v}(f):=\mathbb{E}[c(X)\mathbf{1}\\{\mu_u(P)\in I\\}d_v(P,Y)].$$ Thus, $$\begin{aligned} \mathrm{UC}(f;\mathcal{C},\\{u\\})&=\sup_{c,I}\left|\sum_v\Gamma_{c,u,I,v}(f)\right|,\\\\ \mathrm{HUC}(f;\mathcal{C},\\{u\\})&=\sup_{c,I,v}|\Gamma_{c,u,I,v}(f)|. \end{aligned}$$ UC audits after summation; HUC audits before cancellation (Eqs. (11), (13)). To see the additional guarantee, expand an ancestor-closed set $S$ of internal nodes and stop at its boundary. Let $b_S(y)$ be the first node outside $S$ on the root-to-$y$ path. Averaging unresolved leaf utilities within that subtree gives (Lemma G.1) $$\begin{aligned} u_S(p,y)&:=\mu_{u,b_S(y)}(p),\qquad \mu_{u_S}(p)=\mu_u(p),\\\\ u_S(p,y)-\mu_u(p)&=\sum_{v\in S}d_v(p,y). \end{aligned}$$ These residuals are partial sums, and the unchanged predicted mean allows comparison on **the same subgroup and predicted-utility interval**. In the Weakness 1 example, the root contributes $-0.1$ on average; the node within infection adds $+0.1$, making the final residual zero. Expanding one boundary node adds exactly its contribution, measuring consistency between successive utility evaluations on the same audit population. A small final sum need not imply small partial sums. For every fixed tree boundary, Proposition G.3 gives $$\mathrm{UC}(f;\mathcal{C},\\{u_S\\}) \le |S\cap\mathcal{V}_{\mathcal{T}}(u)|\\,\mathrm{HUC}(f;\mathcal{C},\\{u\\}),$$ where $\mathcal{V}_{\mathcal{T}}(u)$ contains utility-relevant internal nodes. HUC therefore controls calibration of the original utility aggregated at different hierarchical granularities. This concerns the specified aggregated utilities; other decisions require their own utilities. **Concrete applicability.** Utilities can evaluate correctness of final-class, internal-node, or node-set outputs, tree-distance losses, and prediction specificity. For an output rule $a(p)$ and loss $0\le L(a,y)\le L_{\max}$, $L_{\max} > 0$, $$u_a(p,y)=1-\frac{2L(a(p),y)}{L_{\max}}$$ allows auditing the model's expected utility for that decision. Appendix A.1 covers linear, rank-based, and decision-derived utilities; Appendices J and K include correctness, threshold decisions, abstention, and specificity across different trees. Appendix I, added in red, connects these choices to the hierarchical losses of Ramaswamy et al. (2015) and Cao et al. (2024), the metric-dependent output rules of Plaud et al. (2025), and the hierarchical correctness utility and confidence calibration conditions of Goren et al. (2024). Our response to Question 2 explains how the same audit moment also gives a direction for local correction.3. Question 2: Specific advantages of HUC-Boost over UC-Boost
Summary of the question: HUC-Boost resembles UC-Boost, but what specific advantages does it offer?
Connecting the same audit quantity to correction at the corresponding branch
HUC-Boost can individually detect contributions that cancel in the sum and correct their corresponding branches. Both methods use the same utility and subgroup classes, but UC-Boost audits only the summed residual; canceled contributions do not trigger an update under those conditions.
The audit moment in our response to Weakness 2 and Question 1 provides a correction direction. Fix the direction defined by the current prediction,
and let be the predictor obtained by updating only the logits of the corresponding node as . Then
=-\Gamma_{c,u,I,v}(f)$$ holds (Appendix E.2). Adjusting branch probabilities according to this moment's sign gives a descent direction for log loss. HUC-Boost updates when the absolute value of the audit moment $\Gamma_{c,u,I,v}(f)$ exceeds the prescribed threshold. This update preserves the parent subtree's total probability, each child subtree's conditional distribution, and leaf probabilities outside the subtree. With the prescribed step size and stopping threshold and finite initial population log loss, Theorem 3.5 and Appendix E.2 guarantee loss decrease and finite termination. Thus, **a single node contribution connects discrepancy localization, calibration guarantees at each granularity, and corresponding local correction**. These are HUC-Boost's additional guarantees and mechanism; empirical gains depend on the base predictor and initialization. Our response to Weakness 3 presents real-data cancellation and observed improvements. **Finite-sample guarantees with adaptive step sizes** We derive the adaptive step size $\alpha_t=\Gamma_t/\Lambda_t$ from the audit moment $\Gamma_t$ and a curvature upper bound $\Lambda_t$ along the update direction (Appendix E.2). The empirical step also uses the calibration sample, so repeated auditing and updating require control of sample dependence. Appendix H.7, added in red, analyzes independent multiplicative perturbations of the empirical step: $$\alpha_t=\Xi_t\\,\frac{\widehat\Gamma_t}{\widehat\Lambda_t}, \qquad \Xi_t\sim\operatorname{Unif}[1-\rho,1+\rho],\quad 0 < \rho < 1.$$ For an initial predictor independent of the calibration sample, under the manuscript's assumptions, we provide **a finite-sample population HUC guarantee at termination of adaptive HUC-Boost despite repeated sample reuse** (Theorem H.17), beyond empirical loss decrease and termination. **Guarantees for two-stage correction using the same sample** The first-stage parametric recalibrator and second-stage HUC-Boost can use the same calibration sample. By adding a small independent perturbation to the learned initial parameters to control sample dependence, we guarantee population HUC at termination of the two-stage method with a fixed step size, under the prescribed assumptions and sample-size conditions (Appendix H.2, Theorem H.5). We also provide a guarantee at termination for a two-stage method that additionally uses multiplicative perturbations of the adaptive step size (Appendix H.7, Corollary H.19). This two-stage analysis also applies to the corresponding UC-Boost via a star tree with all classes directly below the root. Thus, the analysis of sample reuse is a theoretical contribution applicable to both methods, whereas **the distinction specific to HUC-Boost is that it audits and corrects contributions within the hierarchy and obtains guarantees for those contributions**. For validation-based selection in the experiments, we added in red a population HUC guarantee relative to the best candidate within the selected path, under the prescribed candidate family, when selection uses HUC on an independent validation sample (Appendix H.8, Theorem H.23). For the perturbed method reaching natural termination under the prescribed sample-size conditions, this connects to an absolute HUC guarantee (Corollary H.24). With computational-budget truncation, the guarantee is the within-path comparison. **Fewer updates with curvature-based adaptive steps.** In response to Reviewer 2's request, we compared adaptive HUC-Boost, randomized adaptive HUC-Boost (RHUC), and fixed-step HUC-Boost (FHUC). In two-stage comparisons across many datasets, these methods achieved similar mean test HUC, while adaptive HUC-Boost and RHUC required substantially fewer updates than FHUC (Section 5.2 and Appendix K.9). These results support the benefit of curvature-based steps in achieving comparable performance with fewer updates.
1. Weakness 1: The core idea of HUC
Thank you for reading our paper in detail and for your careful review and constructive comments. We also appreciate your recognition of the motivation, technical soundness, and breadth of the experiments. In response to your comments, we have strengthened the presentation of HUC's core idea, the generality of hierarchical auditing, the specific advantages of HUC-Boost, and the interpretation of the experimental results. The main additions and revisions are marked in red in the revised manuscript.
Summary of the concern: The core idea of HUC needs to be stated clearly in the Introduction.
In response to this comment, we have revised the Introduction (Section 1), with the changes marked in red, to first illustrate the problem through a concrete example and then explain what HUC audits, its applicability to general trees, and the idea connecting auditing to local correction.
Discrepancies that are invisible in the final mean. Consider a simplified medical example in which we first assess whether a patient has an infection requiring treatment and, if infection is confirmed, assess whether ICU care is needed. Suppose every patient receives the same prediction: the predicted probability of infection is 80%, and the predicted conditional probability of requiring ICU care given infection is 25%, whereas the corresponding true probabilities are 40% and 50%. The predicted and true probabilities of the final event, “having an infection and requiring ICU care,” agree because 0.8×0.25=0.4×0.5=0.2. Thus, UC for the indicator utility of this event is zero over the full population. Nevertheless, the overestimation of infection probability and the underestimation of the conditional probability of requiring ICU care given infection remain (the numerical example in Section 1 and the definition of UC in Section 2.1, Eq. (4)).
For example, suppose a hypothetical treatment threshold is set at an infection probability of 60%, and the threshold for ICU care after infection is confirmed is set at a conditional probability of 40%. At both stages, the decisions based on the true probabilities differ from those based on the predictions. The predicted infection probability of 80% exceeds the 60% threshold, whereas the true probability of 40% is below it, so the model chooses treatment that would not be chosen based on the true probability. After infection is confirmed, the predicted probability of requiring ICU care is 25%, below the 40% threshold, whereas the true probability of 50% exceeds it, so the model forgoes necessary ICU care. This cancellation structure is explained by the four-leaf tree decomposition in Section 2.2 (Eq. (5)).
Such cancellation of node contributions is also observed in real data (Section 5.2); we explain the details in our response to Weakness 3 below.
Auditing with HUC and extension to general hierarchies. We propose Hierarchical Utility Calibration (HUC), which decomposes the error in a specified utility into contributions associated with the branches of the label hierarchy and audits them before summation (Section 3.1, Eqs. (9)–(13)). In the example above, for the same target utility corresponding to “having an infection and requiring ICU care,” we separately evaluate the contributions from the branch distinguishing infection from no infection and the branch within infection distinguishing whether ICU care is needed. By keeping the target utility, subgroup, and interval of predicted mean utility the same, we can identify the branches where discrepancies remain even when they cancel in the final mean. Contributions from branches that do not affect the target utility are zero, allowing the audit to focus on the parts relevant to that utility (Eqs. (8), (12)).
This decomposition and auditing apply to general finite rooted trees. In trees with three or more children per node or with different depths across branches, positive and negative contributions associated with the same utility can likewise cancel when summed (Theorem 3.1, Appendix C.2). We further establish the connection to calibration when the original utility is aggregated at different levels of granularity along the hierarchy (Appendix G, Proposition G.3), and introduce HUC-Boost to connect detected violations to correction at the corresponding branches (Section 3.2, Algorithm 1). This provides a framework for detecting utility discrepancies while retaining their locations in the hierarchy and correcting them locally.
The differences from calibrating leaves or individual branches separately are explained in Section 2.2 and Appendices D.2–D.4. We explain the mathematical substance of the decomposition and the additional guarantees in our response to Weakness 2 and Question 1 next, and the mechanism and advantages of local correction in our response to Question 2.