- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces UtilityBench, a 15,000-instance benchmark for evaluating language models when errors have different consequences. It combines answerable and refusal-required queries, assigns sample-specific error costs through multi-model Best–Worst Scaling judgments and Bradley–Terry aggregation, and assesses the severity rankings using human annotations. The proposed UtilityAUC integrates utility across response coverage, accounting for correct answers, costly errors, appropriate refusals, and unnecessary refusals. Experiments across 20 models examine accuracy, reasoning strength, prompting, and response-scoring strategies. The results suggest that higher accuracy or stronger reasoning does not consistently translate into higher utility under the benchmark’s cost assumptions.
Critical strengths
- The paper addresses an important limitation of accuracy: errors with substantially different consequences should not necessarily receive equal penalties.
- The benchmark includes substantial annotation work and human validation, with additional analysis showing agreement beyond broad domain differences.
Critical weaknesses
- The validity of the cardinal costs remains insufficiently established. Human–LLM ranking agreement supports relative severity, but does not validate the cost ratios obtained through exponentiating Bradley–Terry scores and choosing a median anchor. These numerical assumptions directly influence model rankings. Sensitivity to the refusal penalty alone does not establish robustness to uncertainty in the error costs.
- The main experiments provide limited evidence of utility-ranking ability. The headline comparison assigns only binary answer/refusal scores, averaging over all within-group orderings. It therefore primarily measures cost-weighted response outcomes and answering rates, rather than the ability to prioritize beneficial answers over costly mistakes within attempted responses.
Questions for authors
What evidence justifies interpreting the exponentiated Bradley–Terry scores as cardinal, additive harm costs comparable to the fixed reward for correctness and refusal penalty? Without validating this interpretation, what supports treating UtilityAUC as a measure of utility rather than alignment with the judges’ perceived-severity preferences?
4 comments
: What supports interpreting UtilityAUC as a measure of utility rather than alignment with the judges' subjective preferences?
A: Estimating error costs from aggregated BWS judgments is a practical way to approximate real-world consequences, as directly estimating harm ratios across domains can be unreliable. For example:
How many one-hour train delays equal the harm of a medication error that causes permanent injury?
Reaching consensus on this ratio is nearly impossible, but people can generally agree that permanent injury is more severe than a one-hour delay. In BWS, we group samples into overlapping four-sample tuples and ask multiple judges to select the greatest- and least-consequence errors in each tuple (Appendix A.3). The resulting choices reflect numerical cost ratios, as more consistent selection of one error over others indicates a larger severity gap in real-world consequences. Using BWS to extract implicit ratio information from large-scale qualitative comparisons is therefore a reliable and feasible way to approximate real-world error consequences.
To further verify these numerical cost ratios against human judgments, we supplement the ranking analysis in RQ6 by measuring how much two judges disagree on the scale of relative costs across sample pairs using the root-mean-square error (RMSE):
=\sqrt{\frac{1}{|\mathcal P|}\sum_{(i,j)\in\mathcal P} \left(\log\frac{L_i^{(1)}}{L_j^{(1)}}-\log\frac{L_i^{(2)}}{L_j^{(2)}}\right)^2},$$ where $L^{(1)}$ and $L^{(2)}$ are the costs estimated from each judge, which can be an individual human, an individual LLM, or an aggregation of multiple judges: |Comparison|RMSE| |:---|:---:| |Human vs. Human|1.53| |LLM vs. LLM|1.67| |Human vs. LLM|1.76| |**Human vs. LLM (aggregated)**|**1.10**| **As the table shows, LLM and human cost ratios in our annotations are similar to the variation already present among humans.** Moreover, aggregating the choices on both the human and LLM sides brings the difference well below the disagreement between individual humans. Taking the subjective nature of risk judgments into account, these results further support our aggregated BWS pipeline as a practical estimate of real-world cost differences.Dear Reviewer eNAn,
Thank you again for the time and effort you have dedicated to reviewing our paper. As the discussion deadline on Nov. 18 (AoE) is approaching, we wanted to kindly follow up and see if you have had a chance to read our response, in which we:
- Clarified the theoretical basis for deriving numerical cost ratios from BWS comparisons;
- Verified human–LLM agreement on these numerical cost ratios;
- Confirmed that model rankings remain stable under cost-ratio compression;
- Added experiments on models with confidence estimation using token log probabilities.
We would love to hear your thoughts and are more than happy to provide any further clarifications if needed. We sincerely hope that our response fully addresses your remaining concerns and enables you to support the acceptance of our paper.
Best regards,
The Authors of Paper 18333
Thank you again for your time and thoughtful feedback on our paper. As the discussion deadline on Nov. 18 (AoE) is approaching, we wanted to follow up and let you know that we have uploaded the revised manuscript, which incorporates all the new experiments, analyses, and clarifications from our rebuttal:
- Section 2.2 & Appendix A.4: Expanded the discussion of eliciting implicit cost ratios via BWS comparisons and preserving pairwise ratios through Bradley–Terry maximum likelihood estimation ().
- Appendix A.5: Added the pairwise log-cost ratio RMSE analysis comparing human and LLM annotations to validate the numerical cost scale ().
- Section 4.4 & Appendix E.5: Added the sensitivity analysis compressing cost ratios via across all 29 evaluated models ().
- Section 4.3 & Appendix E.4: Added the full-benchmark evaluation of five models using mean token log probability to estimate model confidence, along with the correlation analysis with correctness and error cost ().
- Section 4.1: Expanded the main benchmark evaluation from 20 to 29 models (Table 1).
We invite you to check the updated PDF, where all revised parts are highlighted in . Please let us know if any further clarifications or adjustments are needed. We sincerely hope that our response and these revisions resolve your concerns, allowing you to support the acceptance of our paper.
Thank you for your thoughtful review. We address your concerns below.
A: Fitting a Bradley–Terry model to Best–Worst Scaling (BWS) annotations via maximum likelihood estimation (MLE) is an established approach (relevant literature listed below) that preserves the numerical ratio across samples.
Specifically, the cost ratio Li/Lj is given by eηi−ηj, with η=(η1,…,ηn) denoting the optimal MLE solution over all BWS comparisons. Formally, let C denote all annotated comparison choices across the benchmark:
\widehat{\boldsymbol\eta}=\arg\max_{\boldsymbol\eta}\sum_{c\in\mathcal{C}}\log P(c\mid\boldsymbol\eta) \quad\Longrightarrow\quad \widehat P(i\succ j)=\frac{e^{\widehat\eta_i}}{e^{\widehat\eta_i}+e^{\widehat\eta_j}} \quad\text{(Pairwise probability)},$$ $$\frac{\widehat P(i\succ j)}{\widehat P(j\succ i)} =e^{\widehat\eta_i-\widehat\eta_j} =\frac{2e^{\widehat\eta_i-\operatorname{median}_k\widehat\eta_k}} {2e^{\widehat\eta_j-\operatorname{median}_k\widehat\eta_k}} =\frac{L_i}{L_j}\quad\text{(Cost ratio)}.$$ This mapping preserves the pairwise ratios inferred from BWS. We express these costs relative to the $+1$ correctness reward and set the median cost to $2$ based on prospect theory's finding that losses weigh roughly twice as much as comparable gains (Kahneman et al., *Prospect Theory: An Analysis of Decision under Risk*, 1979). We also evaluate sensitivity to such transformations across all 29 models while keeping model outputs and $C_R$ fixed: $$\widetilde L_i(\gamma) =\operatorname{median}_j(L_j)\, \frac{L_i^\gamma}{\operatorname{median}_j(L_j^\gamma)}.$$ For $0<\gamma<1$, the power function $L_i^\gamma$ is monotonically increasing, thereby preserving the ordering of samples. Meanwhile, the cost ratio is compressed from $L_i/L_j$ to $(L_i/L_j)^\gamma$. Under different compression factors, the Spearman correlations of model rankings and the numbers of model pairs that reverse order are shown below. ||Ranking Spearman $\rho$|Reversed model pairs| |---|---:|---:| |$\gamma=0.75$|0.9906|11/406 (2.7%)| |$\gamma=0.5$|0.9823|18/406 (4.4%)| |$\gamma=0.25$|0.9591|31/406 (7.6%)| **The results show that model rankings remain largely robust to the transformation of cost ratios.** However, we preserve the ratio inferred from BWS, as further compression or expansion introduces an additional preference, such as limiting the influence of high-cost samples in the tail. > $\color{blue}{\textbf{W2}}$: UtilityAUC with binary answer/refusal scores mainly measure response utility and answering rates. They provide limited evidence that models can rank beneficial answers ahead of costly mistakes within attempted responses. **A:** **We additionally evaluate five models that provide token log probabilities and find consistent UtilityAUC improvements over binary scoring, driven primarily by ranking correct answers ahead of incorrect ones.** The table below reports UtilityAUC (with changes relative to binary scoring) and the Spearman correlations $\rho(s_i, c_i)$ and $\rho(s_i, -L_i)$, where $s_i$ is the mean token log-probability, $c_i$ is correctness, and $-L_i$ is the negative error cost: |Model|UtilityAUC|Spearman $\rho(s_i, c_i)$|Spearman $\rho(s_i, -L_i)$| |:---|:---:|:---:|:---:| |GPT-5.4 Mini|0.7287 ($\color{blue}{+0.0691}$)|0.467|-0.099| |GPT-5.6 Luna|0.7746 ($\color{blue}{+0.0604}$)|0.391|-0.055| |DeepSeek V4.1 Flash|0.7917 ($\color{blue}{+0.0497}$)|0.432|0.023| |GPT-6.1 Terra|0.8148 ($\color{blue}{+0.0301}$)|0.354|-0.085| |Qwen Plus|0.7814 ($\color{blue}{+0.0586}$)|0.143|-0.064| **As the table shows, using mean token log probability as the response score improves UtilityAUC by better distinguishing correct answers from incorrect ones, yet it does not rank lower-risk errors ahead of higher-risk ones.** How to elicit such risk-aware ordering is an important direction for future work. UtilityAUC is already designed to evaluate this preference, though it remains difficult to obtain ranking signals that align with this design.