- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
AcuityBench harmonizes five public health datasets into a four-level acuity benchmark containing 914 cases. Of these, 667 receive five physician ratings each and 247 inherit source labels. The resulting evaluation separates consensus cases, including boundary labels, from 217 cases with substantial physician disagreement. Twelve language models are compared in explicit classification and conversational formats; the latter is mapped to acuity labels by a rubric-based model judge. The paper also studies alignment with physician endorsement distributions and model-assisted adjudication of disagreement.
Critical strengths
-
The annotation protocol is a useful contribution. Five ratings per newly annotated case, explicit boundary labels, and retention of disagreement provide more information than a single hard label. The appendices document the annotation and aggregation procedures in considerable detail.
-
The paired comparison of classification and conversational responses addresses a practical evaluation question. The same cases are evaluated in both formats, with confusion matrices, directional errors, bootstrap intervals, and paired tests. Under the stated rubric, over-triage decreases and under-triage increases across all twelve models; reporting these directions is more informative than reporting accuracy alone.
Critical weaknesses
-
The uncertainty-alignment results need reference baselines before their magnitude can support the headline interpretation. Section 5 and Appendix G compare five stochastic samples from one model with the retained votes of up to five physicians, on cases selected specifically for physician disagreement. These are different sources of variation, and the selection makes low panel agreement expected. This does not make the comparison invalid or guarantee its outcome, but it leaves JSD/Wasserstein distances and changes in panel agreement difficult to interpret. A held-out-physician comparison, a sampling-noise reference for the empirical panel distribution, and simple central/uniform predictors would establish a useful scale. Confidence intervals are also needed before interpreting small differences between models. In particular, increasing agreement after replacing a rater is not automatically better uncertainty alignment.
-
Label construction and evaluation denominators require clarification because they affect the reliability of the main comparisons. With the stated squared set-overlap distance, ratings C,C,C,D,D have mean pairwise distance 0.6 and pass the 0.75 consensus threshold despite a 3:2 split. Please show the rater-pattern distribution and sensitivity to the threshold rather than interpreting every resulting single label as unambiguously correct. Separately, the Opus confusion matrix in Figure 5 sums to 509 cases, with 434 correct predictions: 434/509 = 0.853, matching Table 2, whereas the nominal clear-case population is 527. The matrix supports a denominator discrepancy, but does not establish its cause. Opus conversational accuracy is also 0.719 in Table 2 and 0.727 in Table 13. Per-model/per-format exclusions and the paired evaluation population must be stated and the tables reconciled.
Overall assessment: I currently lean against acceptance, narrowly. The annotation resource and paired format evaluation are useful, but the uncertainty-alignment interpretation lacks a calibrated reference scale and the label/denominator issues affect confidence in the results. Clear baselines, sensitivity checks, and reconciled denominators could change my recommendation.
Questions for authors
- How do model rankings and uncertainty metrics compare with held-out-physician, sampling-noise, and simple-predictor baselines, with confidence intervals?
- What rater patterns occur within the annotated consensus split, and how sensitive are the main results to the agreement threshold?
- What accounts for the 509-versus-527 Opus denominator and the 0.719-versus-0.727 conversational accuracies? Please report exclusions and failure handling, including results counting failed outputs as errors where appropriate.
- Does the conversational error-direction result persist with a second judge or an explicit analysis of conditional escalation recommendations? How much of the observed difference depends on the rubric and on whether the four-level framework is shown to the model?
- In adjudication, can the ratings be removed while keeping the prompt and decoding settings fixed, so that the effect of exposing rater evidence is isolated?
3 comments
Response to Reviewer tzvm (2/2)
W2/Q3 Denominators. Thank you for identifying these discrepancies, and we apologize for this oversight. They arose from inconsistent handling of cases where none of a model’s five samples yielded a parseable label. Claude Opus 4.7 had 18 such clear cases in the QA format. The original Table 2 excluded these 18 cases from Opus’s QA denominator only (434/509 = 0.853), while still scoring its conversational accuracy on all 527 cases (0.719). The original Table 13 instead restricted both formats to the 509 cases labeled in both, which produced 0.727. Opus is the only model with any clear case in which all five samples failed.
We now apply one rule throughout, counting failed cases as errors, and the paired QA–conversational comparison uses all 527 cases for every model (Section 4.1.2). Opus’s accuracy is now 0.824 (QA) and 0.719 (conversational) in every table, and its confusion matrix (now Figure 7) annotates the 18 cases, so it accounts for all 527. Appendix E.4 reports failure counts for each model and format (Table 17), along with accuracy under alternative rules (Table 18). Under the counted-as-errors rule, model rankings are unchanged, Opus remains first, and the QA–conversational shift holds for all 12 models. If each failed sample counts against the model, Opus drops to fourth, so we no longer describe its first-place ranking as independent of this choice.
Q4 Second conversational judge model. The result holds with a second judge, Claude Sonnet 4.6. Under this judge from a different model family using the same rubric, over-triage falls and under-triage rises from QA to conversational for all seven re-judged models, as with GPT-4.1 (Appendix B.5, Figure 5). On the 132 human-labeled responses, Claude Sonnet 4.6 agrees with the two human annotators (weighted κ 0.771 and 0.857) as closely as GPT-4.1 does (0.783 and 0.846) and as the annotators agree with each other (0.773), and the two judges agree closely on identical responses (weighted κ 0.87–0.95). We think this strengthens the evidence that the shift reflects model responses rather than the judge.
The rubric deliberately scores the primary recommendation, the level of care a user would act on first, and we think separately scoring conditional escalation would be strong future work. The formats also differ in instructions by design, since giving the conversational prompt the four-level framework would turn it into a free-text version of the classification task. We now list the instruction difference as a limitation (Appendix J) and describe the comparison as explicit classification versus communicated urgency.
Q5 Adjudication control. We ran this control. Holding the adjudication prompt, model, and API settings fixed, we removed only the line listing the physician ratings and re-adjudicated all 217 ambiguous cases, and we re-ran the original with-ratings prompt to measure run-to-run variation (Appendix H.5). Two identical with-ratings runs differ on 12.0% of labels, while removing the ratings changes 28.1%. Rater evidence moves the adjudication toward the panel, since agreement with the panel’s ordinal median is 61.3% with ratings and 39.6% without. Relative to the direct QA prediction, the adjudicated label differs in 70 cases with ratings (net downward) and 48 without (mostly upward), so the downward shift we reported comes from the ratings. We also corrected two details in the Appendix H protocol description. Adjudication calls used the API’s default temperature, since GPT-5.4 does not accept a temperature setting, and the direct QA reference is the first of the five QA samples. We re-derived all reported numbers from the original outputs, and none change.
Post-rebuttal assessment
Thank you for the revision. I am raising my recommendation from 2 to 3 (weak acceptance).
Response to Reviewer tzvm (1/2)
Thank you for your review and for recognizing the value of the annotation protocol and the paired comparison of classification and conversational responses. Your suggestions identify concrete ways to make the results easier to interpret and assess. Below we do our best to address your concerns and welcome any follow-up questions.
Summary. You noted that clear baselines, sensitivity checks, and reconciled denominators could change your recommendation, and the revision addresses each. We replaced the distributional and panel-substitution analyses with an analysis of model and physician disagreement that uses a shuffled-label null, bootstrap intervals, and within-level checks (Section 5.1). We added simple-predictor baselines and rank intervals for the clear-case rankings (Appendix E.5), rater patterns and threshold sensitivity for the consensus split (Appendix A.4), and one denominator rule applied throughout (Appendix E.4). We also added a second judge (Appendix B.5) and the adjudication control you proposed (Appendix H.5).
W1/Q1 Reference baselines for the uncertainty analyses. After careful reconsideration, we agree that the distributional and panel-substitution analyses could not support the conclusions we drew. Repeated samples from one model and ratings from different physicians are different sources of variation, and the comparison was made on cases selected for physician disagreement. Without a reference scale, neither the distributional distances nor the changes in panel agreement could be interpreted, and higher agreement after substituting a rater does not show better alignment. We therefore removed both analyses.
The revised Section 5.1 is built around the references you suggest. Across all 667 annotated cases, it relates model disagreement to physician disagreement with Spearman correlation and AUROC for separating ambiguous from consensus cases, reporting 95% bootstrap intervals, a shuffled-label null, estimates within acuity levels, and results for all 792 five-model panels (Table 3, Appendix G). Disagreement across the 12 models reaches ρ = 0.27 [0.19, 0.34] and AUROC 0.63 [0.58, 0.67], above the null (AUROC 0.45–0.55) but far from perfect separation. Because the splits are defined by physician ratings, a held-out physician is not independent of the case selection, so the shuffled-label null and perfect separation serve as the reference points. For adjudication, a control that removes the ratings changes 28.1% of labels against 12.0% between identical runs, and lowers agreement with the panel median from 61.3% to 39.6% (Section 5.2.1, Appendix H.5).
Q1 Model rankings. For the clear-case rankings, we added simple-predictor baselines and confidence intervals (Appendix E.5, Tables 19 and 20). All 12 models exceed uniform and frequency-matched random predictors. Always-D reaches 0.497 because half of clear cases are D, and Llama 3.3 70B’s margin over it is not significant (+0.036 [−0.011, +0.085]). Under equal-acuity weighting, every constant predictor scores 0.25 and all models exceed it. Bootstrap intervals (2,000 case resamples) show that 50 of 66 model pairs are distinguishable and that the ranking forms tiers. Claude Opus 4.7 ranks first or second in 95% of resamples, while the five models ranked 2–6 cannot be separated, which is now described in Section 4.2.1.
W2/Q2 Rater patterns and threshold sensitivity. We now report the rater-pattern distribution and threshold sensitivity (Appendix A.4, Table 9, Figure 4). Patterns like C,C,C,D,D are rare. 15 of 450 annotated consensus cases show a 3:2 split between adjacent levels, and only 3 are split between C and D. A further 165 are unanimous or fully compatible, 177 differ only within one level, and 11 contain ratings more than one level apart, compared with 170 of 217 ambiguous cases. We agree that consensus labels reflect high but not unanimous agreement, and the rater-pattern table now makes this explicit. A stricter threshold (d̄ < 0.6) that excludes all 3:2 splits leaves the ranking of all 12 models unchanged (Kendall τ = 1.00) and preserves the QA–conversational shift for every model. Across thresholds from 0.3 to 1.5, rankings remain stable (τ ≥ 0.97, with at most one adjacent pair swapped at any threshold) and the shift holds throughout.