acceptodds
Under review as a conference paper at ICLR 2027

AcuityBench: Evaluating Clinical Acuity Identification and Uncertainty Alignment

Abstract

We introduce AcuityBench, a benchmark for evaluating whether language models identify the appropriate urgency of care from user medical presentations. Existing health benchmarks emphasize either medical question answering, broad health interactions, or narrow workflow-specific triage tasks, but they do not provide a unified evaluation of acuity identification across these settings. AcuityBench addresses this gap by harmonizing five public datasets spanning user conversations, online forum posts, clinical vignettes, and patient portal messages under a shared four-level acuity framework ranging from home monitoring to immediate emergency care. The benchmark contains 914 cases, including 697 consensus cases for standard accuracy evaluation and 217 physician-confirmed ambiguous cases for uncertainty-aware evaluation. It supports two complementary task formats: explicit four-way classification in a QA setting, and free-form conversational responses evaluated with a rubric-based judge anchored to the same framework. Across 12 frontier proprietary and open-weight models, we find substantial variation in clear-case acuity accuracy and error direction. Comparing task formats reveals a systematic tradeoff: conversational responses reduce over-triage but increase under-triage relative to QA, especially in higher-acuity cases. In ambiguous cases, no model closely matches the distribution of physician judgments, and model predictions are generally more concentrated than expert clinical uncertainty. We also compare expert and model adjudication on a subset of maximally ambiguous cases, using those cases to examine the role of clinical uncertainty in label disagreement. Together, these results position acuity identification as a distinct safety-critical capability and show that AcuityBench enables systematic comparison and stress-testing of how well models guide users to the right level of care in real-world health use.

Primary area:
datasets and benchmarks
Keywords:
healthcare, LLM, benchmark, uncertainty alignment, evaluation protocol, user safety, ai safety

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.60 over 5 reviews, above 70% of the others.
Reviewer_tzvmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AcuityBench harmonizes five public health datasets into a four-level acuity benchmark containing 914 cases. Of these, 667 receive five physician ratings each and 247 inherit source labels. The resulting evaluation separates consensus cases, including boundary labels, from 217 cases with substantial physician disagreement. Twelve language models are compared in explicit classification and conversational formats; the latter is mapped to acuity labels by a rubric-based model judge. The paper also studies alignment with physician endorsement distributions and model-assisted adjudication of disagreement.

Critical strengths
  1. The annotation protocol is a useful contribution. Five ratings per newly annotated case, explicit boundary labels, and retention of disagreement provide more information than a single hard label. The appendices document the annotation and aggregation procedures in considerable detail.

  2. The paired comparison of classification and conversational responses addresses a practical evaluation question. The same cases are evaluated in both formats, with confusion matrices, directional errors, bootstrap intervals, and paired tests. Under the stated rubric, over-triage decreases and under-triage increases across all twelve models; reporting these directions is more informative than reporting accuracy alone.

Critical weaknesses
  1. The uncertainty-alignment results need reference baselines before their magnitude can support the headline interpretation. Section 5 and Appendix G compare five stochastic samples from one model with the retained votes of up to five physicians, on cases selected specifically for physician disagreement. These are different sources of variation, and the selection makes low panel agreement expected. This does not make the comparison invalid or guarantee its outcome, but it leaves JSD/Wasserstein distances and changes in panel agreement difficult to interpret. A held-out-physician comparison, a sampling-noise reference for the empirical panel distribution, and simple central/uniform predictors would establish a useful scale. Confidence intervals are also needed before interpreting small differences between models. In particular, increasing agreement after replacing a rater is not automatically better uncertainty alignment.

  2. Label construction and evaluation denominators require clarification because they affect the reliability of the main comparisons. With the stated squared set-overlap distance, ratings C,C,C,D,D have mean pairwise distance 0.6 and pass the 0.75 consensus threshold despite a 3:2 split. Please show the rater-pattern distribution and sensitivity to the threshold rather than interpreting every resulting single label as unambiguously correct. Separately, the Opus confusion matrix in Figure 5 sums to 509 cases, with 434 correct predictions: 434/509 = 0.853, matching Table 2, whereas the nominal clear-case population is 527. The matrix supports a denominator discrepancy, but does not establish its cause. Opus conversational accuracy is also 0.719 in Table 2 and 0.727 in Table 13. Per-model/per-format exclusions and the paired evaluation population must be stated and the tables reconciled.

Overall assessment: I currently lean against acceptance, narrowly. The annotation resource and paired format evaluation are useful, but the uncertainty-alignment interpretation lacks a calibrated reference scale and the label/denominator issues affect confidence in the results. Clear baselines, sensitivity checks, and reconciled denominators could change my recommendation.

Questions for authors
  1. How do model rankings and uncertainty metrics compare with held-out-physician, sampling-noise, and simple-predictor baselines, with confidence intervals?
  2. What rater patterns occur within the annotated consensus split, and how sensitive are the main results to the agreement threshold?
  3. What accounts for the 509-versus-527 Opus denominator and the 0.719-versus-0.727 conversational accuracies? Please report exclusions and failure handling, including results counting failed outputs as errors where appropriate.
  4. Does the conversational error-direction result persist with a second judge or an explicit analysis of conditional escalation recommendations? How much of the observed difference depends on the rubric and on whether the four-level framework is shown to the model?
  5. In adjudication, can the ratings be removed while keeping the prompt and decoding settings fixed, so that the effect of exposing rater evidence is isolated?
3 comments
AuthorsOpenReview ↗

Response to Reviewer tzvm (1/2)

Thank you for your review and for recognizing the value of the annotation protocol and the paired comparison of classification and conversational responses. Your suggestions identify concrete ways to make the results easier to interpret and assess. Below we do our best to address your concerns and welcome any follow-up questions.

Summary. You noted that clear baselines, sensitivity checks, and reconciled denominators could change your recommendation, and the revision addresses each. We replaced the distributional and panel-substitution analyses with an analysis of model and physician disagreement that uses a shuffled-label null, bootstrap intervals, and within-level checks (Section 5.1). We added simple-predictor baselines and rank intervals for the clear-case rankings (Appendix E.5), rater patterns and threshold sensitivity for the consensus split (Appendix A.4), and one denominator rule applied throughout (Appendix E.4). We also added a second judge (Appendix B.5) and the adjudication control you proposed (Appendix H.5).

W1/Q1 Reference baselines for the uncertainty analyses. After careful reconsideration, we agree that the distributional and panel-substitution analyses could not support the conclusions we drew. Repeated samples from one model and ratings from different physicians are different sources of variation, and the comparison was made on cases selected for physician disagreement. Without a reference scale, neither the distributional distances nor the changes in panel agreement could be interpreted, and higher agreement after substituting a rater does not show better alignment. We therefore removed both analyses.

The revised Section 5.1 is built around the references you suggest. Across all 667 annotated cases, it relates model disagreement to physician disagreement with Spearman correlation and AUROC for separating ambiguous from consensus cases, reporting 95% bootstrap intervals, a shuffled-label null, estimates within acuity levels, and results for all 792 five-model panels (Table 3, Appendix G). Disagreement across the 12 models reaches ρ = 0.27 [0.19, 0.34] and AUROC 0.63 [0.58, 0.67], above the null (AUROC 0.45–0.55) but far from perfect separation. Because the splits are defined by physician ratings, a held-out physician is not independent of the case selection, so the shuffled-label null and perfect separation serve as the reference points. For adjudication, a control that removes the ratings changes 28.1% of labels against 12.0% between identical runs, and lowers agreement with the panel median from 61.3% to 39.6% (Section 5.2.1, Appendix H.5).

Q1 Model rankings. For the clear-case rankings, we added simple-predictor baselines and confidence intervals (Appendix E.5, Tables 19 and 20). All 12 models exceed uniform and frequency-matched random predictors. Always-D reaches 0.497 because half of clear cases are D, and Llama 3.3 70B’s margin over it is not significant (+0.036 [−0.011, +0.085]). Under equal-acuity weighting, every constant predictor scores 0.25 and all models exceed it. Bootstrap intervals (2,000 case resamples) show that 50 of 66 model pairs are distinguishable and that the ranking forms tiers. Claude Opus 4.7 ranks first or second in 95% of resamples, while the five models ranked 2–6 cannot be separated, which is now described in Section 4.2.1.

W2/Q2 Rater patterns and threshold sensitivity. We now report the rater-pattern distribution and threshold sensitivity (Appendix A.4, Table 9, Figure 4). Patterns like C,C,C,D,D are rare. 15 of 450 annotated consensus cases show a 3:2 split between adjacent levels, and only 3 are split between C and D. A further 165 are unanimous or fully compatible, 177 differ only within one level, and 11 contain ratings more than one level apart, compared with 170 of 217 ambiguous cases. We agree that consensus labels reflect high but not unanimous agreement, and the rater-pattern table now makes this explicit. A stricter threshold (d̄ < 0.6) that excludes all 3:2 splits leaves the ranking of all 12 models unchanged (Kendall τ = 1.00) and preserves the QA–conversational shift for every model. Across thresholds from 0.3 to 1.5, rankings remain stable (τ ≥ 0.97, with at most one adjacent pair swapped at any threshold) and the shift holds throughout.

AuthorsOpenReview ↗

Response to Reviewer tzvm (2/2)

W2/Q3 Denominators. Thank you for identifying these discrepancies, and we apologize for this oversight. They arose from inconsistent handling of cases where none of a model’s five samples yielded a parseable label. Claude Opus 4.7 had 18 such clear cases in the QA format. The original Table 2 excluded these 18 cases from Opus’s QA denominator only (434/509 = 0.853), while still scoring its conversational accuracy on all 527 cases (0.719). The original Table 13 instead restricted both formats to the 509 cases labeled in both, which produced 0.727. Opus is the only model with any clear case in which all five samples failed.

We now apply one rule throughout, counting failed cases as errors, and the paired QA–conversational comparison uses all 527 cases for every model (Section 4.1.2). Opus’s accuracy is now 0.824 (QA) and 0.719 (conversational) in every table, and its confusion matrix (now Figure 7) annotates the 18 cases, so it accounts for all 527. Appendix E.4 reports failure counts for each model and format (Table 17), along with accuracy under alternative rules (Table 18). Under the counted-as-errors rule, model rankings are unchanged, Opus remains first, and the QA–conversational shift holds for all 12 models. If each failed sample counts against the model, Opus drops to fourth, so we no longer describe its first-place ranking as independent of this choice.

Q4 Second conversational judge model. The result holds with a second judge, Claude Sonnet 4.6. Under this judge from a different model family using the same rubric, over-triage falls and under-triage rises from QA to conversational for all seven re-judged models, as with GPT-4.1 (Appendix B.5, Figure 5). On the 132 human-labeled responses, Claude Sonnet 4.6 agrees with the two human annotators (weighted κ 0.771 and 0.857) as closely as GPT-4.1 does (0.783 and 0.846) and as the annotators agree with each other (0.773), and the two judges agree closely on identical responses (weighted κ 0.87–0.95). We think this strengthens the evidence that the shift reflects model responses rather than the judge.

The rubric deliberately scores the primary recommendation, the level of care a user would act on first, and we think separately scoring conditional escalation would be strong future work. The formats also differ in instructions by design, since giving the conversational prompt the four-level framework would turn it into a free-text version of the classification task. We now list the instruction difference as a limitation (Appendix J) and describe the comparison as explicit classification versus communicated urgency.

Q5 Adjudication control. We ran this control. Holding the adjudication prompt, model, and API settings fixed, we removed only the line listing the physician ratings and re-adjudicated all 217 ambiguous cases, and we re-ran the original with-ratings prompt to measure run-to-run variation (Appendix H.5). Two identical with-ratings runs differ on 12.0% of labels, while removing the ratings changes 28.1%. Rater evidence moves the adjudication toward the panel, since agreement with the panel’s ordinal median is 61.3% with ratings and 39.6% without. Relative to the direct QA prediction, the adjudicated label differs in 70 cases with ratings (net downward) and 48 without (mostly upward), so the downward shift we reported comes from the ratings. We also corrected two details in the Appendix H protocol description. Adjudication calls used the API’s default temperature, since GPT-5.4 does not accept a temperature setting, and the direct QA reference is the first of the five QA samples. We re-derived all reported numbers from the original outputs, and none change.

Reviewer_tzvmOpenReview ↗

Post-rebuttal assessment

Thank you for the revision. I am raising my recommendation from 2 to 3 (weak acceptance).

Reviewer_tWcUOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposed a benchmark of LLMs identifying urgency of care in medical conversations. It constructs the benchmark from 5 datasets, results in 914 cases, with 697 consensus cases and 217 ambiguous cases. It tested 12 models on both multi-choice form and free-form responses.

Critical strengths
  1. The ambiguous cases are useful resources, which may enable further research about disagreement among clinicians and uncertainty in AI evaluation.
  2. The paper addresses an important clinical problem.
Critical weaknesses
  1. The largest concern is the limited incremental contribution over HealthBench. HealthBench already includes 482 emergency referral cases with rubrics assessing urgency and care-seeking recommendations. This benchmark combines a subset of HealthBench with four additional datasets, yielding 697 consensus cases. However, there is limited comparison with HealthBench or discussion of what this evaluation adds. In particular, how do the findings differ from those obtained by directly applying HealthBench’s existing rubrics?

  2. In label construction, some labels from HealthBench, Semigran, and Ramaswamy are directly mapped to the new framework without additional physician annotation. However, the paper provides insufficient analysis of whether the original label definitions and annotation criteria align with the new protocol. The authors should reannotate a subset of these cases and report agreement between the original labels and the new physician annotations.

  3. The ambiguous cases are useful, but the claim that models fail to reflect expert disagreement requires stronger justification. Random variation across repeated samples from the same model is not equivalent to variation in judgments across different physicians. The paper should justify why these two distributions should match and what their mismatch establishes about the model’s ability to represent clinical uncertainty.

  4. The annotation protocol lacks transparency. The paper states that 20 physicians participated and each case was independently annotated by five physicians, but does not explain how cases were assigned. Were physicians randomly assigned to cases or organized into fixed panels? How many cases did each physician annotate, and were assignments balanced across source datasets?

  5. The reported agreement relies on a custom distance based on label-set overlap. The authors should additionally report exact label agreement and conventional ordinal Krippendorff’s α, with an explicit explanation of how boundary labels are handled. For example, the custom distance treats A|B and B|C as having zero disagreement because they share B, even though the annotations are different. Therefore, agreement under this metric should be distinguished from agreement on the original labels.

Questions for authors
  1. What findings go beyond those obtained using HealthBench’s existing rubrics?

  2. How consistent are directly mapped labels with physician annotations under the new protocol?

  3. Why should repeated samples from one model match disagreement across physicians?

  4. How were cases assigned, and how many did each physician annotate?

  5. Can you report exact label agreement and standard ordinal Krippendorff’s α?

2 comments

Thank you for your review and for recognizing the clinical importance of the task and the value of retaining ambiguous cases. Below we address your concerns and welcome any follow-up questions.

W1/Q1 Contribution beyond HealthBench. The two benchmarks ask different questions. For the 369 Emergency Referrals cases we include, HealthBench scores each response against a median of 11 physician-written criteria spanning many behaviors, aggregated into a single score. AcuityBench asks which level of care a response recommends and whether it is right, yielding per-case accuracy, error direction (over- versus under-triage), and, by evaluating each case in classification and conversational formats, the gap between explicit classification and communicated urgency. This requires finer labels than HealthBench’s three categories.

For example, of the 80 HealthBench non-emergent conversations in our consensus split, our panels labeled 5 as emergent (D), and the remaining 75 span three levels, 22 home care (A or A|B), 38 semi-urgent (B or B|C), and 15 urgent (C or C|D). Across the 12 models, conversational responses under-triaged 53–87% of the clear C and D cases among these (n = 15) but over-triaged only 0–10% of the clear A and B cases (n = 42), a pattern of delayed care an aggregate rubric score does not reveal.

AcuityBench also curates HealthBench for the acuity task. Of the 453 Emergency Referrals conversations in its consensus release, we excluded 84 that could not support an acuity judgment, because they lacked sufficient clinical information (e.g., solely “infant crying 2 hours”), were not in English, or were removed by a majority of our physicians (Appendix A.1). Of the remaining 369, 230 required new physician labels. AcuityBench also adds 545 cases from vignettes, patient portal messages, and forum posts.

W2/Q2 Directly mapped labels. The mapping rule and each source’s original definitions are now explicit (Section 3.3, Table 5). A source label is mapped directly only when it served as the clinical reference standard in a published triage evaluation and its definition specifies a single AcuityBench level. Ramaswamy et al. (Nature Medicine, 2026), from whom we adopt the framework and boundary labels, assigned labels by three physicians with consensus adjudication (Fleiss’ κ = 0.90), and we use them as-is. HealthBench emergent and Semigran emergent care required both specify immediate emergency care and map to D, and Semigran self care reasonable maps to A. Every other category requiring a finer distinction was annotated by our panel. We agree that re-annotating a subset would provide direct validation. Because these labels already served as reference standards under equivalent definitions, we prioritized physician effort for cases without a specified level of care, and list re-annotation as a limitation (Appendix J).

W3/Q3 Model and physician variation. After careful reconsideration, we agree that variation across repeated samples from one model and across different physicians are different sources of uncertainty, and the paper did not justify why their distributions should match. We removed the distributional comparison.

Section 5.1 now asks whether models disagree more on the cases physicians dispute. Across all 667 annotated cases, we relate disagreement across the 12 models, and within each model across its five samples, to physician disagreement, with bootstrap intervals, a shuffled-label null, and estimates within acuity levels (Table 3, Appendix G). Model disagreement rises with physician disagreement only weakly (ρ = 0.27, AUROC 0.63, null 0.45–0.55). Section 5.2 shows that a model given the ratings uses them and, in an expert case study, identifies the clinical source of the disagreement.

W4/Q4 Case assignment. Cases were assigned to physicians at random, with five independent ratings per case, and each physician rated 168 or 169 of the 675 cases sent to annotation. Assignment was not stratified by source, so balance follows from randomization (Section 3.3, Appendix A.2).

W5/Q5 Exact agreement and ordinal α. We now report both (Appendix A.4, Table 8). Across all 667 annotated cases, ordinal α is 0.514 [0.476, 0.553], exact agreement over the seven labels is 36.5%, and 89.1% of rater pairs are within one level. These measures give boundary labels no overlap credit, so A|B and B|C are one level apart. We use set-overlap (α = 0.550, Table 7) to define the splits because a boundary label endorses both levels as clinically defensible, so raters sharing an endorsed level agree on at least one appropriate level of care. Set-overlap thus measures compatibility, which is less strict than agreement on the labels themselves.

Thank authors for the careful revision. I am raising my score to 3.

Reviewer_RWwNOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary
  • The authors present AcuityBench, an LLM benchmark on identifying the appropriate urgency of care, with 4 labels: A) monitor at home, B) see a doctor within weeks, C) see a doctor within 24-48 hours, D) go to the emergency department now

  • They merged five public datasets (HealthBench, Semigran, Ramaswamy, PMR-Synthetic, PMR-Reddit) and split them into two subsets: consensus cases (697) and ambiguous cases (217)

  • Original labels were either directly mapped to the four AcuityBench labels when possible (247 cases), or adjudicated by 5 physicians drawn from a panel of 20 board-certified emergency medicine attending physicians (667 cases)

  • They evaluated 8 proprietary and 4 open-weights models on AcuityBench under two evaluation formats: 4-way classification (QA) and free-form responses

  • For clear-case acuity classification, they reported the classification accuracies as well as over-triage and under-triage rates

  • For the ambiguous subset, they reported tests of whether model prediction distributions align with expert judgment, whether substituting a model for a human rater materially shifts panel consensus, and how models adjudicate physician disagreements and identify the clinical crux of disagreement.

Critical strengths
  • The paper identifies and fills a gap in the current available medical LLM benchmarks: identifying the appropriate urgency of care. AcuityBench is not just a repackaging of existing benchmarks, but includes new expert labels which is always a valuable contribution.

  • The introduction and related work sections give a good overview of existing medical LLM benchmarks, making the paper accessible to readers unfamiliar with this area.

  • The paper overall is well written, well organized, and easy to understand.

  • Well-chosen set of analyses for ambiguous cases: whether model prediction distributions align with expert judgment, whether substituting a model for a human rater materially shifts panel consensus, and how models adjudicate physician disagreements and identify the clinical crux of disagreement.

Critical weaknesses

None. No material weaknesses that support rejection. However one issue the paper has is that Section 3.2 describes the five source datasets in a single sentence, and the appendix covers only their relabeling, not their origin, construction, or original label schemes. A compact summary of each source in the main text of the paper is needed.

Questions for authors

None

2 comments

Thank you for your review and for recognizing the contribution of the new expert annotations and the retained ambiguous cases. Your suggestion to describe the source datasets more fully helps us explain the harmonization process more clearly.

Source dataset descriptions. We have added a concise description of each source to Section 3.2, covering its origin, construction, and original label scheme, and a new appendix table (Table 5, Appendix A.1) detailing each source’s original categories and definitions, how its labels were assigned, and how each maps to AcuityBench. We hope these additions make the benchmark’s construction easier to evaluate and to build on.

Revisions to the ambiguous-case analyses. Because you highlighted the ambiguous-case analyses as a strength, we want to note how they changed. Based on concerns raised by other reviewers, we removed the distributional comparison and the panel-substitution analysis, since repeated samples from one model and ratings from different physicians are different sources of variation. Section 5.1 now asks, across all 667 annotated cases, whether models disagree more where physicians disagree, with confidence intervals and a shuffled-label null. Section 5.2 keeps the adjudication analysis, now with a control that isolates the effect of the ratings. Our comment to the Area Chair summarizes these changes.

Thank you for the response and the revisions. The expanded source dataset descriptions in Section 3.2 and the new Table 5 in Appendix A.1 directly address my concern. I also appreciate the detailed update on the ambiguous-case analyses. Refocusing Section 5 on whether models disagree more where physicians disagree, and keeping the adjudication analysis with the new control, is a sensible response to the concerns raised, and I think the paper is stronger for it. I stand by my original rating of 4: clear acceptance.

Reviewer_GweoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

AcuityBench evaluates clinical acuity identification for evaluating whether langauge models identify the appropriate urgency of care from user medical presentations across five data sources using a shared four-level framework. It combines existing labels with new annotations from emergency physicians and uses cases with physician disagreement to create to constitute both a consensus and an ambiguous subset. The study compares explicit QA classification with urgency communicated in free-form responses within multi-turn conversation, and also examines model physician distributional alignment, panel substitution, and adjudication. The results show different over-/under-triage patterns across models and a shift toward lower communicated urgency in the conversational protocol.

Critical strengths
  1. The annotation effort makes this a useful resource for acuity evaluation. Five physicians independently rate each newly annotated case, and the use of boundary labels and retained disagreements avoids forcing every case into a single definitive answer, allowing for uncertainty and ambiguity measurement.
  2. The paired QA and conversational evaluation is informative. Separating over- and under-triage shows why similar accuracy scores can hide different errors. The distinction between identifying a source of uncertainty and agreeing on a final care recommendation particularly interesting.
Critical weaknesses
  1. The overall setup is solid, especially considering including ambiguity from professional responses to constitute the ambiguous subset is useful. However, I wonder if the uncertainty evaluation excludes cases where there is too little information to assign a care level, rather than just being ambiguous. This might leave out an important part of patient-facing triage: recognizing when clarification is needed. The scope should be stated more clearly alongside a comparison with earlier triage work. There is also some recent work that I noticed is not included, which could be useful to mention and add to the related work sections or relevant literature.
  • Gaber, Farieda, et al. "Evaluating large language model workflows in clinical decision support for triage and referral and diagnosis." npj Digital Medicine 8.1 (2025): 263.
  • Kopka, Marvin, Niklas von Kalckreuth, and Markus A. Feufel. "Accuracy of online symptom assessment applications, large language models, and laypeople for self–triage decisions." npj Digital Medicine 8.1 (2025): 178.
  • Hua, Yining, Hongbin Na, and Cyrus Ayubcha. "CARE-Bench: Benchmarking Patient-Facing LLM Triage." arXiv preprint arXiv:2608.03731 (2026).
  1. I think Section 5 is generally interesting, but it is difficult to follow without looking at the appendix from time to time. For key methods and findings, such as the distribution construction, I think some details could be moved, or at least have the main takeaways and findings briefly explained in the main paper. Also, the definitions and relationships between the consensus and ambiguous splits, as well as the single-label or boundary-level labels, could be explained better. The clarity of the writing for these could be improved.
Questions for authors
  1. If I am understanding the setting correctly, physicians can choose seven labels, including boundary labels, whereas models must select one of A, B, C, D. Would it be helpful if we also allow LLM to select or map the free-response answer (when using LLM as a judge) the boundary labels as well? How should this asymmetry affect the interpretation of uncertainty alignment?
  2. I would also be interested in whether model stability tracks physician agreement across all 667 newly annotated cases. Appendix G.4 addresses concentration on the ambiguous subset, but not this relationship across both splits.
  3. Most of the models evaluated are not the most up-to-date ones. It could be useful if you can include some of the more recent open-weight baselines, and consider also comparing the performance of these models and see how the accuracy and tendency (for over-/under- triage)of these models vary from time to time.
1 comment

Thank you for your review and for recognizing the value of the annotation effort and paired QA/conversational evaluation. Below we address your concerns and welcome any follow-up questions.

W1 Related work and clarification scope. We agree that recognizing when clarification is needed is an important part of patient-facing triage. AcuityBench excludes cases with too little information to assign any level of care in two ways. Cases lacking sufficient clinical specificity were removed during pre-screening with physician input (Appendix A.1), and annotators could mark a case Remove when information was insufficient to assign even a boundary label, with cases receiving a majority of Remove votes excluded (Appendix A.3). Some retained cases naturally involve missing information, and some disagreement in the ambiguous split may reflect it. We now state this scope in the limitations (Appendix J). The adjudication analysis offers a partial view, since the clinical crux the model identifies often names the information that could resolve the disagreement. By retaining all physician ratings, AcuityBench supports follow-up studies that separate disagreement driven by missing information from genuine clinical disagreement.

We added the three suggested references to the related work (Section 2.1). CARE-Bench is closest to our setting. It evaluates patient-facing triage as an action at each turn while patient information is disclosed sequentially, including when to ask for more information. AcuityBench complements it by asking which level of care a complete current presentation warrants, labeled by physician panels with their disagreement retained.

W2 Clarity of Section 5. We rewrote Section 5 to be self-contained. Model and physician disagreement are now defined in the main text, each subsection opens with its main result, and statistical details are in Appendices G and H. The section introduction also restates how the ambiguous split is defined. Based on feedback from other reviewers and careful consideration, we also removed the distributional comparison, as described in our general response.

Q1 Boundary labels. Great question, and the asymmetry here is deliberate and reflects two different roles. Physicians act as annotators characterizing the case, so endorsing two adjacent levels as clinically defensible conveys real information about it. Models are evaluated as patient-facing advisors, where a person asking how urgently they need care requires a single actionable recommendation. Models are not penalized for this constraint, since on boundary-label cases a prediction of either constituent level is scored as correct (Appendix D), following Ramaswamy et al., from whom we adopt the framework. For the same reason, the conversational judge scores the single level of care a response recommends, the level a user would act on first.

We agree that this asymmetry complicates any direct comparison of model and physician label distributions, since boundary votes add spread that a single-label prediction cannot express. Together with sampling concerns raised by other reviewers, this led us to remove the distributional comparison (see W2).

Q2 Model stability across all annotated cases. Given the concerns other reviewers raised about the distributional comparison, we found that your suggested analysis better captures the question we want to ask, whether model disagreement tracks physician disagreement. We ran this analysis, and it now anchors Section 5.1. Across all 667 annotated cases, single-model instability, the share of a model’s five samples that differ from its most common label, correlates with physician disagreement (ρ = 0.18 [0.10, 0.25]) and separates ambiguous from consensus cases slightly above chance (AUROC 0.58 [0.53, 0.63]), but within the ambiguous split it no longer tracks the degree of disagreement (ρ = 0.01, Appendix G). Disagreement across the 12 models gives a similar signal (ρ = 0.27, AUROC 0.63, against a shuffled-label null of 0.45–0.55). Model disagreement therefore reflects physician disagreement only weakly. We appreciate this suggestion, which made Section 5 and the contribution of the ambiguous split clearer.

Q3 Recent open-weight baselines. We evaluated a current release from each open-weight family in the paper, DeepSeek V4 Pro, Qwen3.5 9B, and Meta’s Muse Spark Open, on the 527 clear cases (Appendix E.6, Table 21). Because one release per family does not support claims about change over time, we report these as additional baselines, and our main findings hold for them. Models with similar accuracy differ in error direction, with DeepSeek V4 Pro and Qwen3.5 9B nearly tied in QA accuracy while Qwen3.5 9B over-triages more than twice as often, and under-triage rises from QA to conversational for all three. Because the benchmark and code are released, newer models can be evaluated as they appear.

Reviewer_rK61OpenReview ↗
Rating
1(−1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AcuityBench, a benchmark for evaluating whether language models identify and communicate appropriate levels of medical urgency. It combines five existing public datasets into 914 cases under a shared four-level acuity framework, ranging from home monitoring to immediate emergency care. Labels are obtained through direct mapping or physician annotation, yielding 697 consensus cases and 217 cases with substantial disagreement. Twelve models are evaluated using explicit classification and free-form conversational responses, with the latter assessed by a GPT-4.1 judge. The authors report variable classification accuracy and a shift toward lower communicated urgency in conversational responses, reducing over-triage while increasing under-triage. Overall, the work provides a limited medical evaluation resource based on integrating existing dataset resource. Some clear limitations are found, including insufficient medical justification for the proposed four-level categorization, vague annotation criteria that rely on physicians' subjective judgments, reuse of existing datasets rather than providing new clinical data resource. The evaluated LLM models are somehow outdated, authors should consider using more recent versions of these model families.

Critical strengths
  • AcuityBench consolidates 914 cases from five public datasets on conversations, clinical vignettes, patient portal messages, and Reddit posts under a shared acuity framework. Source-specific results allow readers to examine performance differences across these settings.

  • The annotation process involves several emergency physicians, with five independent ratings per case. Boundary labels and the retention of 217 disagreement cases preserve information that would be lost through a single majority-vote label.

  • Evaluating the same cases through explicit classification and free-form responses allows the study to examine differences between selected acuity labels and communicated recommendations.

  • The paper separately reports over-triage and under-triage, with breakdowns by acuity level and source dataset, exposing different error patterns across models.

Critical weaknesses

Critical and Inherent Weaknesses of this Work. My most critical concerns to this work are still that:

  • The paper does not introduce a new, original medical dataset. Instead, it integrates 914 clinical cases from five already-existed public datasets. Moreover, these cases are scattered across different clinical domains rather than focusing on a particular medical specialty or disease. As a benchmark & resource paper, the current workload and efforts is substantially insufficient.

  • The limited dataset size raises concerns about the benchmark's utility for evaluating clinical acuity and subsequent model development. A substantially larger volume of data would be needed if the dataset is intended to support developing AI models for clinical tasks. For example, MIMIC-IV v3.1, a well-known open clinical electronic health record dataset, contains records from 364,627 unique patients and 546,028 hospital admissions. Although these records are not directly equivalent to this benchmark, the comparison highlights its limited scale. Evaluating clinical acuity using a dataset of only 914 cases yields results that lack statistical significance.

  • Physicians' judgments can be highly subjective and vary considerably across individuals. The limited number of cases within any particular medical specialty or disease category makes it difficult to mitigate the impact of this subjectivity.


Less important weaknesses:

The quality and presentation of the Figures and Tables should be further improved. For example, Table 1 extends beyond the column width. In the major experimental results Figures 2 and Figure 3, the numbers and texts are too small to read without substantially zooming in on the PDF. Authors should put more efforts to improve the quality of the illustration contents.


W1: Insufficient validation of the harmonized ground truth. This study does not independently establish that labels across all five data sources are clinically interchangeable. It directly maps 247 cases without re-annotation, while instructions for newly annotated cases invoke clinical guidelines without specifying symptom-specific decision criteria or requiring supporting rationales.

W2: The consensus/ambiguous split depends on an unvalidated operational threshold. Appendix A.4 uses a fixed average-distance threshold of 0.75 without reporting sensitivity to this choice or alternative disagreement measures. Under this rule, three C ratings and two D ratings produce a distance of 0.6 and enter the consensus split, despite substantial disagreement about immediate emergency care. Consequently, the split does not cleanly distinguish clinically unambiguous cases from uncertain ones, and the high agreement reported after filtering is partly a consequence of that selection.

W3: The uncertainty analysis does not adequately establish model overconfidence. Model distributions come from only five stochastic single-label responses, whereas physician distributions additionally spread boundary-label endorsements equally across two categories, creating different measurement procedures (Appendix G.2). Without sampling-matched reference baselines or sample-size and temperature sensitivity analyses, the reported distributional distances cannot establish how much mismatch reflects substantive uncertainty misalignment rather than estimation noise and representation choices.

W4: The QA-conversation comparison does not isolate communication failures. QA prompting provides explicit acuity definitions, resource requirements, and step-by-step reasoning instructions, whereas conversational prompting does not provide equivalent guidance (Appendix B.1-B.3). The observed differences therefore combine changes in task instructions, response format, and scoring procedure, limiting the interpretation that models recognize the correct urgency but fail to communicate it.

W5: Panel-substitution metrics are given stronger interpretations than they support. Increased Krippendorff's α after replacing a physician establishes greater panel agreement, but does not independently establish more clinically appropriate or more human-like judgments, particularly on cases selected for low human agreement. Similarly, changes in the panel median measure decision instability without showing whether those changes improve or worsen clinical decisions.

Questions for authors

I thank the authors for their response. However, I believe the current design of AcuityBench has some critical and inherent weaknesses, which can not be resolved in the current version.

1 comment

Thank you for your review and for recognizing the value of the physician annotations and paired evaluation. Below we address your concerns and welcome any follow-up questions.

Summary concerns. The framework is adopted from Ramaswamy et al. (Nature Medicine, 2026), where it served as the physician-adjudicated reference standard in an evaluation of LLM triage, and it defines each level by the timing and resources of care a person should seek (Section 3.1). Our main findings also hold for three recent open-weight releases (Appendix E.6), and because the benchmark and code are released, newer models can be evaluated as they appear.

W1 Label harmonization. We have made the mapping rule and each source’s original labels explicit (Section 3.3, Table 5). A source label is mapped directly only when it served as the clinical reference standard in a published triage evaluation and its definition specifies a single AcuityBench level. Ramaswamy et al. labels, from which we adopt the framework, were assigned by three physicians with consensus adjudication (Fleiss’ κ = 0.90) and are used as-is. HealthBench emergent and Semigran emergent care required both specify immediate emergency care and map to D, and Semigran self care reasonable maps to A. All other categories were annotated by our panel.

The annotation instructions define each level by when care is needed and what resources it requires (Appendix A.3), so mapped and annotated labels share one standard. The criteria are level-based rather than symptom-specific because the cases span heterogeneous presentations and the label captures an emergency physician’s level-of-care judgment. Where that judgment varies, AcuityBench retains the disagreement through boundary labels and the ambiguous split. We list re-annotation of mapped labels as a limitation (Appendix J).

W2 Consensus/ambiguous split threshold. The 0.75 threshold is a design choice for obtaining reliable reference labels, so consensus agreement is high partly by construction. We now state this and report agreement across all annotated cases before splitting (ordinal α = 0.514), alongside exact agreement (Appendix A.4, Table 8). Patterns like C,C,C,D,D are rare (Table 9). 15 of 450 annotated consensus cases show a 3:2 adjacent split, 3 of them between C and D, and only 11 contain ratings more than one level apart, against 170 of 217 ambiguous cases. A stricter threshold (d̄ < 0.6) that excludes all 3:2 splits leaves the ranking of all 12 models unchanged (Kendall τ = 1.00), and across thresholds from 0.3 to 1.5 rankings remain stable (τ ≥ 0.97) and the QA–conversational shift holds (Figure 4).

W3 Distributional comparison. After careful reconsideration, we agree with your concerns. Variation across five samples from one model and across different physicians are different sources of uncertainty, so a distance between their distributions does not directly measure alignment, and without sampling-matched baselines it could not separate misalignment from estimation noise. We removed this comparison.

Section 5.1 now asks whether models disagree more on the cases where physicians disagree. Across all 667 annotated cases, we relate disagreement across the 12 models, and within each model across its five samples, to physician disagreement, with bootstrap intervals, a shuffled-label null, and estimates within acuity levels (Table 3, Appendix G). Disagreement across the 12 models correlates with physician disagreement (ρ = 0.27 [0.19, 0.34]) and separates the splits above chance (AUROC 0.63 [0.58, 0.67], null 0.45–0.55), but only weakly.

W4 QA–conversation comparison. The formats differ in instructions by design, since giving the conversational prompt the QA definitions would make it a free-text version of the classification task. We now describe the comparison as explicit classification versus communicated urgency (Sections 1 and 4.3.2) and list the instruction difference as a limitation (Appendix J). The judge is unlikely to drive the effect. It showed no systematic downward difference from two human annotators on 132 responses (Appendix B.4), and a second judge from a different model family reproduces the shift for all seven re-judged models (Appendix B.5).

W5 Panel substitution. As with the distributional comparison, after careful reconsideration we agree on the limitations of this analysis. Higher panel agreement after substitution does not show more clinically appropriate or more human-like judgment, and shifts in the panel median do not show whether decisions improve. We removed the panel-substitution analysis. The revised Section 5 instead relates model disagreement to physician disagreement across all 667 annotated cases, with a shuffled-label null and confidence intervals, and tests adjudication with a control that isolates the effect of the ratings (Section 5.2.1, Appendix H.5).

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of revisions

We thank the reviewers for careful and constructive reviews. People increasingly ask AI systems how urgently they need care, and errors in either direction matter, from delayed emergency care to unnecessary visits. AcuityBench measures this capability directly, with new labels from 20 emergency physicians that retain their disagreement. The reviews identified real weaknesses, and we believe the revision addresses them and makes the benchmark more rigorous and easier to build on. We welcome any follow-up questions or concerns.

Dataset construction. Section 3.2 now summarizes each source’s origin, construction, and original labels, and Table 5 (Appendix A.1) gives each source’s definitions and how its labels were assigned. A source label is mapped directly only when it served as the clinical reference standard in a published triage evaluation and its definition specifies a single AcuityBench level. All other categories were annotated by our panel. We now report how cases were assigned (at random, 168 or 169 per physician), exact agreement and standard ordinal Krippendorff’s α across all 667 annotated cases before splitting (0.514 [0.476, 0.553]), and the rater patterns the consensus threshold admits (15 of 450 consensus cases show a 3:2 adjacent split). Model rankings remain stable across thresholds from 0.3 to 1.5 (Kendall τ ≥ 0.97), and the QA–conversational shift holds throughout (Appendix A.4).

Clear-case results. We added simple-predictor baselines and bootstrap rank intervals, which show that the ranking forms tiers (Appendix E.5). With a second judge from a different model family, Claude Sonnet 4.6, which agrees with human annotators as closely as GPT-4.1 does, the shift from over-triage to under-triage in conversational responses holds for all seven re-judged models (Appendix B.5). We also evaluated three recent open-weight releases, for which the main findings hold (Appendix E.6). Following Reviewer tzvm, we reconciled all denominators, so failed outputs now count as errors and every model is evaluated on the same 527 clear cases (Appendix E.4).

Ambiguous cases. After careful reconsideration, we agree with the reviewers that the distributional uncertainty-alignment and panel-substitution analyses could not support the conclusions we drew. Repeated samples from one model and ratings from different physicians are different sources of variation, and the comparison was made on cases selected for physician disagreement. We removed both and refocused Section 5 on what the ambiguous split uniquely enables. Section 5.1, following Reviewer Gweo’s suggestion, asks across all 667 annotated cases whether models disagree more where physicians disagree, with bootstrap intervals, a shuffled-label null, and checks within acuity levels. They do, but only weakly (Spearman ρ = 0.27, AUROC 0.63 against a null range of 0.45–0.55). Section 5.2 retains the adjudication analysis, which asks whether a model shown the physicians’ ratings uses them and identifies the clinical source of the disagreement. A new control shows that removing the ratings changes 28.1% of adjudicated labels, against 12.0% between identical runs, and in an expert case study a physician rated the model’s account of the disagreement at least 4 of 5 in all 20 cases. We believe this better reflects the contribution of retaining physician disagreement as a resource.

Additional notes. We corrected two details in the adjudication protocol description (Appendix H). Calls used the API’s default temperature, and the direct QA reference is the first of five samples. No reported numbers change.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.