- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
KlinikeBench presents 333 clinician-authored tasks for evaluating clinical agents beyond diagnostic accuracy, together with a platform for constructing and revising cases. I find the paper interesting. The clinical agent environment is somewhat incremental relative to existing benchmarks, but the clinician involvement and case-authoring platform are valuable contributions.
Critical strengths
I appreciate the emphasis on evaluating history-taking and clinical actions alongside the final diagnosis (i.e. process-level focus). The clinician authoring and revision process is a strength, and the dialogue-quality assessment (30 clinicians providing 4,500 ratings) represents substantial effort. The platform for developing additional cases could also make the contribution useful beyond the released benchmark.
Critical weaknesses
-
Unnecessary or harmful actions are insufficiently addressed (L191–203). The evaluation emphasizes completing required tools and asking required questions. However, clinical decision-making also involves knowing which tests or interventions should not be performed. How does the evaluation penalize unnecessary or potentially harmful actions? Do the argument-level action constraints capture this? A measure of action appropriateness or precision would complement the current emphasis on coverage.
-
Clinical validity and interpretation of task difficulty need clarification (L163–170; Table 2, L275–277). Beyond the authoring and revision process, are cases independently assessed by other clinicians for clarity, solvability, and the appropriateness of their requirements? Performance from two or three independent physicians, reporting diagnostic accuracy and strict pass@1, would help contextualize the low model pass rates. Please also clarify whether “accepted alternative diagnoses” means synonymous diagnostic labels or genuinely different, clinically plausible diagnoses. Some presentations may reasonably support several differential diagnoses even after investigation; how are these handled?
-
The dialogue-quality study needs clearer justification and reporting (L307–314, L370–376; Table 3, L397–403). Why is MTS-Dialog an appropriate comparison, and what exact instructions were given to the human raters? Please report whether the observed differences are statistically significant. I also remain uncertain how strongly progressive history disclosure, emotional appropriateness, and human-likeness establish the clinical quality of the underlying tasks. These dimensions assess aspects of conversational realism, but do not directly establish that a case is clinically coherent or supports its intended diagnosis.
-
The interaction-budget results need a clearer separation of trends and variability (L330–340). Although the point estimates are not monotonic, they appear compatible with an overall improvement as the budget increases. I appreciate the confidence intervals in Appendix Table 8; repeated rollouts and uncertainty shown in the main figure would further help distinguish a reproducible trend from noise. It would also be useful to contextualize this result against AgentClinic’s interaction-budget analysis, which found declining diagnostic accuracy beyond a certain budget.
Questions for authors
Smaller comments and suggestions.
- Related work and novelty (L067–073; Table 1, L076–087): Please include PhysicianBench in Table 1 and discuss its process checkpoints in the main comparison. It is already discussed in Appendix E, but is particularly relevant to the motivation here. Similarly, patient-simulator sensitivity analysis (L364–368) has precedent in AgentClinic, so its novelty should be framed accordingly.
- Table 1, L083: AgentClinic was initially released in 2024.
- L125–127: The discussion of clinician-authored LLM benchmarks could also acknowledge prior work like SMMILE (Neurips '25 I believe).
- Figure 2, L116–122: Show the actual ratings instead of only “Higher mean quality ratings.”
- L241–242: As an optional extension, pass@3 or pass@5 would help characterize the benefit of additional attempts.
- L307–308 and Figure 5, L393–394: Consistently distinguish human clinicians from LLM clinician agents. In Figure 5, explicitly identify the agent models represented by the colored curves and the judge/patient models on the horizontal axes. Showing absolute scores alongside ranks would also make the figure more informative; these scores are currently in Table 9.
- Figure 6, p. 8; caption L429–431: The “Diagnosis” y-axis label is clipped.
- Section 4.2, particularly L432–438: The small SFT/RL study feels peripheral to the main contribution, and the RL findings remain inconclusive. Consider moving it to the appendix to keep the main presentation focused.
7 comments
Expanded Interaction-Budget Sensitivity and Accepted Alternative Diagnoses
4. Extended interaction-budget results against AgentClinic (Appendices D.4–D.5)
Contextualize this result against AgentClinic’s interaction-budget analysis, which found declining diagnostic accuracy beyond a certain budget.
Following your suggestion, we evaluated five models on 50 matched tasks with step caps from 4 to 60, plus an uncapped condition. The default is 32 steps, and the budget includes both dialogue and tool calls.
| Model | 4 steps | 8 steps | 16 steps | 32 (default) | 45 steps | 60 steps | No cap |
|---|---|---|---|---|---|---|---|
| GPT-6-astra | 0 | 0 | 7 | 28 | 34 | 32 | 36 |
| Claude Opus 5 | 1 | 1 | 10 | 34 | 36 | 40 | 38 |
| Grok-4.7 | 1 | 0 | 0 | 22 | 50 | 38 | 30 |
| Gemini-3.8-flash | 0 | 1 | 4 | 20 | 22 | 14 | 20 |
| Claude Sonnet 5 | 0 | 0 | 3 | 20 | 26 | 22 | 26 |
Strict pass@1 falls from 20–34% at 32 steps to 0–10% at 16 steps and 0–1% at 4–8 steps, with all paired-bootstrap 95% confidence intervals excluding zero for these reductions. For GPT-6-astra, halving the budget leaves diagnosis accuracy at 86% but reduces required-tool completion from 46% to 15%, showing that correct diagnosis can coexist with incomplete assessment. Larger budgets do not consistently help: Grok-4.7 rises from 22% at 32 steps to 50% at 45 steps, then falls to 30% without a cap
5. whether “accepted alternative diagnoses” means synonymous diagnostic labels or genuinely different, clinically plausible diagnoses.
In our cases, accepted alternative diagnoses includes both synonymous diagnostic labels and genuinely different. During the data annotation process, we use knowledge graph of sympotoms (L156) to add negative symptoms to disabiguiate the targeted diseases, in the cases there is no way to disguiwish the diseases from symptoms, the clinically plausible diagnoses are added as accepted alterantive diagnoses. We also provide the detailed pseudocode for how the diagnose is compared to the ground truth label in the reply to the review (XdHN) on the second question he/she proposed.
6. Clarification on the Diagnose-quality study.
Purpose of the study As we use simulated patients, it is very critical to ensure the behaviour of these simualted patients follows the real patients pattern. So we design this study to check if the simulated patient is distinguiable from the real patients. To study this, we sample real patient-clinician conversatoin from MTS-Dialog dataset. As performing such study we need to ensure the quality of the data and also the proper license for the data. the data provided by MTS-Dialog is hereby used.
Instructions Given to the clinicians in the study:
Survey format and clinician instructions
The questionnaire collects clinicians’ professional background information and presents patient–physician dialogues for evaluation. Below are the background questions, a sample dialogue, and the rating schema.
1. Clinician background questions
- Department and Current Professional Title
(e.g., Attending Physician, Department of Cardiology) - Which hospital or institution are you currently affiliated with?
- How many years of work experience do you have?
2. Instructions for dialogue evaluation
“Please read the dialogue in the figure and rate the patient's behavior. Use a 5-point scale: 1 = strongly disagree, 5 = strongly agree.”
3. Sample dialogue
Patient: I've had this pain in my belly for about a day and a half, and it's getting really bad, especially when I cough or move around. It started near my belly button but now it's mostly on the lower right side.
Physician: That pattern is concerning for appendicitis. Do you also have fever, nausea/vomiting, loss of appetite, or diarrhea/constipation, and when you press on the lower right side and let go does it worsen?
Patient: Yeah, I feel kind of feverish and sweaty, and I've been really nauseous—I threw up once and I basically haven't wanted to eat at all. I've also had some diarrhea and constipation, and yes, pressing on the lower right side and letting go makes it hurt a lot more.
Physician: This sounds very concerning for appendicitis, and because the pain is worsening with movement/coughing and you have rebound tenderness, you need urgent in-person evaluation in the ER now. Are you able to get to the emergency department right away, and do you have any severe dizziness, fainting, or trouble breathing?
Patient: I can get there right away, and I don't have any trouble breathing. I'm a little shaky and worried, but I haven't fainted or anything like that.
4. Rating schema
For each dialogue, clinicians select one score for each of the following three statements. The response options are 1, 2, 3, 4, and 5, anchored at 1 = strongly disagree and 5 = strongly agree.
| Evaluation dimension | Statement presented to clinicians | Response options |
|---|---|---|
| Progressive information disclosure | The patient only gradually supply the medical history when asked by the physician. | 1 / 2 / 3 / 4 / 5 |
| Emotional Congruence | The patient's mood and worries match how serious the condition is. | 1 / 2 / 3 / 4 / 5 |
| Overall impression | The patient is more like a real human patient rather than a robot. | 1 / 2 / 3 / 4 / 5 |
Each evaluated dialogue therefore receives three separate ratings, one per dimension.
3. Additional metrics on harmful actions and precision (Added two columns in the table 7)
We added two metrics to emphasis on the coverage and harmful behaviour into the main result (table 7) in the updated manuscript, and part of the results is shown bellow. We found that the GPT-5.5/5.6/6 family is the most careful (1–14%), with GPT-6-astra almost always checking allergies first (1.0%). The Claude models differ widely within the family. Opus 5.5 is at 12.3%, while Opus 4.8 (52.6%) and Sonnet 4.6 (79.5%) often prescribe without a check. For precision, broad tool users score lower: Grok, Gemini and DeepSeek are at 64–72%. The strongest models is able to use model wisely while achiving higher tool coverage, e.g. GPT-5.6-sol at 78.5% precision with 36.6% tools, and Opus 5.5 at 76.2% with 49.8%.
Unchecked prescribing: among encounters in which the agent prescribed (40–135 per model), the share where the first prescription came before any allergy or medication check, whether by tool or by asking the patient about allergies.
Action precision: for each encounter, the share of distinct clinical tools the agent called that were required for the case, averaged over encounters. Documentation tools such as record_diagnosis are excluded.
Our annotation platform lets clinician authors mark actions as forbidden for a case, and 204 of the 333 tasks include such entries, most often admit_to_hospital, prescribe_medication, refer_to_specialist and order_imaging. When the cases were reviewed, however, clinicians did not consistently agree on which actions should be prohibited. Whether a patient with panic symptoms or superficial thrombophlebitis should never be admitted, for example, depends on local practice and on details outside the encounter. We therefore did not score forbidden actions as a gate. In the submitted version they are instead withheld from the agent's tool set (Appendix B.1), so they restrict the action space but are not measured. The task format also supports argument-level constraints, such as requiring or prohibiting a specific drug in prescribe_medication(drug_name). This design follows the structure of clinical guidelines. They state what should be done for a presentation (the recommended work-up, safety checks and disposition) far more consistently than what must never be done; prohibitions are usually patient-specific, such as contraindications or allergies, rather than properties of the disease.