acceptodds
Under review as a conference paper at ICLR 2027

ClinTRACE: Benchmarking Clinical Reasoning as Evidence Accumulates

Abstract

AI-assisted clinical decision support requires large language models (LLMs) to reason and make decisions as evidence accumulates, not merely predict final diagnoses. However, existing benchmarks rarely provide expert reasoning traces aligned with the evidence available at each stage. To address this gap, we introduce ClinTRACE, a longitudinal clinical reasoning benchmark built from live, multi-expert discussions of real-world cases. Unlike retrospective rationales, these discussions capture physicians' reasoning before subsequent findings are revealed. Using a physician co-designed taxonomy and an audited data-extraction pipeline, we structure these discussions into time-aligned trajectories that link available evidence to expert interpretations, hypothesis updates, and proposed actions. The dataset includes 1,464 public cases with 84,044 atomic reasoning steps, plus 184 non-public cases. We formulate five complementary tasks to evaluate longitudinal clinical reasoning: progressive diagnosis and reasoning, action planning, hypothesis tracking, evidence-hypothesis relation prediction, and clinical statement typing. Evaluation of 11 general-purpose and medical LLMs shows that strong diagnostic performance coexists with incomplete coverage of documented expert reasoning. The leading model achieves 76.7% top-5 diagnostic accuracy but only 33.3% reasoning comprehensiveness. Supervised fine-tuning improves longitudinal reasoning capabilities and transfers beyond ClinTRACE, with diagnostic gains on two external benchmarks. Together, ClinTRACE supports evaluation and learning of clinical reasoning as evidence accumulates, beyond final-answer accuracy. We will release the public dataset and retain the non-public cases as a protected evaluation set.

Primary area:
datasets and benchmarks
Keywords:
Clinical Reasoning Benchmark, Large Language Models, Longitudinal Clinical Reasoning, Clinical Decision Making

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_QRPrOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
3(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces CLINTRACE, a longitudinal clinical reasoning benchmark constructed from live physician discussions, where clinical evidence is progressively revealed. The benchmark provides time-aligned expert reasoning trajectories and five tasks covering diagnosis, hypothesis tracking, action planning, and evidence interpretation. Evaluating 11 LLMs, the paper identifies a gap between diagnostic accuracy and expert-reasoning coverage, with performance varying across tasks and prompting strategies. SFT on expert trajectories improves certain reasoning metrics, but the gains are not consistent across all tasks. The paper also reports preliminary improvements on external diagnostic benchmarks.

Critical strengths

Novel data source.

By preserving the information available at each stage, it provides a valuable resource for studying how clinical reasoning evolves as evidence accumulates. This helps reduce the hindsight bias of benchmarks that reconstruct rationales from case reports.

Comprehensive evaluation.

The evaluation shows that models performing well on diagnosis do not always perform equally well on hypothesis tracking or evidence interpretation, supporting the value of evaluating these abilities separately.

Useful comparison of prompting settings.

Providing earlier expert reasoning improves reasoning coverage for most models, but does not consistently improve diagnostic accuracy. This helps distinguish the effects of expert traces on reasoning and diagnosis.

Critical weaknesses

Metrics may overlook precision.

Most process metrics focus on recovering reference reasoning without penalizing additional hypotheses or actions. This may favor models that generate more predictions, even when some are unsupported or unnecessary. The trade-off between reasoning coverage and prediction precision is not evaluated.

Limited evaluation of longitudinal consistency.

Hypothesis tracking uses the experts' previously active hypotheses rather than the model's own earlier predictions. This evaluates updates to a given hypothesis set, but does not test whether models can consistently maintain and revise their own hypotheses across a case.

Unclear handling of revealed diagnoses.

The dataset records whether the final diagnosis has been revealed, but it is unclear whether the diagnostic evaluation excludes such timesteps. Given the very high final stage diagnostic accuracy reported for several models, this distinction should be clarified.

Questions for authors

To what extent do the recall-based metrics reflect clinical reasoning quality rather than coverage of expert reasoning? How would accounting for unsupported hypotheses or unnecessary actions affect the evaluation?

SFT improves reasoning coverage but reduces performance on other tasks, particularly evidence hypothesis relation prediction. What might explain this trade-off, and does it suggest a limitation of learning from expert reasoning trajectories?

7 comments
AuthorsOpenReview ↗

Response to Reviewer QRPr (part 1)

We thank the reviewer for recognizing the novelty of our data source, the breadth of our evaluation, and the value of the prompting comparisons! We appreciate the thoughtful feedback and address the concerns and questions below.

W1 / Q1: Precision, valid alternatives, and exhaustive predictions

We agree that recall-based metrics measure how much documented expert reasoning a model recovers. They may favor longer outputs. We added precision for reasoning traces, action planning, added hypotheses, and ruled-out hypotheses. Precision evaluates the proportion of predicted statements or decisions that matched the clinician reference.

The precision results show the reference alignment among predictions that penalize the lengthy model outputs. Claude achieved higher reasoning trace recall than GPT (76.9% vs. 58.2%), but GPT achieved slightly higher precision. While Claude performed the best in added hypothesis recall, Grok and Kimi achieved the highest added hypothesis precision at 45.2% and 45.0%. Notably, Kimi achieved the highest rule-out hypothesis recall at 68.1%, but its precision was only 16.0%, compared with GPT’s 60.4% recall and 25.0% precision. We conclude that higher coverage did not imply more precise hypothesis removal. Additional predictions that fail to match the reference can lower precision without reducing recall.

We have updated Table 3 and Figure 3 given the newly added precision scores, as well as corresponding evaluation protocol and results analysis.

W2: Maintaining and revising the model’s own hypotheses

The current hypothesis-tracking task supplies the experts’ previously active hypotheses, giving models a common starting state and isolating their updates after new evidence. To evaluate longitudinal consistency, we evaluate each model’s proposed hypothesis list across timesteps and compare it with the corresponding expert reference hypotheses using hypothesis alignment F1, rule-in alignment, and rule-out alignment (under the same setting as used in Table 3 of the manuscript). The results below show limited hypotheses overlapping across all models, with hypothesis alignment F1 ranging from 25.1% to 39.8%, rule-in alignment from 6.8% to 22.3%, and rule-out alignment from 5.0% to 14.2%.

ModelHypothesis alignment F1 (%)Rule-in alignment (%)Rule-out alignment (%)
GPT37.2 ± 2.420.9 ± 2.914.2 ± 3.6
Claude39.8 ± 2.922.3 ± 2.011.1 ± 3.4
Grok29.5 ± 2.115.4 ± 1.78.3 ± 0.9
Kimi29.7 ± 3.812.0 ± 1.99.8 ± 1.3
DeepSeek28.4 ± 1.611.7 ± 2.27.5 ± 1.1
Llama26.7 ± 1.36.8 ± 0.95.9 ± 2.2
Qwen9B30.3 ± 2.710.8 ± 1.35.9 ± 2.9
Qwen27B32.4 ± 3.713.3 ± 3.07.0 ± 0.8
Lingshu27.4 ± 3.69.3 ± 2.15.6 ± 2.6
Huatuo25.1 ± 3.212.1 ± 2.95.0 ± 0.9
MedGemma28.6 ± 2.711.6 ± 1.86.3 ± 1.6

Table 1: Model-proposed hypothesis alignment with expert reference hypotheses. Scores (%) are means across three seeds evaluation, with standard deviations.

Overall, compared with Table 3 of the manuscript, where models achieve higher recall when updating an expert hypothesis list, these results show that models struggle in proposing an expert-aligned hypothesis list when they must rely on their own earlier predictions.

W3: Handling stages that reveal the final diagnosis

In Figure 14, we report timestep-wise performance in progressive diagnostic reasoning tasks. It shows that most models’ performance on all metrics increase with time evolving. Only progressive diagnostic reasoning task evaluation involves the final timestamp given the hypothesis tracking and action planning are not valid at the final timestep. Because of the final timestep uniqueness, we also reported final timestep results in Table 12 and 13 to show with all available case information, what the performance looks like.

To evaluate performance before explicit diagnosis disclosure, we reaggregate Tasks 1–3 (i.e., progressive diagnostic reasoning, action planning, and hypothesis tracking) for all 11 models under the same setting in Table 3, keeping only timesteps before the first case_include_final_dx=True flag and excluding that timestep and all subsequent timesteps. Compared with the all-timestep results, GPT’s top-1 diagnostic accuracy is 14.6 percentage points (pp) lower compared with not excluding earlier diagnosis disclosure, while Kimi’s top-5 accuracy is 9.1 pp lower. Claude remains the highest LLJ-based diagnostic accuracy, with the highest reasoning trace recall (66.5%), action recall (64.9%), and added hypothesis recall (32.5%). Claude still leads in diagnostic and reasoning quality criteria before diagnostic disclosure. We added this analysis in the revised Appendix B.5 and Table 15 highlighted in orange color.

AuthorsOpenReview ↗

Response to Reviewer QRPr (part 2)

Q2: SFT gains and declines across tasks

Our SFT setting is a challenging multi-task SFT problem, where a model is trained across five tasks with different output structures and evaluation objectives. The existing results support task-dependent benefits: DDx accuracy and reasoning recall and comprehensiveness all get improved for both fine-tuned models, while evidence–hypothesis relation macro-F1 decreases for both. Our SFT experiments are intended as a demonstration that expert reasoning trajectories provide useful training supervision instead of an optimized training approach expected to improve all five tasks at the same time. The observed improvements in reasoning coverage, hypothesis tracking, and diagnostic metrics support this potential. A possible explanation of the declines in the relation prediction task is the heterogeneous tasks share next-token supervision, while relation prediction is evaluated using macro-F1. The improved alignment of expert reasoning does not ensure improvement in every task metric. Thus, the task-specific adaptation and multi-objective optimization strategies can be further applied.

I appreciate the author’s detailed responses:

Additional analyses about precision metric results and handling stages that reveal the final diagnosis has helped resolve my previous questions.

These are my follow-up concerns/questions

  • Regarding the experiments in the response to W2, it is still unclear whether the model’s hypotheses from the previous timestep are actually passed to the next timestep.

  • The additional metrics (e.g, hypothesis alignment F1 and rule-in alignment, and rule-out alignment) measures alignment with the experts , but how they the directly measure  the model’s longitudinal consistency is still unclear to me.

  • Regarding the responses to SFT results, I understand the possible explanation about multi-task performance but it would be helpful if it was supported by additional analysis.

For now, I increase the informativeness score to 3 considered the information provided in this rebuttal.

Reviewer_TkTJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ClinTRACE, a benchmark for clinical reasoning as new evidence becomes available. It uses live CPSolvers discussions to collect expert reasoning, diagnostic hypotheses, and proposed actions at different stages of a case. The dataset contains 1,464 public cases and 184 protected evaluation cases, with five tasks covering diagnosis and intermediate reasoning. Experiments on 11 LLMs show that models can predict diagnoses more successfully than they can recover the documented expert reasoning. The paper also studies supervised fine-tuning, which improves some reasoning metrics and shows preliminary gains on two external diagnostic benchmarks.

Critical strengths
  1. The live discussions provide a substantial collection of expert reasoning as evidence arrives, with a protected evaluation set that helps reduce exposure to public cases.
  2. The physician-designed schema and the 48-case audit provide useful checks on the quality of the extracted annotations.
  3. The experiments cover 11 models, five tasks, several prompting settings, and fine-tuning, giving a broad picture of model performance on the benchmark.
Critical weaknesses
  1. The recall-based metrics can reward exhaustive lists and penalize valid alternative reasoning, without evidence that the resulting scores agree with independent clinical judgments.
  2. The related-work comparison misses the earlier CPSolvers staged-diagnosis study and the sequential evaluation in ER-Reason v3, making the added contribution harder to assess.
  3. The paper does not clearly explain how later findings and explicit final diagnoses are kept out of earlier annotations and diagnostic evaluation inputs.
Questions for authors
  1. Do coverage scores agree with independent physicians' assessments of the clinical quality of model outputs?
  2. How are valid alternative reasoning, unsupported hypotheses, and unnecessary actions handled in the evaluation?
  3. What does ClinTRACE add beyond the earlier CPSolvers study and ER-Reason v3, and do any cases overlap with the earlier CPSolvers study?
  4. Does the extraction model see the full transcript, and how do you prevent later findings from influencing earlier annotations?
  5. How is case_include_final_dx used, and what are the diagnostic results when stages explicitly revealing the diagnosis are excluded?
3 comments
AuthorsOpenReview ↗

Response to Reviewer TkTJ (part 1)

We thank the reviewer for recognizing the value of the staged expert discussions, the physician-designed schema and annotation audit, and the breadth of our experiments! We address the evaluation limitations, prior work comparisons, and temporal-information safeguards below.

W1/Q1/Q2: Recall limitations, clinical quality validation, and legitimate alternatives

We agree that recall alone can reward exhaustive generation. We therefore add precision metrics for progressive diagnostic reasoning, action planning, and hypothesis tracking, and update the corresponding tables and figures in the latest manuscript.

Our evaluation targets recovery of documented expert reasoning under sequential information. ClinTRACE preserves contributions from multiple expert discussants, providing a diverse reference than a single physician’s reasoning. As a result, clinically valid alternatives that are absent from the documented discussion may receive less credit. We acknowledge this limitation and interpret our coverage and precision scores as measures of reference alignment rather than a comprehensive assessment of clinical reasoning quality.

Regarding independent clinical judgments, the current study does not establish agreement between coverage scores and independent assessments of overall clinical quality. All evaluation prompts are co-designed with physicians, and the 44-case calibration compares automated reference-based scoring with one physician’s judgments. Table 17 reports 88.0–90.4% agreement for reasoning comprehensiveness and 75.4–76.9% for reasoning-trace recall. These provide preliminary, metric-specific calibration.

Assessing clinical quality and alternative reasoning under incomplete evidence requires a separate evaluation framework which would involve evaluators’ personal evaluation standards for physicians or LLM judges, as well as substantial effort to reach agreement across evaluators. These judgments are inherently subjective and can be hard to reconcile in our setting. Prior studies showed these evaluation challenges. ER-Reason reported 52.6% exact agreement between physicians on ordinal diagnostic judgments [1]. Gershon et al. found stronger physician–LLM agreement for text-comparison criteria, such as faithfulness and completeness, than for interpretive criteria, such as reasoning and safety [2]. Byun et al. similarly reported substantial disagreement among automated evaluators of evidence–conclusion coherence, with judge–physician agreement varying across evaluators [3]. Although these studies evaluated different tasks, they showed why physicians or LLM judges should not be treated as a definitive standard for evaluating valid reasoning paths.

W2 / Q3: Contribution beyond earlier CPSolvers work and ER-Reason v3; case overlap

Dinc et al. [4] evaluated LLM diagnostic accuracy under staged information with 104 CPSolvers cases and 60 common clinical cases. However, their evaluation focused on diagnostic accuracy instead of the intermediate reasoning processes of experts. In contrast, our dataset contributes as follows: (1) a substantially larger dataset involving 1,464 public cases with 84,044 reasoning steps and 184 protected cases; (2) structured longitudinal reasoning trajectories showing how multiple experts interpret evidence, update hypotheses, and propose actions before subsequent evidence is revealed, with extraction fidelity validated by CPSolver-affiliated physicians; and (3) five evaluation tasks assessing model’s reasoning processes compared with human beyond diagnostic accuracy. We also evaluated 11 general-purpose and medical LLMs, including open-weight models, but Dinc et al. [4] focused on proprietary models only.

ER-Reason v3 [1] evaluated sequential clinical reasoning using Script Concordance Tests (SCTs). Its 194 SCT cases are collected from the emergency department, with 3–5 differential diagnoses (DDx) authored by a single emergency physician, and the physician-written reasoning is based on the provided DDx at each timestep. In contrast, ClinTRACE includes more diverse cases from the CPSolvers platform and collects clinical reasoning trajectories produced by multiple clinical experts as case information is revealed sequentially, without access to future evidence, DDx, or the final diagnosis. This design provides naturally diverse multi-expert reasoning trajectories across instead of single expert reasoning.

Overall, our ClinTRACE dataset provides unique multi-experts staged reasoning trajectories and evaluation setup that are not provided in the prior studies. We have updated Table 1 in the revised manuscript to include Dinc et al. [4].

AuthorsOpenReview ↗

Response to Reviewer TkTJ (part 2)

W3/Q4: Full-transcript extraction and temporal leakage

Yes, the extraction model receives the full transcript, but we distinguish the information available to the extractor from the information available to the evaluated models.

The annotation target is the content expressed at each discussion stage, including the case information presented, the evidence interpretations and hypotheses stated by discussants, and the actions requested at that point. The extractor is intended to structure these recorded statements rather than reconstruct earlier reasoning from the eventual diagnosis. Our physician audit assessed source faithfulness, source-unsupported content, and temporal organization in the sampled cases. The reported 99.6% faithfulness of assessed reasoning steps supports correspondence with the discussion.

For diagnostic evaluation, cumulative-context prompting supplies only case presentations through the evaluated timestep. Future presentation segments and expert reasoning traces are excluded in this setting. For all prompting settings, the final diagnosis is explicitly excluded from the case presentation. The final diagnosis label and disclosure flag are not supplied as additional diagnostic inputs.

Q5: Diagnostic performance before diagnosis disclosure

case_include_final_dx is a timestep-level tag indicating whether the case presentation reveals the final diagnosis. If so, the final diagnosis will be removed from the case to prevent leakage. The flag itself is not included in the model prompt. To evaluate performance before explicit diagnosis disclosure, we reaggregate progressive diagnostic reasoning, action planning, and hypothesis tracking tasks for all 11 models under the same setting in Table 3, keeping only timesteps before the first case_include_final_dx=True flag and excluding that timestep and all subsequent timesteps. Compared with the all-timestep results, GPT’s top-1 diagnostic accuracy is 14.6 percentage points (pp) lower compared with not excluding earlier diagnosis disclosure, while Kimi’s top-5 accuracy is 9.1 pp lower. Claude remains the highest LLJ-based diagnostic accuracy, with the highest reasoning trace recall (66.5%), action recall (64.9%), and added hypothesis recall (32.5%). Claude still leads in diagnostic and reasoning quality criteria before diagnostic disclosure. We added this analysis in the revised Appendix B.5 and Table 15 highlighted in orange color.


References

[1] Mehandru, N., et al. (2025). ER-Reason: A benchmark dataset for LLM clinical reasoning in the emergency room. arXiv:2505.22919.

[2] Gershon, R., et al. (2026). Large language models as judges for clinical generative AI evaluation. npj Digital Medicine.

[3] Byun, H., et al. (2026). Multi-axial analysis of clinical reasoning in large language models. Journal of Medical Systems.

[4] Dinc, M. T., et al. (2025). Comparative analysis of large language models in clinical diagnosis. JAMIA Open.

AuthorsOpenReview ↗

Gentle Reminder

Dear reviewer TkTJ,

As the discussion period will end soon, we wanted to follow up and see whether our response addressed the points you raised. We would be glad to clarify or expand on anything that still feels unclear. We sincerely appreciate your time and thoughtful feedback, which have helped make the paper stronger and more rigorous!

Reviewer_vpecOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ClinTRACE, a benchmark for evaluating clinical reasoning as evidence accumulates. It organizes clinical discussions into time-aligned trajectories linking available evidence to reasoning steps, diagnostic hypothesis updates, and subsequent actions. Its construction methodology, comparisons with existing benchmarks, and broad experimental evaluation establish a useful temporal perspective beyond final-diagnosis accuracy.

Critical strengths
  1. The benchmark construction and comparisons with prior benchmarks clearly motivate temporally evaluating the clinical reasoning process. Linking evidence, hypotheses, reasoning, and actions offers a useful evaluation target beyond final diagnosis.

  2. The experiments provide a useful capability profile rather than a single diagnosis score. Eleven models, multiple prompting conditions, the protected-source comparison, and SFT analysis show that diagnosis, hypothesis updates, and expert-reasoning coverage behave differently. The text appropriately acknowledges task-dependent SFT gains and preliminary external transfer.

Critical weaknesses

1.My main reservations concern the clinical grounding of the evaluation framework and the scope of expert validation. The relationship between the three stage-level elements introduced in the introduction and physicians’ practical experience needs further explanation. Although the paper reports a physician audit of sampled cases, the coverage of that audit across the benchmark and the treatment of reasonable differences among physicians’ reasoning paths remain unclear to me. These are requests for clarification rather than established methodological errors.

  1. The reliability of the central reasoning evaluator is only partially established. The 44-case calibration uses one physician, and reasoning-trace agreement is 75.4-76.9%, substantially below the diagnosis and action agreements (Section 4.2; Table 16). Agreement on extraction is a separate validation question and cannot resolve this evaluator uncertainty. Since the main diagnosis-versus-reasoning analysis and several SFT conclusions depend on these judgments, an independent check of reasoning-score robustness across raters and model outputs is important for interpreting the benchmark as a measurement instrument, rather than treating its automated ordering as definitive.
Questions for authors
  1. Could the authors explain how the three stage-level elements and evaluation criteria relate to the physician co-designed taxonomy, and what the 48-case audit validates beyond extraction faithfulness and labels? Please clarify the scope of sampled review, what remains unaudited, and how clinically reasonable alternatives to the documented expert path are treated. This asks for the scope and interpretation of the current validation, not physician review of every case.

  2. Beyond checking extraction faithfulness and label correctness, have physicians validated the clinical appropriateness of the benchmark and its evaluation criteria? Since a clinical problem may admit multiple reasonable reasoning paths, how does the evaluation accommodate legitimate disagreements among experts?

  3. Do the main diagnostic and reasoning comparisons remain when restricted to timesteps before the final diagnosis is explicitly revealed, using the dataset's case_include_final_dx flag? This would clarify how much of the endpoint contrast reflects answering after disclosure rather than inference under partial information.

4 comments
AuthorsOpenReview ↗

Response to Reviewer vpec (part 1)

We thank the reviewer for recognizing the importance of our work and for the thoughtful suggestions! We address the concerns and questions below and welcome further discussion.

W1/Q1a: Clinical grounding of the framework and scope of the physician audit

The three stage-level elements describe repeated components of clinical reasoning rather than three fixed phases. Physicians evaluate the information currently available, make judgments before future findings are known, and revise their hypotheses and actions as evidence accumulates. This formulation is grounded in prior frameworks of real-world clinical reasoning and medical education [1–4]. We represent these elements using our physician co-designed schema. Case presentations and observed evidence capture the information available at each stage. Reasoning-step types and evidence–hypothesis relations capture how physicians interpret that information. Active hypotheses and requested actions capture changes in diagnoses and clinical decisions. The tasks and evaluation criteria assess these corresponding components.

Physicians participate in the design of the taxonomy, task definitions, and evaluation prompts. Their involvement helps ensure that the evaluation framework reflects clinically meaningful reasoning. The extraction audit addresses a different question, which is whether the structured annotations faithfully represent the original discussions. Specifically, 12 physicians reviewed 48 randomly sampled VMR cases, with six cases independently reviewed by two physicians and the remaining 42 reviewed by one physician. The review covered the schema fields represented in these cases, including evidence, reasoning statements, relations, actions, and hypotheses. The audit is a sample-based check of extraction faithfulness and annotation correctness.

Q1b/Q2: Clinical appropriateness and legitimate alternative reasoning

We agree that clinical appropriateness and agreement with documented expert reasoning are different evaluation targets. ClinTRACE preserves reasoning from multiple discussants, providing a more diverse reasoning reference than a single physician’s reasoning paths. However, these references are still not complete. A clinically reasonable alternative that did not appear in the reference may receive less credit in our current reference-based evaluation. Similarly, matching the expert reasoning paths does not mean that the reasoning is clinically appropriate. We therefore interpret our evaluation framework as recovery of documented expert reasoning paths under sequential information, rather than a comprehensive assessment of clinical reasoning appropriateness and alternatives. We added precision metrics to our revised manuscript to complement recall scores by measuring the proportion of generated content that matches the reference.

Evaluating alternative reasoning paths under incomplete evidence requires a separate framework with explicit criteria and calibration across physicians. Prior studies illustrated these measurement challenges. ER-Reason reported 52.6% exact agreement between physicians on ordinal diagnostic judgments [5]. Gershon et al. found stronger physician–LLM agreement for text-comparison criteria, such as faithfulness and completeness, than for interpretive criteria, such as reasoning and safety [6]. Byun et al. similarly reported substantial disagreement among automated evaluators of evidence–conclusion coherence, with judge–physician agreement varying across evaluators [7]. Although these studies evaluated different tasks, they showed why physicians or LLM judges should not be treated as a definitive standard for appropriate reasoning paths.

Accordingly, our study focuses on a defined reference-alignment target and does not add unrestricted clinical appropriateness ratings for generated reasoning paths. Such ratings would measure a different target and would involve evaluators’ personal evaluation standards for physicians or LLM judges, as well as substantial effort to reach agreement across evaluators. These judgments are inherently subjective and can be hard to reconcile in our sequentially incomplete information setting. We distinguish physician involvement in benchmark design, validation of extracted annotations, and validation of model-output scoring. The reliability of the reference-coverage evaluator is addressed separately in our response to W2.

AuthorsOpenReview ↗

Response to Reviewer vpec (part 2)

W2: Independent validation of the reasoning evaluator

We acknowledge the reliability limitations of using LLM judges for reasoning coverage evaluation. As shown in Table 17, reasoning coverage shows lower judge–physician agreement than diagnostic accuracy and action recall, while GPT-5.6 Luna agrees more with the physician than Claude on reasoning coverage, with higher coverage scores closer to the physician’s evaluation. The reported medium agreement rates of reasoning trace recall are lower than other metrics because the physician tends to strictly label traces as aligned compared to humans and model even under the same evaluation instruction. The presented reasoning trajectories may involve implicit rationale that the physician captures, but the judge models fail to do so. These results show that LLM judges remain imperfect. To reduce reliance on LLM judges, we also report non-LLM-as-a-judge metrics that provide more objective measurements based on explicit reference matching (i.e., recall and precision scores). We report both types of metrics as complementary evaluations rather than treating either as definitive.

Q3: Diagnostic and reasoning comparisons before diagnosis disclosure

In Figure 14 of our manuscript, we report timestep-wise performance on diagnosis and reasoning. It shows how diagnostic and reasoning performance evolve throughout the case presentation. The average timestep per case is around 6. For final-timestep performance, we report both VMR and RLR in Tables 12 and 13.

​To evaluate performance before explicit diagnosis disclosure, we reaggregate progressive diagnostic reasoning, action planning, and hypothesis tracking tasks for all 11 models under the same setting in Table 3, keeping only timesteps before the first case_include_final_dx=True flag and excluding that timestep and all subsequent timesteps. Compared with the all-timestep results, GPT’s top-1 diagnostic accuracy is 14.6 percentage points (pp) lower than when not excluding earlier diagnosis disclosure, while Kimi’s top-5 accuracy is 9.1 pp lower. Claude remains the highest LLJ-based diagnostic accuracy, with the highest reasoning trace recall (66.5%), action recall (64.9%), and added hypothesis recall (32.5%). Claude still leads in diagnostic and reasoning quality criteria before diagnostic disclosure. We added this analysis in the revised Appendix B.5, and Table 15 highlighted in orange color.


References

[1] Sox et al. (2024). Medical Decision Making. John Wiley & Sons.

[2] Kassirer, J. P. (2010). Teaching clinical reasoning: case-based and coached. Academic medicine.

[3] Gamborg et al. (2023). Clinical decision-making and adaptive expertise in residency: a think-aloud study. BMC medical education.

[4] Pelaccia et al. (2014). How and when do expert emergency physicians generate and evaluate diagnostic hypotheses? A qualitative study using head-mounted video cued-recall interviews. Annals of Emergency Medicine.

[5] Mehandru, N., et al. (2025). ER-Reason: A benchmark dataset for LLM clinical reasoning in the emergency room. arXiv:2505.22919.

[6] Gershon, R., et al. (2026). Large language models as judges for clinical generative AI evaluation. npj Digital Medicine.

[7] Byun, H., et al. (2026). Multi-axial analysis of clinical reasoning in large language models. Journal of Medical Systems.

Thank you for the clarifications and the pre-disclosure analysis.These substantially address my concerns about the evaluation scope and diagnosis disclosure. I will maintain my rating of 3 (Weak acceptance). One point remains: do the main model comparisons and SFT conclusions remain consistent across the LLM judges and reference-matching metrics? A brief summary using the existing results would help clarify evaluator robustness.

AuthorsOpenReview ↗

Follow-up Response to Reviewer vpec

We thank the reviewer for finding our clarifications and additional analyses helpful! We provide a brief summary of the existing results with newly added metrics and analysis below:

Follow-up comment: A brief summary using the existing results. ​

Yes, the model comparisons remain consistent across evaluators for both the main results and the SFT results. In the 44-case validation audit, GPT-5.6 Luna and Claude Sonnet 5 agreed on 100% of diagnosis and action-recall decisions and on 86.2–95.2% of reasoning decisions (Table 17). Both judges also showed high agreement with the physician on DDx, action recall and precision, and reasoning comprehensiveness. Across the 11 models in Table 3, the LLM judge and reference-matching metrics produced broadly similar rankings on most tasks, although the leading model may vary across specific metrics, such as added hypothesis precision and ruled-out hypothesis recall. Claude remained the strongest model overall in progressive diagnostic reasoning, action planning, and hypothesis tracking across both metric types on the VMR and RLR datasets (Tables 3 and 14).

The SFT results also showed improvements across complementary reasoning metrics (Table 16). For both Llama and Qwen9B, the reasoning performance of SFT improved under the LLM judge metrics and reference-matching metrics. However, Top-1 diagnostic accuracy did not improve under either evaluation type. Llama improved in both action planning and hypothesis tracking after SFT, while Qwen9B showed declines in action recall and ruled-out hypothesis performance. These results indicated both the potential of expert-trajectory supervision to improve the model’s clinical reasoning and diagnosis capabilities and the challenge of maintaining consistent gains across multiple prediction targets under multi-task SFT.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary

We sincerely thank all three reviewers for their thoughtful and constructive feedback. We are grateful that Reviewer QRPr increased the rating from 2 to 3 following our rebuttal and additional analyses. Reviewer vpec maintained the rating (3), noting that our clarifications and pre-disclosure analysis substantially addressed their concerns.


Consensus Among Reviewers

  • Novel longitudinal benchmark and source: All three reviewers (vpec, TkTJ, QRPr) recognized the value of our benchmark by preserving real-world, multi-expert clinical reasoning sequentially beyond final diagnosis evaluation.

  • Comprehensive evaluation: All three reviewers highlighted the broad evaluation across 11 LLMs, five tasks, multiple prompting settings, and supervised fine-tuning.

  • Clinical grounding and data quality: Reviewers vpec and TkTJ recognized our physician-designed taxonomy, structured reasoning trajectories, and physician extraction audit.

  • Additional benchmark value: Reviewer TkTJ highlighted our protected evaluation set. Reviewer QRPr recognized the useful comparisons between prompting strategies and the distinction between diagnostic accuracy and reasoning coverage.


Summary of Revisions and Clarifications

  • Precision metrics and evaluation scope (vpec W1/Q1/Q2; TkTJ W1/Q1/Q2; QRPr W1/Q1): We added precision metrics for reasoning traces, action planning, and hypothesis updates. Results showed recall–precision trade-offs of assessed LLMs. We clarified that our metrics assess alignment with documented expert reasoning, not unrestricted clinical appropriateness or quality.

  • Clinical grounding and physician validation (vpec W1/Q1/Q2): We clarified the three stage-level reasoning elements that are designed given prior clinical reasoning literature. We clarified the scope of the 48-case physician audit and distinguished extraction faithfulness, reference alignment, and clinical appropriateness.

  • LLM judge reliability (vpec W2): We clarified the reason for medium reasoning trace physician-judge agreements and the limitations of automated reasoning evaluation. We emphasized complementary reference-matching metrics alongside LLM judge evaluation rather than treating either as definitive.

  • Diagnosis disclosure and temporal leakage (vpec Q3; TkTJ W3/Q4–Q5; QRPr W3): We clarified that future case presentations and expert reasoning are excluded from diagnostic prompts. We added pre-disclosure evaluations across all 11 models, excluding timesteps at and after explicit diagnosis revelation.

  • Model’s hypothesis tracking (QRPr W2): We added hypothesis alignment F1, rule-in alignment, and rule-out alignment of model-predicted hypotheses across timesteps, showing the difficulty of models in maintaining expert-aligned hypotheses across sequential evidence updates.

  • Comparison with existing benchmarks (TkTJ W2/Q3): We explicitly compared our benchmark with prior work. Dinc et al.’s staged CPSolvers dataset included only 104 cases and evaluated diagnostic accuracy without modeling intermediate reasoning processes. ER-Reason v3 collected only 194 SCT cases, with clinical reasoning trajectories authored by a single physician based on disclosed differential diagnoses.

  • SFT performance trade-offs (QRPr Q2): We explained the limitation of multi-task SFT in improving performance of all our five tasks. We also clarified that our SFT goal is to show that expert reasoning trajectories provided useful training supervision in improving diagnostic performance.


Follow-up Comments

  • Cross-evaluator robustness (vpec Follow-up): We showed broadly consistent model comparisons and SFT conclusions across LLJ and reference-matching metrics. Claude remained strongest overall across the main tasks, while SFT gains and limitations were consistent across complementary metrics.

  • Model-generated hypothesis propagation (QRPr Follow-up Q1): We clarified that the model's previously predicted active hypotheses are passed into the next timestep, while keeping the same hypothesis-tracking experiment design.

  • Direct longitudinal consistency analysis (QRPr Follow-up Q2): We provided additional analysis of hypothesis addition and removal rates across sequential model predictions. Models removed 21.7–39.7% of hypotheses that experts retained and 43.1–67.6% of hypotheses that experts also removed.

  • Additional SFT analysis (QRPr Follow-up Q3): We provided SFT task output structures, validation loss changes, and label distributions. Longer explanations can dominate shared token-level training objectives, while 76.8% of relation labels are "supporting," potentially favoring the majority class despite macro-F1 evaluation.


We have revised the manuscript to include precision metrics, pre-disclosure diagnosis evaluations, and expanded benchmark comparisons. We thank the area chair and reviewers again for their time and valuable feedback!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.