- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 3(+1)
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces CLINTRACE, a longitudinal clinical reasoning benchmark constructed from live physician discussions, where clinical evidence is progressively revealed. The benchmark provides time-aligned expert reasoning trajectories and five tasks covering diagnosis, hypothesis tracking, action planning, and evidence interpretation. Evaluating 11 LLMs, the paper identifies a gap between diagnostic accuracy and expert-reasoning coverage, with performance varying across tasks and prompting strategies. SFT on expert trajectories improves certain reasoning metrics, but the gains are not consistent across all tasks. The paper also reports preliminary improvements on external diagnostic benchmarks.
Critical strengths
Novel data source.
By preserving the information available at each stage, it provides a valuable resource for studying how clinical reasoning evolves as evidence accumulates. This helps reduce the hindsight bias of benchmarks that reconstruct rationales from case reports.
Comprehensive evaluation.
The evaluation shows that models performing well on diagnosis do not always perform equally well on hypothesis tracking or evidence interpretation, supporting the value of evaluating these abilities separately.
Useful comparison of prompting settings.
Providing earlier expert reasoning improves reasoning coverage for most models, but does not consistently improve diagnostic accuracy. This helps distinguish the effects of expert traces on reasoning and diagnosis.
Critical weaknesses
Metrics may overlook precision.
Most process metrics focus on recovering reference reasoning without penalizing additional hypotheses or actions. This may favor models that generate more predictions, even when some are unsupported or unnecessary. The trade-off between reasoning coverage and prediction precision is not evaluated.
Limited evaluation of longitudinal consistency.
Hypothesis tracking uses the experts' previously active hypotheses rather than the model's own earlier predictions. This evaluates updates to a given hypothesis set, but does not test whether models can consistently maintain and revise their own hypotheses across a case.
Unclear handling of revealed diagnoses.
The dataset records whether the final diagnosis has been revealed, but it is unclear whether the diagnostic evaluation excludes such timesteps. Given the very high final stage diagnostic accuracy reported for several models, this distinction should be clarified.
Questions for authors
To what extent do the recall-based metrics reflect clinical reasoning quality rather than coverage of expert reasoning? How would accounting for unsupported hypotheses or unnecessary actions affect the evaluation?
SFT improves reasoning coverage but reduces performance on other tasks, particularly evidence hypothesis relation prediction. What might explain this trade-off, and does it suggest a limitation of learning from expert reasoning trajectories?
7 comments
Response to Reviewer QRPr (part 2)
Q2: SFT gains and declines across tasks
Our SFT setting is a challenging multi-task SFT problem, where a model is trained across five tasks with different output structures and evaluation objectives. The existing results support task-dependent benefits: DDx accuracy and reasoning recall and comprehensiveness all get improved for both fine-tuned models, while evidence–hypothesis relation macro-F1 decreases for both. Our SFT experiments are intended as a demonstration that expert reasoning trajectories provide useful training supervision instead of an optimized training approach expected to improve all five tasks at the same time. The observed improvements in reasoning coverage, hypothesis tracking, and diagnostic metrics support this potential. A possible explanation of the declines in the relation prediction task is the heterogeneous tasks share next-token supervision, while relation prediction is evaluated using macro-F1. The improved alignment of expert reasoning does not ensure improvement in every task metric. Thus, the task-specific adaptation and multi-objective optimization strategies can be further applied.
I appreciate the author’s detailed responses:
Additional analyses about precision metric results and handling stages that reveal the final diagnosis has helped resolve my previous questions.
These are my follow-up concerns/questions
-
Regarding the experiments in the response to W2, it is still unclear whether the model’s hypotheses from the previous timestep are actually passed to the next timestep.
-
The additional metrics (e.g, hypothesis alignment F1 and rule-in alignment, and rule-out alignment) measures alignment with the experts , but how they the directly measure the model’s longitudinal consistency is still unclear to me.
-
Regarding the responses to SFT results, I understand the possible explanation about multi-task performance but it would be helpful if it was supported by additional analysis.
For now, I increase the informativeness score to 3 considered the information provided in this rebuttal.
Response to Reviewer QRPr (part 1)
We thank the reviewer for recognizing the novelty of our data source, the breadth of our evaluation, and the value of the prompting comparisons! We appreciate the thoughtful feedback and address the concerns and questions below.
We agree that recall-based metrics measure how much documented expert reasoning a model recovers. They may favor longer outputs. We added precision for reasoning traces, action planning, added hypotheses, and ruled-out hypotheses. Precision evaluates the proportion of predicted statements or decisions that matched the clinician reference.
The precision results show the reference alignment among predictions that penalize the lengthy model outputs. Claude achieved higher reasoning trace recall than GPT (76.9% vs. 58.2%), but GPT achieved slightly higher precision. While Claude performed the best in added hypothesis recall, Grok and Kimi achieved the highest added hypothesis precision at 45.2% and 45.0%. Notably, Kimi achieved the highest rule-out hypothesis recall at 68.1%, but its precision was only 16.0%, compared with GPT’s 60.4% recall and 25.0% precision. We conclude that higher coverage did not imply more precise hypothesis removal. Additional predictions that fail to match the reference can lower precision without reducing recall.
We have updated Table 3 and Figure 3 given the newly added precision scores, as well as corresponding evaluation protocol and results analysis.
The current hypothesis-tracking task supplies the experts’ previously active hypotheses, giving models a common starting state and isolating their updates after new evidence. To evaluate longitudinal consistency, we evaluate each model’s proposed hypothesis list across timesteps and compare it with the corresponding expert reference hypotheses using hypothesis alignment F1, rule-in alignment, and rule-out alignment (under the same setting as used in Table 3 of the manuscript). The results below show limited hypotheses overlapping across all models, with hypothesis alignment F1 ranging from 25.1% to 39.8%, rule-in alignment from 6.8% to 22.3%, and rule-out alignment from 5.0% to 14.2%.
Table 1: Model-proposed hypothesis alignment with expert reference hypotheses. Scores (%) are means across three seeds evaluation, with standard deviations.
Overall, compared with Table 3 of the manuscript, where models achieve higher recall when updating an expert hypothesis list, these results show that models struggle in proposing an expert-aligned hypothesis list when they must rely on their own earlier predictions.
In Figure 14, we report timestep-wise performance in progressive diagnostic reasoning tasks. It shows that most models’ performance on all metrics increase with time evolving. Only progressive diagnostic reasoning task evaluation involves the final timestamp given the hypothesis tracking and action planning are not valid at the final timestep. Because of the final timestep uniqueness, we also reported final timestep results in Table 12 and 13 to show with all available case information, what the performance looks like.
To evaluate performance before explicit diagnosis disclosure, we reaggregate Tasks 1–3 (i.e., progressive diagnostic reasoning, action planning, and hypothesis tracking) for all 11 models under the same setting in Table 3, keeping only timesteps before the first
case_include_final_dx=Trueflag and excluding that timestep and all subsequent timesteps. Compared with the all-timestep results, GPT’s top-1 diagnostic accuracy is 14.6 percentage points (pp) lower compared with not excluding earlier diagnosis disclosure, while Kimi’s top-5 accuracy is 9.1 pp lower. Claude remains the highest LLJ-based diagnostic accuracy, with the highest reasoning trace recall (66.5%), action recall (64.9%), and added hypothesis recall (32.5%). Claude still leads in diagnostic and reasoning quality criteria before diagnostic disclosure. We added this analysis in the revised Appendix B.5 and Table 15 highlighted in orange color.