- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose ClinSeekAgent, an agentic framework for clinical reasoning that retrieves evidence from raw EHR records and medical imaging tools. They introduce ClinSeek-Bench, combining existing EHR and multimodal benchmarks to compare automated evidence seeking against curated inputs. They show improvements with frontier LLMs and further fine-tune Qwen3.5-35B-A3B on GPT-5.6 Sol generated trajectories, achieving substantial improvements on AgentEHR-Bench.
Critical strengths
-
I think the motivation is good. Clinical agents should be able to actively retrieve evidence from EHRs and medical images rather than relying on manually curated information.
-
The experiments are quite extensive, covering different clinical tasks and LLM backbones. I like the full-record and single-pass retrieval ablations, which show the benefit of iterative evidence seeking. The multiple-run analysis also supports the reliability of the improvements.
-
The improvements from fine-tuning on agent trajectories are quite good (+16.1 F1 over the base model). The analysis of SQL queries and evidence-seeking behavior is also interesting, showing that the model learns to interact with the EHR tools more effectively.
Critical weaknesses
-
I am not fully convinced by the methodological novelty. The framework essentially gives an LLM access to EHR and imaging tools and allows iterative tool calling. While integrating these into a unified clinical framework is useful, I don't see much beyond a standard tool-using agent. There has already been work on EHR agents [1,2]. It would be useful to compare against existing agentic approaches under similar tool access. How do coding agents work in general for this? Why do the authors not compare to existing agentic frameworks? These 2 have been mentioned in the Introduction.
-
The inference-time improvements are relatively small, around 1.5-2.5 F1 points for the two main models. Are these gains significant? I see no statistical analysis.
-
What is the x-ray analysis model used? I notice the
image.chest_xray_classifierbut what classifier is used?
[1] Shi, Wenqi, et al. "Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.
[2] Liao, Yusheng, et al. "Agentehr: Advancing autonomous clinical decision-making via retrospective summarization." arXiv preprint arXiv:2601.13918 (2026).
Overall, I think the paper addresses a relevant problem and provides a good amount of experimentation. My main concerns are the limited methodological novelty and the lack of training ablations to isolate the contribution of evidence-seeking trajectories. I look forward to the discussion with the authors.
Questions for authors
Mentioned along with weaknesses.
5 comments
W3: Image-tool models and tool-removal ablation
The chest_xray_classifier uses TorchXRayVision DenseNet-121 with the densenet121-res224-all weights. The learned image tools use the following pretrained models:
| Tool | Model / checkpoint |
|---|---|
| Chest X-ray classification | TorchXRayVision DenseNet-121, densenet121-res224-all |
| Findings generation | IAMJB/chexpert-mimic-cxr-findings-baseline |
| Impression generation | IAMJB/chexpert-mimic-cxr-impression-baseline |
| Phrase grounding | microsoft/maira-2 |
| Anatomical segmentation | TorchXRayVision PSPNet |
MAIRA-2 was trained on the official MIMIC-CXR training split. For the classifier, report-generation checkpoints, and anatomical segmentation model, the released documentation does not provide exact training splits or image-level training lists sufficient to establish their precise overlap with the evaluation data.
To assess how much the results depend on these pretrained image tools, we performed an ablation with Claude Opus 5 on ClinSeek-Bench-MM, disabling the image tools above. Removing these image tools changes overall F1 by only 0.8 points, which is not statistically significant. ClinSeekAgent without image tools still significantly outperforms Curated Input.
We will add the checkpoint names and available training-data information to Appendix F, and include the tool-removal analysis in the revised manuscript.
W4: Training value of evidence-seeking trajectories
Thank you for highlighting “the lack of training ablations to isolate the contribution of evidence-seeking trajectories”. We address the contribution of evidence-seeking trajectories with the following two analyses.
1. Training ablation tests the effect of multi-round trajectories.
We add a controlled comparison between multi-round and single-round trajectory supervision. The single-shot baseline controls for general EHR task adaptation and tool-use proficiency. Its trajectories demonstrate how to formulate retrieval queries and answer from the returned records. The teacher issues all retrieval calls before seeing any results, so the student receives no demonstrations of using earlier evidence to decide what to query next.
The two students share the GPT-5.6 Sol teacher, source training questions, tools, answer format, and fine-tuning recipe. Each is evaluated in its corresponding retrieval mode. The table reports F1 on AgentEHR-Bench (500 questions), with ΔF1 = single-pass − multi-round and 95% paired-bootstrap confidence intervals (4,000 resamples).
| Task | Multi-round | Single-pass | ΔF1 |
|---|---|---|---|
| Overall | 35.6 | 32.8 | −2.8 [−4.6, −0.9] |
| Diagnoses | 41.7 | 35.0 | −6.7 [−10.5, −2.8] |
| Lab events | 40.4 | 38.4 | −2.0 [−4.2, +0.3] |
| Microbiology | 27.1 | 25.5 | −1.6 [−6.5, +3.0] |
| Procedures | 41.3 | 36.2 | −5.1 [−10.4, +0.0] |
| Transfers | 27.4 | 28.9 | +1.5 [−3.3, +6.1] |
The student trained on multi-round trajectories scores 2.8 higher overall F1, with the largest gain on diagnoses (+6.7). The gain over a baseline that already receives EHR task and tool-use supervision supports the additional value of multi-round evidence-seeking demonstrations.
2. Trajectory analysis checks whether training improves evidence acquisition.
We searched the base model's and trained student's tool outputs for the teacher's supporting facts in 288 label-level checks. We also measured how often gold labels appeared in candidate lookups and what fraction of the available EHR tables each model accessed.
| Evidence-acquisition measure | Base | Trained student | Teacher |
|---|---|---|---|
| Supporting facts retrieved (288 label-level checks) | 74% | 98% | — |
| Gold labels appearing in candidate lookup | 52% | 79% | 81% |
| Coverage of available EHR tables | 37% | 74% | 77% |
Coverage of the checked supporting facts rises from 74% to 98% after training. Candidate-label coverage rises from 52% to 79%, close to the teacher's 81%. These measurements show that the student acquires more of the evidence needed for the clinical predictions during tool interaction, connecting the training benefit to evidence-seeking behavior.
We will add both the training ablation and evidence-acquisition analysis to the revised manuscript.
Thank you again for your thoughtful review. We have provided detailed responses and additional agent comparisons and training ablations. As the discussion deadline approaches, we would greatly appreciate your feedback on whether these results address your concerns. If any issues remain, please let us know so we can respond in time.
Thank you for your thoughtful review and constructive suggestions. We appreciate your recognition of the importance of active clinical evidence seeking, the breadth of our experiments, and the gains from trajectory-based training. We address your concerns and questions point by point below.
W1: Methodological contribution and comparisons with existing agents
Contributions relative to prior work:
Beyond existing clinical agent work, our work offers three distinguishing contributions:
Harness comparison and analysis:
EHRAgent [1] generates executable code plans using EHR metadata, medical knowledge, and few-shot demonstrations, then revises them using execution feedback. AgentEHR’s RetroSum [2] periodically revisits the interaction history to generate summaries and, in its evolved version, retrieves strategies from an external experience bank. ClinSeekAgent lets the host model organize its retrieval and reasoning workflow without mandatory periodic summarization or a prescribed few-shot prompting stage.
We evaluated clinical adaptations of RetroSum (non-evolved) [2], ReSum, mini-SWE-agent, and EHRAgent [1]. All methods use GPT-5.6 Sol on the same Text (1,800) and MM (989) questions, with the same clinical tool permissions, prediction-time cutoffs, answer formats, and scoring.
The table reports overall F1 and ΔF1 = baseline − ClinSeekAgent, with 95% paired-bootstrap confidence intervals (4,000 resamples).
Our harness achieves the highest overall F1 among the evaluated harnesses on both benchmarks. We will clarify our contributions and include these comparisons in the revised manuscript.
[1] Shi, Wenqi, et al. "EHRAgent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records."
[2] Liao, Yusheng, et al. "AgentEHR: Advancing autonomous clinical decision-making via retrospective summarization."
W2: Statistical significance of the inference-time gains
Thank you for raising this point. Statistical significance is indicated in Table 1 and the paired-bootstrap procedure is described in Appendix H.1, but we recognize that these results could be presented more prominently. In the main paper, Table 1 marks statistically significant differences in bold, explicitly defined in its caption as differences whose paired-bootstrap 95% confidence intervals exclude zero. Appendix H.1 specifies the statistical procedure: paired bootstrap over questions with 4,000 resamples.
For clarity, we reproduce the overall gains below, where ΔF1 = ClinSeekAgent − Curated Input and brackets give the 95% confidence interval.
Section 3.2 (“The improvements are stable across multiple runs”) also reports that ClinSeekAgent outperforms Curated Input in all five GPT-5.6 Sol runs on both benchmarks, with an overall F1 standard deviation of at most 0.4 points within each mode. Appendix C, Table 6 reports run-averaged gains of +2.0 F1 [0.9, 3.1] on Text and +2.7 [1.5, 4.0] on MM, whose confidence intervals also exclude zero.