acceptodds
Under review as a conference paper at ICLR 2027

ClinSeekAgent: Automating Multi-modal Evidence Seeking for Agentic Clinical Reasoning

13 views

Abstract

Large language models (LLMs) and agentic systems have shown promise for clinical decision support, but existing works largely assume that evidence has already been curated and handed to the model. Real-world clinical workflows instead require agents to actively seek, iteratively plan, and synthesize multimodal evidence from heterogeneous sources. In this paper, we introduce **ClinSeekAgent**, an automated agentic framework for dynamic multimodal evidence seeking that shifts the paradigm from passive evidence consumption to active evidence acquisition. Given a clinical task and access to raw patient data, ClinSeekAgent queries longitudinal EHR records and invokes imaging analysis tools when relevant; refines its hypotheses as new information emerges; and integrates the collected evidence into grounded clinical decisions. We validate ClinSeekAgent in two roles: as an inference-time agent for frontier LLMs and as a training pipeline for open-source models. For inference-time validation, we construct **ClinSeek-Bench**, which covers 13 text-only EHR task categories and 6 multimodal EHR–chest X-ray task categories, and pairs each question with both a *Curated Input* of pre-selected evidence and the raw patient data for automated evidence seeking. Without any benchmark-specific curation, ClinSeekAgent significantly outperforms curated input for both Claude Opus 5 and GPT-5.6 Sol, improving overall F1 from 61.7 to 63.9 and from 57.8 to 59.3 on text-only EHR tasks, and from 66.5 to 68.8 and from 60.0 to 62.5 on multimodal tasks. The largest gains occur in text-based decision-making and multimodal phenotype prediction, and ablations show that they come from multi-round evidence seeking rather than from access to more of the patient record. For training-time validation, we fine-tune Qwen3.5-35B-A3B on ClinSeekAgent trajectories generated by GPT-5.6 Sol. The resulting **ClinSeek-35B-A3B** reaches 35.6 average F1 on AgentEHR-Bench (**+16.1** over its base model), outperforming all evaluated open-source models, including ones more than ten times larger, as well as Claude Opus 5. We will release our model, data, and code to facilitate future research.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Clinical Agent, EHR, Multimodal Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 2 reviews, above 85% of the others.
Reviewer_iEmHOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose ClinSeekAgent, an agentic framework for clinical reasoning that retrieves evidence from raw EHR records and medical imaging tools. They introduce ClinSeek-Bench, combining existing EHR and multimodal benchmarks to compare automated evidence seeking against curated inputs. They show improvements with frontier LLMs and further fine-tune Qwen3.5-35B-A3B on GPT-5.6 Sol generated trajectories, achieving substantial improvements on AgentEHR-Bench.

Critical strengths
  • I think the motivation is good. Clinical agents should be able to actively retrieve evidence from EHRs and medical images rather than relying on manually curated information.

  • The experiments are quite extensive, covering different clinical tasks and LLM backbones. I like the full-record and single-pass retrieval ablations, which show the benefit of iterative evidence seeking. The multiple-run analysis also supports the reliability of the improvements.

  • The improvements from fine-tuning on agent trajectories are quite good (+16.1 F1 over the base model). The analysis of SQL queries and evidence-seeking behavior is also interesting, showing that the model learns to interact with the EHR tools more effectively.

Critical weaknesses
  • I am not fully convinced by the methodological novelty. The framework essentially gives an LLM access to EHR and imaging tools and allows iterative tool calling. While integrating these into a unified clinical framework is useful, I don't see much beyond a standard tool-using agent. There has already been work on EHR agents [1,2]. It would be useful to compare against existing agentic approaches under similar tool access. How do coding agents work in general for this? Why do the authors not compare to existing agentic frameworks? These 2 have been mentioned in the Introduction.

  • The inference-time improvements are relatively small, around 1.5-2.5 F1 points for the two main models. Are these gains significant? I see no statistical analysis.

  • What is the x-ray analysis model used? I notice the image.chest_xray_classifier but what classifier is used?

[1] Shi, Wenqi, et al. "Ehragent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records." Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024.

[2] Liao, Yusheng, et al. "Agentehr: Advancing autonomous clinical decision-making via retrospective summarization." arXiv preprint arXiv:2601.13918 (2026).

Overall, I think the paper addresses a relevant problem and provides a good amount of experimentation. My main concerns are the limited methodological novelty and the lack of training ablations to isolate the contribution of evidence-seeking trajectories. I look forward to the discussion with the authors.

Questions for authors

Mentioned along with weaknesses.

5 comments

Thank you for your thoughtful review and constructive suggestions. We appreciate your recognition of the importance of active clinical evidence seeking, the breadth of our experiments, and the gains from trajectory-based training. We address your concerns and questions point by point below.

W1: Methodological contribution and comparisons with existing agents

Contributions relative to prior work:

Beyond existing clinical agent work, our work offers three distinguishing contributions:

  1. A unified framework for multimodal clinical agent evaluation and training. Unlike prior text-only EHR agents focusing on specific tasks or stages, ClinSeekAgent unifies benchmarking, inference, and training across diverse clinical tasks, including joint EHR–image evidence seeking. As Reviewer hwMT recognized, it provides “a useful foundation for developing and evaluating clinical agents.”
  2. Key insights into why and when automated evidence seeking works. Our controlled experiments show that adaptive multi-round evidence seeking, rather than more data or compute, drives performance gains, and identify the tasks and host-model capabilities where the automated evidence seeking is most effective. As Reviewer 1kMF noted, our paired evaluation “makes the central research question much cleaner than simply comparing two unrelated benchmarks or systems.”
  3. Multi-round evidence-seeking supervision enables effective agent training. Building on these insights, we train ClinSeek-35B-A3B, achieving +16.1 F1 over its base model and open-source SOTA. Our new training ablation further validates the necessity of multi-round supervision (+2.8 F1 over single-round).

Harness comparison and analysis:

EHRAgent [1] generates executable code plans using EHR metadata, medical knowledge, and few-shot demonstrations, then revises them using execution feedback. AgentEHR’s RetroSum [2] periodically revisits the interaction history to generate summaries and, in its evolved version, retrieves strategies from an external experience bank. ClinSeekAgent lets the host model organize its retrieval and reasoning workflow without mandatory periodic summarization or a prescribed few-shot prompting stage.

We evaluated clinical adaptations of RetroSum (non-evolved) [2], ReSum, mini-SWE-agent, and EHRAgent [1]. All methods use GPT-5.6 Sol on the same Text (1,800) and MM (989) questions, with the same clinical tool permissions, prediction-time cutoffs, answer formats, and scoring.

The table reports overall F1 and ΔF1 = baseline − ClinSeekAgent, with 95% paired-bootstrap confidence intervals (4,000 resamples).

HarnessText F1ΔF1 [95% CI]MM F1ΔF1 [95% CI]
ClinSeekAgent59.3Reference62.5Reference
RetroSum (non-evolved)57.4−1.9 [−3.3, −0.5]59.9−2.6 [−4.2, −1.1]
ReSum (training-free)58.4−0.9 [−2.1, +0.2]59.6−3.0 [−4.5, −1.4]
mini-SWE-agent57.7−1.7 [−2.9, −0.4]61.4−1.1 [−2.6, +0.3]
EHRAgent44.7−14.6 [−16.8, −12.3]57.6−5.0 [−7.2, −2.7]

Our harness achieves the highest overall F1 among the evaluated harnesses on both benchmarks. We will clarify our contributions and include these comparisons in the revised manuscript.

[1] Shi, Wenqi, et al. "EHRAgent: Code empowers large language models for few-shot complex tabular reasoning on electronic health records."

[2] Liao, Yusheng, et al. "AgentEHR: Advancing autonomous clinical decision-making via retrospective summarization."

W2: Statistical significance of the inference-time gains

Thank you for raising this point. Statistical significance is indicated in Table 1 and the paired-bootstrap procedure is described in Appendix H.1, but we recognize that these results could be presented more prominently. In the main paper, Table 1 marks statistically significant differences in bold, explicitly defined in its caption as differences whose paired-bootstrap 95% confidence intervals exclude zero. Appendix H.1 specifies the statistical procedure: paired bootstrap over questions with 4,000 resamples.

For clarity, we reproduce the overall gains below, where ΔF1 = ClinSeekAgent − Curated Input and brackets give the 95% confidence interval.

Host modelBenchmarkΔF1
Claude Opus 5ClinSeek-Bench-MM+2.3 [+0.4, +4.1]
GPT-5.6 SolClinSeek-Bench-Text+1.6 [+0.1, +3.1]
GPT-5.6 SolClinSeek-Bench-MM+2.5 [+0.8, +4.3]
Claude Opus 5ClinSeek-Bench-Text+2.2 [+0.6, +3.7]

Section 3.2 (“The improvements are stable across multiple runs”) also reports that ClinSeekAgent outperforms Curated Input in all five GPT-5.6 Sol runs on both benchmarks, with an overall F1 standard deviation of at most 0.4 points within each mode. Appendix C, Table 6 reports run-averaged gains of +2.0 F1 [0.9, 3.1] on Text and +2.7 [1.5, 4.0] on MM, whose confidence intervals also exclude zero.

W3: Image-tool models and tool-removal ablation

The chest_xray_classifier uses TorchXRayVision DenseNet-121 with the densenet121-res224-all weights. The learned image tools use the following pretrained models:

ToolModel / checkpoint
Chest X-ray classificationTorchXRayVision DenseNet-121, densenet121-res224-all
Findings generationIAMJB/chexpert-mimic-cxr-findings-baseline
Impression generationIAMJB/chexpert-mimic-cxr-impression-baseline
Phrase groundingmicrosoft/maira-2
Anatomical segmentationTorchXRayVision PSPNet

MAIRA-2 was trained on the official MIMIC-CXR training split. For the classifier, report-generation checkpoints, and anatomical segmentation model, the released documentation does not provide exact training splits or image-level training lists sufficient to establish their precise overlap with the evaluation data.

To assess how much the results depend on these pretrained image tools, we performed an ablation with Claude Opus 5 on ClinSeek-Bench-MM, disabling the image tools above. Removing these image tools changes overall F1 by only 0.8 points, which is not statistically significant. ClinSeekAgent without image tools still significantly outperforms Curated Input.

We will add the checkpoint names and available training-data information to Appendix F, and include the tool-removal analysis in the revised manuscript.

W4: Training value of evidence-seeking trajectories

Thank you for highlighting “the lack of training ablations to isolate the contribution of evidence-seeking trajectories”. We address the contribution of evidence-seeking trajectories with the following two analyses.

1. Training ablation tests the effect of multi-round trajectories.

We add a controlled comparison between multi-round and single-round trajectory supervision. The single-shot baseline controls for general EHR task adaptation and tool-use proficiency. Its trajectories demonstrate how to formulate retrieval queries and answer from the returned records. The teacher issues all retrieval calls before seeing any results, so the student receives no demonstrations of using earlier evidence to decide what to query next.

The two students share the GPT-5.6 Sol teacher, source training questions, tools, answer format, and fine-tuning recipe. Each is evaluated in its corresponding retrieval mode. The table reports F1 on AgentEHR-Bench (500 questions), with ΔF1 = single-pass − multi-round and 95% paired-bootstrap confidence intervals (4,000 resamples).

TaskMulti-roundSingle-passΔF1
Overall35.632.8−2.8 [−4.6, −0.9]
Diagnoses41.735.0−6.7 [−10.5, −2.8]
Lab events40.438.4−2.0 [−4.2, +0.3]
Microbiology27.125.5−1.6 [−6.5, +3.0]
Procedures41.336.2−5.1 [−10.4, +0.0]
Transfers27.428.9+1.5 [−3.3, +6.1]

The student trained on multi-round trajectories scores 2.8 higher overall F1, with the largest gain on diagnoses (+6.7). The gain over a baseline that already receives EHR task and tool-use supervision supports the additional value of multi-round evidence-seeking demonstrations.

2. Trajectory analysis checks whether training improves evidence acquisition.

We searched the base model's and trained student's tool outputs for the teacher's supporting facts in 288 label-level checks. We also measured how often gold labels appeared in candidate lookups and what fraction of the available EHR tables each model accessed.

Evidence-acquisition measureBaseTrained studentTeacher
Supporting facts retrieved (288 label-level checks)74%98%—
Gold labels appearing in candidate lookup52%79%81%
Coverage of available EHR tables37%74%77%

Coverage of the checked supporting facts rises from 74% to 98% after training. Candidate-label coverage rises from 52% to 79%, close to the teacher's 81%. These measurements show that the student acquires more of the evidence needed for the clinical predictions during tool interaction, connecting the training benefit to evidence-seeking behavior.

We will add both the training ablation and evidence-acquisition analysis to the revised manuscript.

Thank you again for your thoughtful review. We have provided detailed responses and additional agent comparisons and training ablations. As the discussion deadline approaches, we would greatly appreciate your feedback on whether these results address your concerns. If any issues remain, please let us know so we can respond in time.

Reviewer_hwMTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ClinSeekAgent, a framework for automated multimodal evidence seeking in clinical reasoning. Instead of relying on pre-curated patient information, the framework enables LLM agents to iteratively retrieve longitudinal EHR records and invoke medical imaging tools to gather task-relevant evidence.

The paper makes three main contributions:

  • ClinSeekAgent: A unified agentic framework that provides 16 EHR-related tools and 6 imaging tools, allowing models to perform multi-round evidence retrieval and reasoning over heterogeneous clinical data.

  • ClinSeek-Bench: A benchmark constructed from existing EHR and multimodal datasets, containing 1,800 text-based and 989 multimodal examples. The authors compare automated evidence seeking against curated input settings and report consistent overall improvements for two frontier models. Ablation studies further investigate the contributions of full-record access and iterative retrieval.

  • Trajectory-based training: The authors use GPT-5.6 Sol to generate evidence-seeking trajectories and fine-tune Qwen3.5-35B-A3B through supervised fine-tuning. The resulting model improves from 19.5 to 35.6 average F1 on AgentEHR-Bench, outperforming the evaluated open-source models and approaching the teacher's performance of 38.8.

Overall, the paper demonstrates the potential of automated clinical evidence acquisition both as an inference-time framework and as a source of supervision for training smaller clinical agents.

Critical strengths

S1. A practically meaningful framework for automated clinical evidence acquisition.

The paper addresses an important limitation of conventional clinical reasoning benchmarks, where task-relevant patient information is selected before inference. By allowing agents to interact directly with longitudinal EHR records and medical imaging tools, ClinSeekAgent moves toward more autonomous clinical reasoning workflows without requiring benchmark-specific evidence curation at inference time. The unified tool environment, together with temporal availability constraints, provides a useful foundation for developing and evaluating clinical agents. This end-to-end automation capability is, in my view, one of the most valuable aspects of the work.

S2. Strong empirical results across both inference-time and training-time settings.

The experimental evaluation is relatively comprehensive, covering multiple clinical task categories, frontier models, ablations, repeated runs, and trajectory-level behavioral analyses. The training-time results are particularly compelling: using approximately only 2.6k teacher-generated trajectories, the fine-tuned 35B model improves by 16.1 F1 points and achieves competitive performance against substantially larger models. The analysis of SQL usage and targeted evidence retrieval provides additional evidence that training changes the student's interaction behavior, rather than merely improving its final predictions.

Critical weaknesses

W1. The evaluation does not sufficiently establish the claimed advantages of autonomous evidence seeking over alternative evidence-management strategies.

A central motivation of this work is to replace task-specific evidence curation with autonomous, end-to-end evidence acquisition. I find this direction valuable, and I do not consider outperforming curated evidence to be a necessary condition for its success. Achieving comparable performance without task-specific curation could itself be an important contribution.

However, the current evaluation does not sufficiently characterize this advantage.

First, the Curated Input baseline reflects the evidence-packaging procedures of the source benchmarks, rather than necessarily representing a strong or optimized evidence-management strategy. The Full-Record Input baseline simply provides the available patient record within a fixed context window, truncating older records when necessary. In practice, long clinical records could also be handled through hierarchical summarization, retrieval-based compression, or selective retrieval initialized from existing patient summaries. A naive hybrid approach that combines curated context with additional on-demand retrieval seems particularly natural, yet is not evaluated.

Second, while the paper emphasizes removing benchmark-specific curation, the experiments primarily measure prediction F1 on tasks derived from existing benchmarks. They do not directly examine how well the framework generalizes to new clinical tasks, different EHR schemas, or settings where existing task-specific curation rules are unavailable. The experiments demonstrate that the framework can replace the original curated inputs on these benchmarks, but provide more limited evidence for its broader generality.

Finally, the additional computational and tool-use costs are not sufficiently characterized. Such costs are important when assessing the practical trade-off between autonomous retrieval and simpler evidence-management pipelines.

As a result, the experiments support the effectiveness of the proposed pipeline, but do not yet establish its practical advantages over stronger automated or hybrid evidence-selection strategies, nor fully substantiate its broader autonomy and generalization claims.

W2. The training-time contribution lacks sufficient analysis of what the student learns and why it still underperforms the teacher.

The improvement from 19.5 to 35.6 F1 is impressive. However, the training procedure is essentially standard supervised fine-tuning on teacher-generated trajectories. While this is a reasonable approach, the current experiments do not sufficiently distinguish learning an adaptive evidence-seeking policy from acquiring task-specific knowledge, SQL proficiency, tool-use conventions, or output-format familiarity.

More importantly, the analysis focuses primarily on comparing the fine-tuned student with its base model. Given the substantial improvement already demonstrated, I find the remaining gap between the student (35.6) and teacher (38.8) more informative for understanding the limitations of trajectory-based learning.

For example, it remains unclear whether the student underperforms because it fails to retrieve critical evidence, cannot correctly reason over retrieved evidence, selects suboptimal follow-up queries, or terminates the search prematurely. These failure modes imply substantially different bottlenecks and potential improvements. The reported increases in SQL complexity, table coverage, and targeted retrieval frequency do not by themselves establish that the student has learned an effective adaptive search policy.

The paper would benefit from a systematic teacher–student trajectory comparison and a decomposition of their remaining performance differences. Controlled training comparisons, such as answer-only SFT versus tool-call or full-trajectory SFT, would also help isolate the contribution of learning the evidence-seeking process.

Relatedly, the authors could better motivate the choice of offline trajectory SFT relative to iterative approaches such as on-policy distillation or reinforcement learning. I do not consider implementing these alternatives necessary for this paper, but their potential relevance and the limitations of behavioral cloning deserve more discussion.

Without these analyses, the training results convincingly demonstrate effective model adaptation, but provide less conclusive evidence about the acquisition and remaining limitations of adaptive evidence-seeking capabilities.

Questions for authors

Q1. Evaluating autonomy and generalization (related to W1).

The paper emphasizes eliminating benchmark-specific evidence curation. Beyond performance on the reconstructed benchmarks, what evidence supports generalization to previously unseen task types or EHR schemas without additional task-specific engineering? Have the authors evaluated, or could they characterize, the amount of task-specific configuration required when adapting ClinSeekAgent to a new clinical task?

Clarifying the scope of the demonstrated autonomy would materially affect my assessment of the framework's contribution.

Q2. Stronger evidence-management baselines (related to W1).

Could the authors compare against, or provide preliminary results for, stronger alternatives such as hierarchical patient-record summarization, retrieval followed by summarization, or curated input augmented with selective tool-based retrieval?

In particular, the hybrid setting seems important because it combines the efficiency of existing patient summaries with the ability to retrieve missing evidence. Such a comparison would help establish the practical benefit of fully autonomous evidence seeking beyond the current baselines.

Q3. Understanding the teacher–student gap (related to W2).

Can the authors analyze cases where the teacher succeeds but the fine-tuned student fails, distinguishing at least evidence acquisition failures from reasoning or final prediction failures?

For instance, how often does the student retrieve the same critical evidence as the teacher but still produce an incorrect prediction? Conversely, how often is its failure attributable to missing evidence? Even a targeted analysis of the existing evaluation trajectories would substantially strengthen the interpretation of the training results.

Q4. Training attribution and computational cost (related to W2).

To what extent does the student improvement come specifically from learning multi-round evidence-seeking behavior rather than general EHR task adaptation or tool-use proficiency? Are there results comparing full-trajectory SFT against simpler supervision, such as final-answer-only or tool-call-only training?

Additionally, could the authors report the approximate GPU-hours, peak GPU/CPU memory requirements, and teacher trajectory generation costs? The reported full-parameter fine-tuning of a 35B MoE model on two H200 GPUs is technically plausible with the described optimizer offloading and memory optimizations, but more detailed resource reporting would be valuable for assessing reproducibility and scalability.

5 comments

Thank you for your thoughtful review and constructive suggestions. We appreciate your recognition of the practical value of autonomous clinical evidence acquisition and the strong empirical results in both inference-time and training-time settings. We address your concerns and questions point by point below.

W1/Q1: Autonomy and generalization

The generalization of ClinSeekAgent is supported by two designs:

  • System prompt without task-specific configuration: We use a unified system prompt template across all 45 ClinSeek-Bench-Text subtasks (Sec. 3.1) without specifying the retrieval strategy and workflow for each task. The agent inspects the available tables and columns, chooses the evidence sources and time windows to search, and analyzes each round’s results to decide what to retrieve next. In Fig. 5, for example, a CT order found in one round leads the agent to query the corresponding radiology report.

  • Tool space without database dependency: The EHR tool interface is independent of any particular relational schema. It provides general operations for inspecting tables and columns and executing free-form SQL (Appendix F, Table 9). The agent discovers the table and column names at runtime and uses them to construct the filters, joins, and aggregations the task requires. The same tool interface can therefore be used across relational EHRs.

Together, these designs let ClinSeekAgent automatically decide what evidence to seek for a clinical task and how to query the connected EHR to obtain it. Once an EHR is available through this interface, inference requires only the clinical task description, patient identifier, and prediction timestamp.

W1/Q2: Stronger evidence-management baselines

Thank you for this suggestion. We evaluated all three proposed alternatives on ClinSeek-Bench-Text and ClinSeek-Bench-MM, using GPT 5.6 Sol with the same reasoning effort.

Experiment Settings:

  • For hierarchical summarization, we divided the most recent 100K tokens of the pre-cutoff record into segments of approximately 40K tokens. Each segment was summarized without seeing the question, preserving timestamps, values, and sources; the segment summaries were then merged into a patient-level summary from which the model answered without retrieval.

  • For Single-Pass + Summarization, we reused the records retrieved by the original Single-Pass run, generated a question-focused factual summary, and answered from it without further retrieval.

  • For curated input augmented with selective tool-based retrieval (hybrid setting), the agent instead received the original curated evidence package at initialization and could either answer directly or retrieve additional evidence, with the same tools, turn limit, and answer format as ClinSeekAgent.

The following tables report ΔF1 = setting − ClinSeekAgent, in F1 points, with 95% paired-bootstrap confidence intervals over questions (4,000 resamples). Bold values indicate that the confidence interval excludes zero. On ClinSeek-Bench-Text (1,800 questions), we obtained:

SettingOverallDecision makingRisk prediction
Hierarchical summarization−5.7 [−7.3, −4.0]−8.1 [−10.6, −5.7]−1.9 [−3.9, +0.1]
Single-Pass + Summarization−1.7 [−3.0, −0.4]−2.2 [−4.3, −0.2]−1.0 [−2.5, +0.6]
Curated Input + selective retrieval (hybrid)−0.3 [−1.7, +1.1]−1.6 [−3.6, +0.5]+1.7 [−0.1, +3.5]

The corresponding results on ClinSeek-Bench-MM (989 questions) were:

SettingOverallEHRXQA (497)MedMod (492)
Hierarchical summarization−5.4 [−7.3, −3.5]−3.5 [−6.5, −0.5]−7.3 [−9.5, −5.0]
Single-Pass + Summarization−4.1 [−5.8, −2.3]−2.7 [−5.8, +0.3]−5.5 [−7.3, −3.7]
Curated Input + selective retrieval (hybrid)−3.6 [−5.1, −2.0]−2.0 [−4.8, +0.7]−5.1 [−6.5, −3.7]

Hierarchical summarization falls behind ClinSeekAgent by 5.7 F1 points on Text and 5.4 on MM, with the largest Text loss in decision-making tasks (8.1 points). This gap persists on the 972 Text questions whose complete records fit within the 100K-token budget: 49.3 versus 55.2 F1.

Adding a question-focused summary after one retrieval round also does not recover ClinSeekAgent's accuracy: Single-Pass + Summarization is lower by 1.7 F1 points on Text and 4.1 on MM. In these comparisons, selecting evidence once and compressing it is less effective than continuing to acquire evidence through interaction.

The hybrid comparison is particularly informative because it gives the agent the benchmark's curated evidence at the outset. It shows no statistically significant overall advantage on Text (−0.3 F1, 95% CI [−1.7, +1.1]) and is lower by 3.6 F1 points on MM. On Text, it answers directly from the curated input on 78% of questions and retrieves additional evidence on 22%. Its cost is approximately 1.9 times as much of ClinSeekAgent, because the curated context is retained across model calls.

Overall, ClinSeekAgent achieves competitive performance against these stronger alternatives directly from raw patient records, without requiring a task-specific curated evidence package. This supports the practical value identified in the review: comparable performance without task-specific evidence curation.

W2/Q3: Understanding the teacher–student gap

Thank you for this suggestion. We added a paired, fact-level trajectory analysis to distinguish evidence-acquisition failures from reasoning and final-prediction failures. We compared the original ClinSeek-35B-A3B student trained on ClinSeekAgent trajectories with its GPT-5.6 Sol teacher on AgentEHR-Bench. The student achieved 35.6 F1 and the teacher 38.8 F1, a gap of 3.2 F1 points.

We analyzed 200 questions on which the teacher achieved higher per-question F1. For labels correctly predicted by the teacher but missed by the student, we identified the supporting record facts in the teacher trajectory, searched for those facts in the student's complete tool outputs, and assigned a primary failure stage to each question. The resulting breakdown was:

Primary failure categoryQuestionsShare of 200
Evidence utilization / clinical inference7638.0%
Answer calibration8341.5%
Label mapping3517.5%
Evidence acquisition42.0%
Other21.0%

Clinical inference, answer calibration, and label mapping account for 194/200 (97%) of these teacher-better cases; missing key evidence is the primary cause in only 4/200 (2%). Consistently, among the 415 labels correctly predicted by the teacher but missed by the student, the teacher's supporting facts were already present in the student's tool outputs in 87% of cases. Thus, the dominant difficulties occur after evidence acquisition: deriving a clinical conclusion from the retrieved facts, deciding which labels are justified, and mapping clinical concepts to the official label vocabulary.

For example, both models read the same urine results: mild pyuria, negative leukocyte esterase and nitrites, and a urine culture with <10,000 organisms/mL. The student treated these results as insufficient evidence of infection and predicted urine culture and its variants. The teacher considered urethritis in light of the pyuria and low-growth culture in a young patient, and included chlamydia and gonorrhea nucleic acid tests, both gold labels. The student had retrieved the relevant findings but did not draw the clinical inference needed to identify these tests.

The analysis was annotated by Claude Opus 5. Independent GPT-5.6 Sol reannotation of 40 questions yielded 88% agreement on the primary failure category (κ = 0.82), and 362/363 evidence-presence judgments were supported by searchable facts in the corresponding student tool outputs.

Taken together, these results show that our trajectory training teaches the student to acquire task-relevant evidence, consistent with our claim. The remaining teacher–student gap is concentrated in interpreting and using evidence that has already been retrieved, suggesting limitations in the student's underlying medical knowledge. Small-scale trajectory SFT does not fully compensate for this broader knowledge gap. We will add the teacher–student analysis and discuss these findings and limitations in the revised manuscript.

Comments on the paper

5 comments

Thank you for your review and for highlighting the importance of evidence acquisition in clinical reasoning. We address the questions on inference compute and the image tools below.

W1/Q1: Test-time compute and evidence seeking

Thank you for raising this point. We report the inference cost of each setting and add a fixed-evidence multi-round control.

Inference cost:

The following table reports overall F1, as in the paper, and per-question usage on ClinSeek-Bench-Text (1,800 questions) for both host models. Output tokens include reasoning tokens.

HostSettingOverall F1Model callsTool callsOutput tokensCost / question
Claude Opus 5Curated Input61.71.00576\$0.145
Claude Opus 5Full-Record Input62.61.00500\$0.250
Claude Opus 5Single-Pass Retrieval58.52.08.4941\$0.090
Claude Opus 5ClinSeekAgent63.910.015.45,913\$0.204
GPT-5.6 SolCurated Input57.81.00547\$0.089
GPT-5.6 SolFull-Record Input59.61.00528\$0.154
GPT-5.6 SolSingle-Pass Retrieval57.62.011.31,707\$0.118
GPT-5.6 SolClinSeekAgent59.38.326.22,518\$0.121

ClinSeekAgent makes 10.0 (Claude Opus 5) and 8.3 (GPT-5.6 Sol) model calls per question on average and generates 10.3 and 4.6 times as many output tokens as Curated Input. For both hosts, its cost per question is about 1.4 times that of Curated Input (\0.204vs.0.204 vs. \\0.145 for Claude Opus 5; \0.121vs.0.121 vs. \\0.089 for GPT-5.6 Sol) and lower than that of Full-Record Input (\0.250and0.250 and \\0.154).

Fixed-evidence multi-round control:

To test whether more reasoning over the same evidence can match ClinSeekAgent, we evaluated Curated Input + Multi-Round Reasoning on ClinSeek-Bench-Text with GPT-5.6 Sol. Each question receives as many model calls as ClinSeekAgent made on that question (4–18 calls, mean 8.3). The first call is the original Curated Input request. Each intermediate call asks the model to check every item of its current answer against the given record, look for contradictions and overlooked findings, and revise the answer; the last call gives the final answer.

The following table compares the multi-round control with Curated Input and ClinSeekAgent on ClinSeek-Bench-Text with GPT-5.6 Sol. Each ΔF1 is the row setting minus the reference setting named in the column header, with the 95% paired-bootstrap confidence interval over questions (4,000 resamples).

SettingCost / questionOverall F1ΔF1 vs. Curated InputΔF1 vs. ClinSeekAgent
Curated Input\$0.08957.8Reference−1.6 [−3.0, −0.1]
Curated Input + Multi-Round Reasoning\$0.20357.7−0.1 [−1.3, +1.2]−1.7 [−3.2, −0.2]
ClinSeekAgent\$0.12159.3+1.6 [+0.1, +3.0]Reference

Curated Input + Multi-Round Reasoning reaches 57.7 F1. It does not significantly improve on Curated Input (−0.1, 95% CI [−1.3, +1.2]) and remains significantly below ClinSeekAgent (−1.7, 95% CI [−3.2, −0.2]). It makes the same number of model calls per question as ClinSeekAgent and uses more computation: 4,832 output tokens and \0.203perquestion,comparedwith2,518outputtokensand0.203 per question, compared with 2,518 output tokens and \\0.121 for ClinSeekAgent, because the full curated input is sent with every call. These results indicate that what Curated Input lacks is the necessary evidence that the model must decide to acquire on its own, rather than a larger reasoning budget. This control thus distinguishes the benefit of iterative evidence acquisition from the benefit of additional test-time computation.

We will add these cost statistics, the corresponding statistics for ClinSeek-Bench-MM, and the fixed-evidence control to the revision.

W2/Q2: Image models and tool-removal analysis

Thank you for raising this point. The table below lists the checkpoints of the learned image tools and the training data reported in their public documentation.

ToolModel / checkpointTraining data
Chest X-ray classificationTorchXRayVision DenseNet-121, densenet121-res224-allNIH ChestX-ray14 (with NIH-Google labels), PadChest, CheXpert, MIMIC-CXR, OpenI, and RSNA Pneumonia
Findings generationIAMJB/chexpert-mimic-cxr-findings-baselineMIMIC-CXR and CheXpert Plus
Impression generationIAMJB/chexpert-mimic-cxr-impression-baselineMIMIC-CXR and CheXpert Plus
Phrase groundingmicrosoft/maira-2MIMIC-CXR (official training split), MS-CXR, PadChest, and a private dataset (USMix)
Anatomical segmentationTorchXRayVision PSPNet1,000 annotated chest X-rays; source not reported

Data overlap:

MAIRA-2 was trained on the official MIMIC-CXR training split. For the classifier, report-generation checkpoints, and anatomical segmentation model, the released documentation does not provide exact training splits or image-level training lists sufficient to establish their precise overlap with the evaluation data.

Tool-removal analysis:

To test whether the multimodal gain depends on the image tools, we evaluated ClinSeekAgent with Claude Opus 5 after removing all six image tools. The chest X-ray images remained attached to the prompt. Removing these image tools changes overall F1 by only 0.8 points, which is not statistically significant. ClinSeekAgent without image tools still significantly outperforms Curated Input.

We will add the checkpoints, available training-data information, and tool-removal analysis to Appendix F.

Q3: Generality across host models

We analyzed the paired per-question results and the agent trajectories of the three additional hosts (Claude Opus 4.6, GLM-5.2, and Kimi K2.6) on ClinSeek-Bench-Text, with GPT-5.6 Sol as the strong-host reference.

Capabilities a host needs to benefit:

A per-question comparison with Sol shows that for 58–71% of the questions on which a weaker host scores lower with ClinSeekAgent than with Curated Input, GPT-5.6 Sol with ClinSeekAgent scores at least as high as that host with Curated Input. These losses therefore reflect host capability rather than question difficulty. The trajectories and failure annotations of all lost questions identify three capabilities in which the weaker hosts fall short of Sol:

  • Reliable query execution. Kimi K2.6 gets empty results on 28% of its data queries and has five or more consecutive failed or empty queries on 22% of the questions (Sol: 12% and 8%).

  • Retrieval planning. Claude Opus 4.6 reads the fewest tables and covers 48% of the tables in the curated input (Sol: 70%). In 11% of its lost questions, it never retrieves key evidence that the curated input contains (Sol: 0%).

  • Recognizing the necessary evidence. GLM-5.2 and Kimi K2.6 often retrieve the key evidence without recognizing it as the basis for the answer, and keep reasoning until a model call reaches its output limit without an answer. This occurs in 19% and 35% of their lost questions (Sol: 0%); in 96% and 93% of these questions, the key evidence had already been retrieved in full or in part.

Graceful degradation:

When a host ends its trajectory without a final answer, as in the cases above, ClinSeekAgent stops evidence seeking and adds one answer-only round, in which the host can only call the finish tool and answers from the evidence it has already retrieved. This round produces an answer for 98–99% of the affected questions and raises the overall F1 of ClinSeekAgent from 50.0 to 51.7 with GLM-5.2 and from 45.4 to 49.9 with Kimi K2.6.

We will add this analysis and the answer-only round to the revision.

Thank you for your detailed response. I have just one remaining question: what do you consider to be the clinical significance of this approach? Given the large number of similar protocols currently available, what makes yours unique?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.