acceptodds
Under review as a conference paper at ICLR 2027

MedCaseAgent: From Multimodal Clinical Evidence to Rare Disease Case Reports

Abstract

Although Large Language Models (LLMs) have demonstrated potential in clinical documentation, generating the rigorous, evidence-based reasoning required for unprecedented, atypical and rare disease case reports remains a significant challenge. We present MedCaseAgent, an end-to-end agentic reasoning paradigm that bridges this gap by synthesizing publication-quality reports from unstructured clinical notes. MedCaseAgent leverages a hierarchical structure, integrating patient-level data mining with iterative, tool-integrated reasoning. By dynamically querying medical literature and knowledge bases, the agent maintains high clinical fidelity and assesses case novelty. The framework is inherently modular, supporting various backbone LLMs in a plug-and-play fashion. To facilitate this research, we release MedCase-150K, a large-scale multimodal clinical case-report corpus, and MedCase-Bench, a robust temporally held-out evaluation suite. We also introduce an open-source model post-trained on Qwen3.5-9B optimized specifically for this task. Under both LLM-judged and rule-based evaluation, MedCaseAgent outperforms existing general-purpose and specialized medical LLMs and agents. Blinded expert reviews and ablations further support the system's clinical relevance and evidence grounding.

Primary area:
datasets and benchmarks
Keywords:
Language and multimodal language models, Agents

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_ewP8OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MedCaseAgent, a tool-augmented framework for generating clinical case reports from fragmented clinical facts and medical images. It proposes a planner–writer–editor workflow with literature retrieval, image inspection, gene lookup, and similar-case retrieval. The authors introduce MedCase-150K and a 584-case temporal evaluation benchmark, train a 9B planner, and evaluate the system.

Critical strengths
  1. evidence-aware workflow. The framework explicitly separates patient-specific facts from external literature and plans citation and figure placement before drafting, addressing concrete challenges in producing coherent, well-supported case reports.

  2. Task-specific resources and complementary evaluation. The large case-report corpus and dedicated benchmark support research on an underexplored writing task. Backbone comparisons, tool ablations, and evaluation on original hospital records provide complementary evidence about the system’s capabilities.

Critical weaknesses
  1. unreal tasks. The main quantitative results are based on simulated inputs derived from published case reports. Validation on real clinical records is limited in scale, leaving the gap between extracted clinical atoms and real-world clinical inputs insufficiently assessed and limiting the study’s generalizability.

  2. No comparisons between generated and reference reports. The paper does not provide statistical comparisons between generated and human-authored reference reports, such as whether the generated reports are longer or contain more references. These factors can directly influence the preferences of LLM judges. Furthermore, although the paper requires generated reports to include at least 10 verified references, it does not compare evaluation scores with and without controlling for reference count. These ambiguities in the evaluation limit confidence in the paper’s claims.

Questions for authors

N/A

5 comments

We sincerely thank the reviewer for the careful review and constructive feedback, as well as for recognizing the value of our evidence-aware framework and complementary evaluations. We address the concerns below.

W1. Realism of the task and scale of real-world validation.

We thank the reviewer for raising this important concern regarding the relationship between report-derived clinical atoms and original clinical records. We would like to further clarify the motivation behind our benchmark design and the complementary role of our hospital-based evaluation.

Benchmark design and motivation. Our goal is to establish a scalable and controlled benchmark for clinical case-report generation. Clinical atoms provide a standardized representation of patient-specific evidence, retaining key clinical observations while removing background literature and reference-driven claims that the agent must independently reconstruct. This representation enables systematic comparisons across methods under consistent input conditions. We acknowledge that atoms are not equivalent to complete EHR records; rather, they offer a practical basis for large-scale evaluation of the reasoning, literature retrieval, novelty assessment, and multimodal synthesis involved in case-report writing. Collecting matched longitudinal multimodal hospital records and corresponding clinician-written reports at a comparable scale remains challenging due to privacy requirements, institutional governance, cross-hospital coordination, and the substantial involvement of clinical experts.

Clinical relevance of novelty recognition. We would also like to emphasize that clinical case-report generation involves more than reorganizing patient records. An important purpose of case reports is to identify and communicate rare diseases, unprecedented findings, and atypical clinical presentations, thereby contributing potentially valuable observations to the broader medical community. Such reports can help clinicians recognize similar presentations, investigate potential mechanisms, and improve understanding of rare conditions. Accordingly, novelty recognition through literature retrieval and comparison is an important component of this task. MedCaseAgent explicitly addresses this aspect and achieves 88.35% novelty recognition on MedCase-Bench (Table 2), alongside evidence-grounded report generation.

Validation on original hospital records. To complement the large-scale benchmark, we conducted a hospital-based evaluation involving 30 de-identified multimodal clinical cases from 10 hospitals. These cases involve rare or clinically complex presentations and include original longitudinal records and corresponding clinician-prepared reports. Fourteen clinical experts participated, with three independently assessing each case against the original records. For each patient, we generated reports using both original clinical records and report-derived atoms, alongside the clinician-written report. The results are summarized below:

ReportOverall qualityPatient timelineOmission control
MedCaseAgent (raw records)7.47.57.8
MedCaseAgent (clinical atoms)7.67.67.6
Clinician-written7.57.87.9
Raw − clinician, mean [95% CI]−0.10 [−0.36, 0.16]−0.30 [−0.62, 0.02]−0.10 [−0.39, 0.19]

The two input settings yield closely comparable expert scores, with differences of at most 0.2 points across the three dimensions. Reports generated directly from original records also achieve comparable overall quality to clinician-written reports (7.4 vs. 7.5). In the source-grounded assessment, experts identified no hallucinated content across the 30 cases and omitted patient facts in only one case. These findings provide encouraging initial evidence of the system's applicability to original clinical documentation.

We recognize that report-derived atoms may condense certain aspects of the original records. As illustrated by the same-patient comparison in Appendix F.1 (Table 7), atoms retain key clinical observations while compressing some temporal context, source provenance, and conflicting information. Processing original records therefore involves additional challenges in document filtering, evidence reconciliation, and timeline reconstruction. These differences motivate our complementary evaluation design: the large-scale atom-based benchmark enables controlled and reproducible comparisons, while the hospital-based evaluation examines system performance directly on original clinical evidence.

We agree that prospective validation and a broader collection of hospital cases would further strengthen the evidence. However, assembling matched longitudinal multimodal records and clinician-reviewed reports for rare and clinically complex cases at scale requires considerable clinical coordination and expert involvement. We are continuing to collaborate with clinical partners to expand this evaluation. We hope that the combination of large-scale benchmark evaluation and expert assessment on original hospital records provides useful evidence of the potential of MedCaseAgent for clinical case-report generation.

W2. Length and reference count as potential confounders.

We thank the reviewer for highlighting this important concern. We agree that manuscript length and reference count may influence LLM-based quality assessments. To better understand their effects, we examined the corresponding distributions and conducted additional controlled comparisons on the Table 1 cases, where generated reports and human publications are evaluated using the same rubric.

ReportWords, mean ± SDReferences, median [IQR]
Human Publication1,477 ± 77714 [10–20]
GPT-5.4 (without MedCaseAgent)1,648 ± 73013 [10–17]
MedCaseAgent-Base (Qwen3.5-9B)1,545 ± 71612 [10–16]
MedCaseAgent-Tuned (Qwen3.5-9B, SFT+GRPO)1,567 ± 78413 [11–17]
MedCaseAgent-Max (GPT-5.4)1580 ± 69716 [12–21]

To assess whether these differences account for the observed performance advantage, we further examined several settings:

Evaluation settingMedCaseAgent-Max − Human Publication
Original evaluation (Table 1)+7.59
Regression-adjusted for log word count and reference count+5.84
References capped at the published report's count, then re-evaluated+6.18
Generated without the 10-reference minimum+6.43
Length-matched subset (±15% words, n = 192)+5.67

MedCaseAgent-Max maintains a consistent advantage across these settings, suggesting that the observed improvements cannot be explained solely by manuscript length or reference count.

We would also like to clarify that the minimum requirement of 10 references is intended to encourage adequate literature grounding instead of maximizing citation quantity. The median number of references in MedCase-150K publications is 14, placing this threshold below typical case-report citation counts.

Furthermore, to complement LLM-based assessment, we conducted an additional blinded clinician evaluation covering the same five quality dimensions. Across 15 cases, three clinicians independently evaluated each report, including MedCaseAgent-Base, MedCaseAgent-Max, and human-authored publications. The correlations between LLM-judge scores and clinician assessments ranged from 0.70 to 0.81 across individual dimensions, with an overall Spearman correlation of 0.82 for the five-dimension average.

Together with our hospital-based evaluations, where clinicians assess generated reports directly against original patient records, these findings provide complementary support for the reliability of our report-quality evaluation.

We sincerely appreciate the reviewer's thoughtful suggestions, which have helped us strengthen the evaluation and clarify the limitations of our study. We hope these additional analyses adequately address the concerns, and we welcome any further feedback.

Reviewer_uNGTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents MedCaseAgent, a multi-phase planner-writer-editor workflow for generating rare-disease clinical case reports from structured clinical atoms and medical images, alongside the MedCase-150K corpus and a post-January-2026 evaluation split (MedCase-Bench, 584 cases). Validation combines multi-LLM judge ensembles across report quality dimensions, automated grounding checks, and blind clinician evaluation on 30 retrospective hospital cases.

Critical strengths
  • Curating a large-scale multimodal rare-disease case collection with a temporally separated post-2026 evaluation split and a 30-case clinician study addresses a meaningful documentation bottleneck in rare-disease care.
  • Integrating literature verification, differential diagnosis reasoning, and visual figure sequencing into a structured planner-writer-editor pipeline offers a well-organized design for long-form clinical document generation.
  • Evaluating both open-literature cases and real-world hospital records across content quality and factual grounding dimensions provides a multi-faceted empirical assessment.
Critical weaknesses
  • Constructing input "clinical atoms" by compressing published case reports into JSON that explicitly includes the final diagnosis array (e.g., "diagnosis": ["Erdheim-Chester disease (ECD)", ...] in Appendix B, Figure 6, p. 21) and supplies source section headings to the planner (Figure 8, p. 22) provides post-diagnostic cues that simplify literature retrieval and report structuring compared to raw pre-diagnostic EHRs.
  • Comparing against medical agentic baselines (MedAgents, MDAgents, MedAgent-Pro, MMedAgent) in Table 1 (p. 7) on 453 cases sampled from MedCase-150K—which spans 2000–2026 and also serves as the RAG retrieval pool and planner training source—makes it difficult to rule out corpus overlap, especially since those medical agentic baselines are not evaluated on the temporally held-out MedCase-Bench split.
  • Planner training combines SFT on 3,000 trajectories distilled from Qwen3.5-397B-A17B with 500 GRPO steps (Section 3.4; Appendix E, p. 28), but Table 3 and Appendix Table 6 do not include an SFT-only baseline, leaving it unclear how much of the gain over the base model comes from teacher distillation versus GRPO.
Questions for authors
  • How does MedCaseAgent perform when the definitive diagnosis field and source section headers are withheld from the input clinical atoms, and was PubMed retrieval on MedCase-150K temporally restricted to each case's publication date?
  • Could the authors report an SFT-only checkpoint prior to GRPO in Table 3, and evaluate the medical agentic baselines (or MedCaseAgent-Tuned) on the post-2026 MedCase-Bench split?
  • Could the authors clarify the minor numerical differences for MedCaseAgent-Tuned between Table 3/Figure 5 (59.80% / 67.79) and the text/Appendix Table 6 (59.08% / 67.23), as well as between Table 2 (69.18%) and Appendix Table 9 (36.09%) on Authorship Indistinguishability?
2 comments

We sincerely thank the reviewer for the careful review and constructive feedback, as well as for recognizing the value of our work. We address the concerns and questions below.

W1. Diagnosis information and source section headings.

We would like to clarify that MedCaseAgent focuses on post-diagnostic case-report writing instead of clinical diagnosis. In clinical practice, case reports are typically prepared after a diagnosis has been established. The confirmed diagnosis provides essential context for literature retrieval, comparison with prior cases, and novelty assessment.

Our clinical atoms retain the diagnosis but remove the original authors' diagnostic exposition, differential reasoning, and literature support, which the agent must reconstruct through independently retrieved evidence. Therefore, diagnosis-guided retrieval is consistent with our intended clinical writing workflow.

We also acknowledge that source-specific headings may provide structural cues. To examine their influence, we conducted additional input ablations, reported in Q1 below. Furthermore, our hospital-based evaluation demonstrates that MedCaseAgent can generate reports directly from original clinical records without source-derived headings or clinical atoms, achieving comparable overall quality to clinician-written reports (7.4 vs. 7.5).

W2. Potential overlap with MedCase-150K and information leakage.

We appreciate the reviewer's concern regarding potential data overlap. We clarify the separation between training, retrieval, and evaluation:

  • Training: The planner's 3,000 SFT trajectories are derived from the MedCase-150K training split, which excludes the 453 Table 1 evaluation cases. MedCaseAgent-Base does not involve task-specific training.
  • Retrieval: Disease-Index RAG excludes the evaluated source reports, associated records, and duplicate versions, and retrieves only cases published before the evaluated case. For PubMed retrieval, source matches can be detected for Exact DOI scoring, but the source title, abstract, and full text remain masked from the planner.
  • Evaluation: MedCase-Bench provides a separate temporally held-out evaluation with 584 cases published after January 2026. We additionally evaluated all four medical-agent baselines on this benchmark, as reported in Q2.

W3. Contribution of SFT and GRPO.

We agree that separating the contributions of teacher distillation and GRPO is important. We therefore include an SFT-only checkpoint using the same Qwen3.5-9B backbone and evaluation protocol:

Planner (Qwen3.5-9B)Average qualityNovelty rec. (%)Citation depthImage integration
No training (Base)64.4352.4059.0660.00
SFT only66.3156.5161.4262.68
SFT + GRPO (Tuned)67.7959.8063.1664.54

SFT improves average report quality by 1.88 points, with GRPO contributing a further 1.48 points. Novelty recognition also increases progressively from 52.40% to 56.51% and 59.80%, respectively.

We acknowledge that the overall gains from post-training are relatively modest. Nevertheless, the consistent improvements from SFT to SFT+GRPO, achieved with only 3,000 teacher-generated trajectories despite the substantial effort involved in collecting high-quality agentic trajectories, suggest that MedCase-150K provides useful task-specific supervision for case-report planning. The additional gains from GRPO further support the value of our reward design beyond supervised fine-tuning alone.

Q1. Input ablations and temporal restrictions.

Input ablations. Following the reviewer's suggestion, we evaluated MedCaseAgent-Max on MedCase-Bench under different input configurations:

Planner inputReport qualityExact DOINovel Rec.Fig. Seq.
Full input (original)80.7484.5988.3587.34
Generic CARE headings80.1683.3987.1684.25
Diagnosis field removed77.8372.6082.8886.30
Both modifications77.1171.7581.8583.22

Replacing source-specific headings with generic CARE headings has only a modest impact on report quality (80.74 → 80.16), suggesting that the framework does not heavily depend on source-specific structure. Removing the diagnosis introduces an additional diagnostic inference requirement beyond our intended post-diagnostic writing task, so we regard this as a stress test.

Temporal restrictions. Disease-Index RAG retrieves only cases published before the evaluated case, in addition to excluding source reports and associated records. We apologize that this restriction was not explicitly stated in the submission and will clarify it in the revised manuscript.

For live PubMed retrieval, the source article was masked, but no publication-date restriction was originally imposed. To examine the potential influence of later publications, we re-evaluated the Table 1 systems with PubMed restricted to literature published before each case's source publication:

SystemOriginalPrior literature onlyChange
MedAgent-Pro63.5162.73−0.78
MedCaseAgent-Base65.1864.48−0.70
MedCaseAgent-Max83.1782.25−0.92

The performance drops are small and comparable across systems, with the relative ranking unchanged. These results suggest that access to later publications has a limited impact on the reported performance advantages.

Q2. SFT-only checkpoint and medical-agent baselines on MedCase-Bench.

The SFT-only results are provided in W3, distinguishing teacher distillation from the additional contribution of GRPO.

We also evaluated all four medical-agent baselines on the temporally held-out MedCase-Bench using the same Qwen3.5-9B backbone and evaluation protocol:

Method (Qwen3.5-9B)Report qualityExact DOINovel Rec.Fig. Seq.
MedAgents58.9226.0344.5256.16
MDAgents59.9627.2346.2357.88
MedAgent-Pro62.8529.2850.3460.10
MMedAgent62.4128.7749.6659.59
MedCaseAgent-Base64.4330.6052.4061.10
MedCaseAgent-Tuned67.7934.7659.8064.55

MedCaseAgent-Base outperforms all four medical agents across the reported metrics, even without task-specific planner training. MedCaseAgent-Tuned further improves report quality and novelty recognition, supporting the effectiveness of both the agentic workflow and planner adaptation on the held-out benchmark.

Q3. Clarification of numerical differences.

We thank the reviewer for carefully identifying these inconsistencies and apologize for the insufficient clarification in the submission.

MedCaseAgent-Tuned: 59.80% / 67.79 vs. 59.08% / 67.23. These values come from separate evaluation runs of the same checkpoint under the same evaluation protocol. Stochastic sampling during agent execution and judging, together with live PubMed retrieval, introduces small variations between runs. Table 3 and Figure 5 report one run, whereas Appendix Table 6 reports a separate reward-weight sensitivity evaluation. The observed differences (0.72 and 0.56 points) are smaller than the effects of removing the DOI reward within the same sensitivity experiment (3.30 and 1.86 points). We will clarify the distinction between these evaluation runs in the revised manuscript.

Authorship Indistinguishability: 69.18% vs. 36.09%. These numbers correspond to different datasets and evaluation definitions. Table 2 evaluates MedCase-Bench (584 cases) using a case-level indistinguishability criterion, whereas Appendix Table 9 reports individual judge-level identification results on the 453 Table 1 cases. In Appendix Table 9, MedCaseAgent-Max is identified as generated in 63.91% of judgments, with the remaining 36.09% identified as human-authored.

We acknowledge that these definitions were not sufficiently distinguished. In the revised manuscript, we will clarify the dataset and metric definitions in the corresponding captions and adopt a consistent reporting convention to avoid confusion.

We sincerely appreciate the reviewer's careful reading and constructive suggestions. We hope these clarifications and additional results adequately address the concerns and help strengthen our work.

Reviewer_w3uzOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces MedCaseAgent, an agent for generating clinical case reports. It also introduces a large case-report dataset and evaluates the system against other models and agentic approaches. The authors report quality and evidence grounding, supported by evaluations on published case reports and a smaller study using real clinical records.

Critical strengths

The paper tackles a useful application of LLM agents and provides a relatively comprehensive evaluation across different models and clinical domains.

Most importantly, the authors also evaluate the system on real clinical cases. This strengthens the paper substantially, since it evaluates the system beyond its synthetic report reconstruction benchmark.

Critical weaknesses
  1. The main benchmark evaluates reconstruction of existing reports.

The benchmark is constructed by using an LLM to compress published case reports into clinical observations, then asking the system to reconstruct the original reports. This raises questions about how closely the task resembles writing reports from original clinical evidence, and whether the compression preserves information from the final narrative. The real clinical evaluation alleviates this concern, but deserves substantially more attention.

  1. The relative contributions of the agent architecture, backbone model, and fine-tuning are unclear.

The distinctions between MedCaseAgent-Base, MedCaseAgent-Tuned, MedCaseAgent-Max, and standalone model baselines take considerable effort to understand. The improvement from SFT/GRPO is comparatively modest, while stronger backbone models account for much larger gains. A clearer presentation of which comparisons isolate the agent architecture and model effects would help. In particular, I would suggest:

  • Relegating the GRPO finetune with modest effects to the appendix. Although undoubtedly effortful, it does not add much.
  • Run a symmetric experiment of with/without harness, i.e. both on the same set of models. Ablating components of the agentic framework could also be interesting, but perhaps an unreasonably large addition at this point.
  1. Source-document retrieval as an evaluation target. [AI-identified concern]

The exact DOI retrieval metric rewards finding the published report from which the clinical observations were generated. Although the source is masked during report generation, retrieving that specific document is not necessarily equivalent to finding useful independent clinical evidence. This metric, which is also part of the planner-training reward, may favor reconstructing the original publication rather than supporting the creation of a new case report. It is not clear how this metric relates to the goal of writing high-quality, new, case reports,

Questions for authors
  1. Can the authors expand Section 4.4? In particular, which MedCaseAgent configuration was evaluated on real clinical records, and what are the detailed results relative to clinician-written reports?

  2. How much of the reported performance improvement comes from the agent architecture rather than the backbone model? A symmetric comparison with and without the agent framework across the same models would be helpful.

  3. In Appendix F.3, could the authors report the distribution of correct AI identifications per clinician (0–5), overall accuracy, and a statistical comparison with chance? The reported "26/32 clinicians achieving at most 3/5 correct identifications" appears needlessly incomplete.

A few minor presentation comments:

  • The introduction mentions specific models somewhat arbitrarily, and the claim of producing "academic-grade" case reports could be better substantiated.
  • The related-work section reads largely as a citation dump. Fewer references with more contextual discussion would be preferable.
  • Section 3.1 introduces mathematical notation that adds little and is barely reused. Similarly, the set-union notation on page 5 is confusing if concatenation of retrieved evidence is intended.
5 comments

We sincerely thank the reviewer for the thoughtful and constructive feedback, as well as for recognizing the value of our proposed framework and its evaluation on real clinical cases. We appreciate the insightful suggestions and address the concerns in detail below.

W1. Reconstruction from compressed reports.

We thank the reviewer for raising this important concern. We agree that generating reports from compressed published cases differs from writing directly from original clinical records. Nevertheless, we believe that reconstruction-based evaluation can provide a useful and complementary perspective on the capabilities required for real-world clinical writing.

Benchmark design and motivation. Our motivation for using published case reports is to establish a controlled evaluation setting with human-authored references, enabling systematic comparisons across models. Our atomization process preserves patient-specific clinical evidence while removing background literature, reference-driven claims, diagnostic exposition, figure captions, and explicit image–text links. Although these inputs are derived from published reports, the model must still organize clinical evidence, retrieve and verify supporting literature, assess case novelty, interpret medical images, and synthesize a coherent scientific narrative. The published reports serve as evaluation references instead of targets for exact textual reproduction.

Validation on real clinical records. We recognize that report-derived atoms provide cleaner inputs than original clinical documentation. To examine whether the system's capabilities extend beyond this controlled setting, we additionally evaluated MedCaseAgent on 30 original multimodal clinical cases from 10 hospitals. Fourteen clinical experts assessed generated reports against the original patient records, alongside clinician-written reports. Reports generated directly from raw records achieved an average quality score of 7.4/10, compared with 7.6/10 for atom-based generation and 7.5/10 for clinician-written reports (Section 4.4, Table 4).

These results provide encouraging evidence of the practical relevance of our benchmark. Nevertheless, we recognize that the most appropriate evaluation paradigm for clinical case-report generation remains an open question. The reviewer's thoughtful observation has prompted us to reflect further on how controlled benchmarks and real-world clinical evaluations can best complement each other, and we sincerely appreciate this valuable perspective.

W2/Q2. Contributions of the agent architecture, backbone, and fine-tuning.

We thank the reviewer for this valuable suggestion. We agree that a clearer separation of the contributions from the agent framework, backbone models, and planner training would strengthen the presentation of our work.

Configuration clarification. MedCaseAgent-Base uses Qwen3.5-9B within our planner–writer–editor framework, while MedCaseAgent-Tuned uses the same backbone with an SFT+GRPO-trained planner. MedCaseAgent-Max employs GPT-5.4 within the same framework. In contrast, standalone model baselines use the corresponding backbone with direct tool access, without the MedCaseAgent workflow.

Contribution of the agent framework. Following the reviewer's suggestion, we compare the same backbone with and without MedCaseAgent on the 453 Table 1 cases, using the same external tools and evaluation protocol.

BackboneWithout MedCaseAgentWith MedCaseAgentGain
Qwen3.5-9B55.9865.18+9.20
Qwen3.5-397B-A17B69.8878.43+8.55
Claude Opus 4.674.6481.48+6.84
Gemini 3.1 Pro75.6781.92+6.25
GPT-5.476.4383.17+6.74

The results show consistent improvements of 5.84–9.20 points across all five backbones. We acknowledge that stronger backbone models substantially improve overall performance. Nevertheless, the consistent gains under matched-backbone comparisons suggest that the MedCaseAgent workflow provides additional benefits beyond backbone capabilities alone.

Contribution of planner training. We further distinguish the effects of SFT and GRPO using the same Qwen3.5-9B backbone on MedCase-Bench. Average report quality improves from 64.43 (Base) to 66.31 (SFT only) and 67.79 (SFT+GRPO), corresponding to gains of +1.88 and +1.48 points, respectively. Novelty recognition similarly increases from 52.40% to 56.51% and 59.80%.

We agree with the reviewer that overall performance is substantially influenced by the capabilities of the underlying backbone models, while planner training provides more modest but consistent improvements. Nevertheless, the gains observed across different backbones suggest that our agentic workflow offers additional value beyond backbone scaling alone. We sincerely appreciate the reviewer's insightful suggestion, which has helped us better distinguish and communicate these contributions.

W3. Source-document retrieval as an evaluation target.

We thank the reviewer for raising this important point. We agree that retrieving the original publication is not equivalent to identifying independent supporting evidence. Our motivation for Exact DOI Retrieval is to provide a verifiable measure of fine-grained clinical literature retrieval, instead of to encourage reconstruction of the source report.

Specifically, identifying a particular case report from a large PubMed corpus requires the agent to formulate discriminative search queries that capture distinctive clinical features, including rare manifestations, diagnostic findings, and patient-specific characteristics. This capability is closely related to retrieving clinically analogous cases and important prior literature, which are essential for assessing case novelty, establishing clinical context, and developing an evidence-grounded discussion. We therefore consider Exact DOI Retrieval a useful auxiliary signal for the literature-search capabilities underlying case-report writing, instead of a direct measure of writing quality.

Importantly, the source DOI is used only for scoring, while its title, abstract, and full text remain masked from the planner. The agent must rely on independently retrieved literature for report generation. Furthermore, the DOI reward is combined with separate objectives for evidence support, clinical coverage, image integration, and plan quality. Our existing ablation (Appendix D, Table 6) shows that removing this reward decreases novelty recognition from 59.08% to 55.78% and report quality from **67.23 to 65.37, suggesting that this retrieval-oriented supervision contributes to the broader writing task.

We will clarify that Exact DOI Retrieval serves as an auxiliary indicator of retrieval specificity, while the quality of generated reports is evaluated separately through citation quality, clinical coherence, and expert assessments.

Q1. Details of the real-world clinical evaluation (Section 4.4).

We thank the reviewer for this suggestion. We used MedCaseAgent-Max (GPT-5.4) with the same planner–writer–editor framework and external tools. The evaluation includes 30 de-identified multimodal clinical cases from 10 hospitals, each with original clinical records and a matched clinician-written report. Fourteen clinical experts participated, with three independently evaluating each case against the original patient records.

We evaluated reports generated directly from raw clinical records and from report-derived clinical atoms, alongside clinician-written reports. The detailed results are as follows:

ReportOverall QualityPatient TimelineOmission Control
MedCaseAgent (raw records)7.47.57.8
MedCaseAgent (clinical atoms)7.67.67.6
Clinician-written7.57.87.9

Reports generated directly from raw clinical records achieved comparable overall quality to clinician-written reports (7.4 vs. 7.5), with similarly close omission-control scores (7.8 vs. 7.9). The paired overall-quality difference was −0.10 (95% CI: −0.36 to 0.16), and generated reports matched or exceeded clinician-written reports in 14 of 30 cases.

These findings suggest that MedCaseAgent's capabilities extend beyond the controlled atom-based benchmark to original clinical documentation. We will expand Section 4.4 to clarify the evaluated configuration, assessment protocol, and detailed results.

Q2. Contribution of the agent framework versus backbone models.

We thank the reviewer for this valuable suggestion. We agree that distinguishing the contributions of the agent framework and the underlying backbone is important.

To better isolate the contribution of our framework, we compare the same backbone with and without MedCaseAgent on all 453 cases in Table 1, using identical external tools and evaluation protocols.

BackboneWithout MedCaseAgentWith MedCaseAgentGain
Qwen3.5-9B55.9865.18+9.20
Qwen3.5-397B-A17B69.8878.43+8.55
Claude Opus 4.674.6481.48+6.84
Gemini 3.1 Pro75.6781.92+6.25
GPT-5.476.4383.17+6.74

The results show consistent improvements of 6.25–9.20 points across all five backbones. We acknowledge that stronger backbone models substantially improve overall performance. Nevertheless, the consistent gains under matched-backbone comparisons demonstrate that the planner–writer–editor workflow provides additional benefits beyond backbone capabilities alone.

Q3. Clinician A/B Test Statistics (Appendix F.3).

We thank the reviewer for pointing out that our original summary was incomplete. We provide the full distribution of correct AI identifications across 32 clinicians, each evaluating five report pairs:

Correct identifications (out of 5)012345
Observed clinicians14101151
Expected under random guessing15101051

Overall, clinicians correctly identified the AI-generated reports in 82 of 160 comparisons (51.3%), with a 95% Wilson confidence interval of 43.6%–58.9%. An exact two-sided binomial test against random guessing (50%) yielded p = 0.813, indicating no statistically significant difference from chance. A clinician-level cluster bootstrap produced a similar 95% confidence interval of 43.8%–58.8%.

We also acknowledge that the original 26/32 statistic alone is insufficient to establish indistinguishability, as the same criterion is satisfied with 81.25% probability under random guessing. We will include the complete distribution and statistical analysis in Appendix F.3.

Minor presentation comments.

We sincerely thank the reviewer for the careful reading and attention to these details. We acknowledge that some aspects of our manuscript could have been presented more carefully and clearly. We greatly appreciate these constructive suggestions and will address all the points raised to improve the clarity, precision, and overall presentation of the paper in the revised version.

We sincerely thank the reviewer again for the thoughtful and detailed feedback, as well as for recognizing the practical value of our work. Your comments have been particularly helpful in clarifying our methodology, strengthening the experimental analysis, and improving the presentation of the paper. We hope our responses and additional results adequately address your concerns. We greatly appreciate your time and consideration, and would welcome any further feedback or suggestions.

Reviewer_UfCVOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present MedCaseAgent, a planner–writer–editor framework for generating clinical case reports from patient notes and medical images. The system combines literature retrieval, image inspection, genetic lookup, and similar-case retrieval with citation and factual-consistency checks. They introduce MedCase-150K and a temporally held-out benchmark, and train a Qwen3.5-9B planner through supervised fine-tuning and GRPO. Experiments include comparisons with tool-equipped models and medical agents, tool ablations, and clinician evaluation using published reports and hospital records.

Critical strengths
  1. The authors develop a practical framework for writing clinical case reports from multimodal evidence. The planner–writer–editor workflow connects patient facts, images, and literature with explicit consistency checks. MedCase-150K and the manually verified temporal benchmark also provide useful resources for studying this task.

  2. The evaluation provides evidence of improved report quality and practical usefulness. Comparisons with shared backbones and tools, planner-training experiments, and tool ablations support the reported gains. Clinician assessments using raw hospital records and blinded comparisons with published reports further support the system’s usefulness.

Critical weaknesses
  1. The evaluation metrics are insufficiently defined and justified. The scoring criteria for citation depth, patient-history nuance, clinical coherence, image integration, and readability in Table 1 and Figure 5 lack detailed scoring anchors. Although the authors describe the LLM judges, score rescaling, and averaging procedure, equal weighting is not justified. This makes it difficult to determine whether higher aggregate scores reflect better clinical quality or mainly better presentation. The clinician calibration also does not directly validate all five dimensions, limiting the interpretation of comparisons with human publications.

  2. The evaluation lacks key baseline comparisons and training ablations. Although a direct GPT-5.4 baseline is included, the medical-agent baselines in Table 1 use Qwen3.5-9B. The authors should also evaluate these existing agent baselines with GPT-5.4 to distinguish MedCaseAgent-Max’s workflow contribution from the benefit of a stronger backbone. Table 2 omits medical-agent systems compared in Table 1. The performance gains from post-training are also limited. Furthermore, the authors do not compare a planner trained with SFT alone against one trained with SFT followed by GRPO, so the additional benefit of GRPO remains unclear.

Questions for authors
  1. Could the authors provide detailed scoring criteria for the five metrics in Table 1 and Figure 5, explain their equal weighting, and clarify how each dimension was validated against clinician judgments?

  2. Could the authors clarify how the disease subsets in Table 1 were selected, including category definitions, publication-date restrictions, and their relationship to the temporally held-out MedCase-Bench?

  3. Could the authors clarify how “related literature” is identified and excluded from Disease-Index RAG, and how duplicate or alternative versions of evaluated source reports are filtered across retrieval tools?

3 comments

We thank the reviewer for a careful review and for recognizing the workflow, the resources and the clinician assessments. Below we address the metric definitions, the baselines and the leakage questions.

W1/Q1. Scoring rubric, equal weighting, and clinician validation.

We thank the reviewer for highlighting the need for clearer evaluation criteria and justification of the scoring methodology. Our evaluation considers five complementary dimensions of clinical case-report quality:

  • Citation Depth: Relevance and evidential support of medical citations.
  • Patient-History Nuance: Preservation of patient-specific details, clinical chronology, and important contextual information.
  • Clinical Coherence: Logical consistency across clinical presentation, diagnostic findings, management, and outcomes.
  • Image Integration: Accuracy of image interpretation and meaningful integration of medical images into the clinical narrative.
  • Readability: Clarity, organization, and academic writing quality.

Each dimension is independently scored on a 0–10 scale by four VLM judges (Qwen3.5-397B-A17B, Gemini 3.1 Pro, Claude Opus 4.6, and GPT-5.4), with scores linearly rescaled to 0–100 for reporting.

Equal weighting and robustness. We adopt equal weighting as a transparent default because the five dimensions capture complementary aspects of report quality, without an established clinical consensus on their relative importance. To examine the robustness of this choice, we conducted weighting sensitivity analyses:

Weighting strategyKendall's τ vs. equal weightsMedCaseAgent-Max − Human Publication
Equal weighting1.00+7.59
Leave-one-dimension-out (worst case)0.93+6.82
Random weights (1,000 Dirichlet samples)0.98 [0.93–1.00]+7.54 [5.86–9.12]
Clinician-elicited weights0.97+7.21

The consistently high ranking correlations indicate that system rankings remain stable across alternative weighting strategies. MedCaseAgent-Max also maintains a substantial advantage over human publications under all tested configurations.

Clinician validation. To directly assess whether the five dimensions reflect clinical judgments, we conducted an additional blinded clinician evaluation covering all five dimensions. Specifically, 15 MedCase-Bench cases were evaluated using three report types: MedCaseAgent-Base, MedCaseAgent-Max, and the corresponding human-authored publications. Three clinicians independently assessed each report. The results are summarized below:

DimensionSpearman ρ, judges vs. clinician meanClinician ICC(2,k)Pairwise order agreement
Citation Depth0.720.8077.8%
Patient-History Nuance0.750.8380.0%
Clinical Coherence0.790.8582.2%
Image Integration0.700.7975.6%
Readability0.810.8684.4%
Five-dimension mean0.820.8884.4%

The results show consistent agreement between the LLM judge ensemble and clinical experts across all five dimensions, with Spearman correlations ranging from 0.70 to 0.81. Clinician inter-rater reliability is also consistently high (ICC = 0.79–0.86), supporting the reliability of the expert assessments.

Furthermore, we examined potential judge-specific bias. As reported in Appendix G.1 (Table 10), MedCaseAgent-Max outperforms human publications under each individual judge, with margins of 7.13–8.63 points. Excluding GPT-5.4, which also serves as the MedCaseAgent-Max backbone, preserves a 7.33-point advantage.

Taken together, the weighting sensitivity analysis, five-dimension clinician validation, and cross-judge consistency provide complementary evidence that our evaluation reflects report quality instead of artifacts of a particular weighting scheme or judge preference.

W2. Baselines and post-training

Medical agents with GPT-5.4. We re-ran all four agents with GPT-5.4 on the Table 1 subsets (average over the five subsets):

MethodQwen3.5-9B (Table 1)GPT-5.4 (new)
MedAgents59.4878.12
MDAgents60.5178.94
MedAgent-Pro63.5180.61
MMedAgent63.2480.22
MedCaseAgent65.1883.17

At a shared GPT-5.4 backbone, MedCaseAgent leads the strongest medical agent by 2.56 points (1.67 at 9B), so MedCaseAgent-Max’s advantage is not only a backbone effect.

Medical-agent baselines on MedCase-Bench. We evaluated the four medical agents from Table 1 on the 584 MedCase-Bench cases using the same Qwen3.5-9B backbone as MedCaseAgent-Base.

MethodReport qualityExact DOI retrieval (%)Novelty recognition (%)Figure sequence accuracy (%)
MedAgents58.9226.0344.5256.16
MDAgents59.9627.2346.2357.88
MedAgent-Pro62.8529.2850.3460.10
MMedAgent62.4128.7749.6659.59
MedCaseAgent-Base64.4330.6052.4061.10

MedCaseAgent-Base outperformed all four baselines on each reported measure, with a 2.06 percentage point lead in novelty recognition over the strongest baseline, MedAgent-Pro.

Post-training. We agree the average gain is modest: +3.36 (64.43 to 67.79). It is concentrated in what the planner controls: novelty recognition rises from 52.40% to 59.80%, citation depth by 4.10 points and image integration by 4.54. In the blinded LLM A/B test on the Table 1 cases, preference over the published report rises from 36.29% to 43.87% (App. Table 9). These gains come from only 3,000 trajectories and 500 GRPO steps, which suggests that MedCase-150K provides a useful training signal.

SFT only vs. SFT + GRPO:

Planner (Qwen3.5-9B)Citation depthPatient-history nuanceClinical coherenceImage integrationNovelty rec. (%)Average
No training (Base)59.0664.9866.4160.0052.4064.43
SFT only, 3,000 teacher trajectories61.4266.5767.8162.6856.5166.31
SFT + GRPO (Tuned)63.1667.8368.9764.5459.8067.79

SFT improves average report quality by 1.88 points, with GRPO adding a further 1.48 points. Novelty recognition also increases progressively, from 52.40% to 56.51% after SFT and 59.80% after GRPO.

Q2. Disease subsets in Table 1

Table 1 provides a disease-stratified evaluation using randomly sampled cases from MedCase-150K. The five categories are COVID-19, small-cell lung cancer, Erdheim–Chester disease (ECD), rare genetic disorders, and ultra-rare diseases. The first three correspond to specific disease diagnoses, while the latter two cover broader groups of genetically defined and epidemiologically rare conditions. We randomly sampled 100 cases from each category, except ECD, for which all 53 available cases were included.

No additional category-specific publication-date restrictions or performance-based case selection were applied. The purpose of this evaluation is to examine model performance across different disease categories, instead of constructing another temporally controlled benchmark.

MedCase-Bench serves a different purpose: it is a manually verified temporal holdout comprising 584 case reports published after January 2026. This cutoff helps reduce potential pretraining contamination, as recently published reports are less likely to have been included in the pretraining data of evaluated LLMs.

The disease-specific subsets in Table 1 are separate from MedCase-Bench and the planner-training split. Thus, Table 1 provides disease-stratified evaluation, whereas MedCase-Bench serves as our primary temporally held-out benchmark.

Q3. Excluding related literature and duplicates.

We thank the reviewer for raising this important question regarding potential information leakage. Here, "related literature" refers to records directly associated with an evaluated source report, including PubMed-linked corrections, comments, errata, and follow-up reports describing the same patient, instead of general literature concerning the same disease.

To prevent leakage, we exclude the evaluated source and its associated records from Disease-Index RAG using PMCID, PMID, and DOI identifiers, while filtering duplicate or alternative versions of the same publication. Disease-Index RAG also retrieves only prior cases published before the evaluated report, preventing later publications from revealing case-specific information.

For PubMed retrieval, the evaluated source may still be identified for Exact DOI Retrieval scoring, but its title, abstract, and full text are masked from planner-visible results. Thus, the agent cannot directly access the original report during generation, while remaining able to retrieve independent supporting medical literature.

We sincerely thank the reviewer again for the thoughtful and constructive feedback. We hope these clarifications and additional results adequately address the concerns and strengthen the evaluation of our work. We would be happy to address any further questions or suggestions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.