- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MedCaseAgent, a tool-augmented framework for generating clinical case reports from fragmented clinical facts and medical images. It proposes a planner–writer–editor workflow with literature retrieval, image inspection, gene lookup, and similar-case retrieval. The authors introduce MedCase-150K and a 584-case temporal evaluation benchmark, train a 9B planner, and evaluate the system.
Critical strengths
-
evidence-aware workflow. The framework explicitly separates patient-specific facts from external literature and plans citation and figure placement before drafting, addressing concrete challenges in producing coherent, well-supported case reports.
-
Task-specific resources and complementary evaluation. The large case-report corpus and dedicated benchmark support research on an underexplored writing task. Backbone comparisons, tool ablations, and evaluation on original hospital records provide complementary evidence about the system’s capabilities.
Critical weaknesses
-
unreal tasks. The main quantitative results are based on simulated inputs derived from published case reports. Validation on real clinical records is limited in scale, leaving the gap between extracted clinical atoms and real-world clinical inputs insufficiently assessed and limiting the study’s generalizability.
-
No comparisons between generated and reference reports. The paper does not provide statistical comparisons between generated and human-authored reference reports, such as whether the generated reports are longer or contain more references. These factors can directly influence the preferences of LLM judges. Furthermore, although the paper requires generated reports to include at least 10 verified references, it does not compare evaluation scores with and without controlling for reference count. These ambiguities in the evaluation limit confidence in the paper’s claims.
Questions for authors
N/A
5 comments
We agree that prospective validation and a broader collection of hospital cases would further strengthen the evidence. However, assembling matched longitudinal multimodal records and clinician-reviewed reports for rare and clinically complex cases at scale requires considerable clinical coordination and expert involvement. We are continuing to collaborate with clinical partners to expand this evaluation. We hope that the combination of large-scale benchmark evaluation and expert assessment on original hospital records provides useful evidence of the potential of MedCaseAgent for clinical case-report generation.
W2. Length and reference count as potential confounders.
We thank the reviewer for highlighting this important concern. We agree that manuscript length and reference count may influence LLM-based quality assessments. To better understand their effects, we examined the corresponding distributions and conducted additional controlled comparisons on the Table 1 cases, where generated reports and human publications are evaluated using the same rubric.
| Report | Words, mean ± SD | References, median [IQR] |
|---|---|---|
| Human Publication | 1,477 ± 777 | 14 [10–20] |
| GPT-5.4 (without MedCaseAgent) | 1,648 ± 730 | 13 [10–17] |
| MedCaseAgent-Base (Qwen3.5-9B) | 1,545 ± 716 | 12 [10–16] |
| MedCaseAgent-Tuned (Qwen3.5-9B, SFT+GRPO) | 1,567 ± 784 | 13 [11–17] |
| MedCaseAgent-Max (GPT-5.4) | 1580 ± 697 | 16 [12–21] |
To assess whether these differences account for the observed performance advantage, we further examined several settings:
| Evaluation setting | MedCaseAgent-Max − Human Publication |
|---|---|
| Original evaluation (Table 1) | +7.59 |
| Regression-adjusted for log word count and reference count | +5.84 |
| References capped at the published report's count, then re-evaluated | +6.18 |
| Generated without the 10-reference minimum | +6.43 |
| Length-matched subset (±15% words, n = 192) | +5.67 |
MedCaseAgent-Max maintains a consistent advantage across these settings, suggesting that the observed improvements cannot be explained solely by manuscript length or reference count.
We would also like to clarify that the minimum requirement of 10 references is intended to encourage adequate literature grounding instead of maximizing citation quantity. The median number of references in MedCase-150K publications is 14, placing this threshold below typical case-report citation counts.
Furthermore, to complement LLM-based assessment, we conducted an additional blinded clinician evaluation covering the same five quality dimensions. Across 15 cases, three clinicians independently evaluated each report, including MedCaseAgent-Base, MedCaseAgent-Max, and human-authored publications. The correlations between LLM-judge scores and clinician assessments ranged from 0.70 to 0.81 across individual dimensions, with an overall Spearman correlation of 0.82 for the five-dimension average.
Together with our hospital-based evaluations, where clinicians assess generated reports directly against original patient records, these findings provide complementary support for the reliability of our report-quality evaluation.
We sincerely appreciate the reviewer's thoughtful suggestions, which have helped us strengthen the evaluation and clarify the limitations of our study. We hope these additional analyses adequately address the concerns, and we welcome any further feedback.
We sincerely thank the reviewer for the careful review and constructive feedback, as well as for recognizing the value of our evidence-aware framework and complementary evaluations. We address the concerns below.
W1. Realism of the task and scale of real-world validation.
We thank the reviewer for raising this important concern regarding the relationship between report-derived clinical atoms and original clinical records. We would like to further clarify the motivation behind our benchmark design and the complementary role of our hospital-based evaluation.
Benchmark design and motivation. Our goal is to establish a scalable and controlled benchmark for clinical case-report generation. Clinical atoms provide a standardized representation of patient-specific evidence, retaining key clinical observations while removing background literature and reference-driven claims that the agent must independently reconstruct. This representation enables systematic comparisons across methods under consistent input conditions. We acknowledge that atoms are not equivalent to complete EHR records; rather, they offer a practical basis for large-scale evaluation of the reasoning, literature retrieval, novelty assessment, and multimodal synthesis involved in case-report writing. Collecting matched longitudinal multimodal hospital records and corresponding clinician-written reports at a comparable scale remains challenging due to privacy requirements, institutional governance, cross-hospital coordination, and the substantial involvement of clinical experts.
Clinical relevance of novelty recognition. We would also like to emphasize that clinical case-report generation involves more than reorganizing patient records. An important purpose of case reports is to identify and communicate rare diseases, unprecedented findings, and atypical clinical presentations, thereby contributing potentially valuable observations to the broader medical community. Such reports can help clinicians recognize similar presentations, investigate potential mechanisms, and improve understanding of rare conditions. Accordingly, novelty recognition through literature retrieval and comparison is an important component of this task. MedCaseAgent explicitly addresses this aspect and achieves 88.35% novelty recognition on MedCase-Bench (Table 2), alongside evidence-grounded report generation.
Validation on original hospital records. To complement the large-scale benchmark, we conducted a hospital-based evaluation involving 30 de-identified multimodal clinical cases from 10 hospitals. These cases involve rare or clinically complex presentations and include original longitudinal records and corresponding clinician-prepared reports. Fourteen clinical experts participated, with three independently assessing each case against the original records. For each patient, we generated reports using both original clinical records and report-derived atoms, alongside the clinician-written report. The results are summarized below:
The two input settings yield closely comparable expert scores, with differences of at most 0.2 points across the three dimensions. Reports generated directly from original records also achieve comparable overall quality to clinician-written reports (7.4 vs. 7.5). In the source-grounded assessment, experts identified no hallucinated content across the 30 cases and omitted patient facts in only one case. These findings provide encouraging initial evidence of the system's applicability to original clinical documentation.
We recognize that report-derived atoms may condense certain aspects of the original records. As illustrated by the same-patient comparison in Appendix F.1 (Table 7), atoms retain key clinical observations while compressing some temporal context, source provenance, and conflicting information. Processing original records therefore involves additional challenges in document filtering, evidence reconciliation, and timeline reconstruction. These differences motivate our complementary evaluation design: the large-scale atom-based benchmark enables controlled and reproducible comparisons, while the hospital-based evaluation examines system performance directly on original clinical evidence.