- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents CASE, an agentic framework for longitudinal medical reasoning that enables vision-language models to actively acquire and synthesize clinical evidence. The authors construct a novel benchmark based on UK Biobank, comprising 50,401 questions from 4,739 participants, with access to longitudinal MRI, anatomical measurements, and generated medical reports. The proposed training pipeline combines rejection-sampling fine-tuning (RSFT) with agentic reinforcement learning, incorporating rubric-based AI feedback (RLAIF) and on-policy privileged self-distillation (OPSD). Extensive experiments demonstrate substantial improvements over the base VLMs and several frontier multimodal models.
Critical strengths
1. Meaningful problem formulation and benchmark design: The paper addresses an important limitation of conventional medical QA by explicitly modeling evidence acquisition as part of the reasoning process. The longitudinal multimodal environment, heterogeneous evidence sources, and role-conditioned tool interactions provide a valuable setting for studying autonomous evidence-seeking policies beyond fixed-context question answering.
2. Strong empirical results with extensive evaluations: CASE achieves substantial improvements over both its RSFT initialization and standard GRPO training. The experiments include comparisons against multiple medical and general-purpose models, role-specific evaluations, and component-level ablations. In particular, the sensitivity experiments in Table 15 demonstrate the importance of adaptive information acquisition, strengthening the empirical motivation for the proposed framework.
Critical weaknesses
1. Insufficient Validation of Evidence Grounding for Clinical Conclusions
Although the paper distinguishes label agreement from evidence sufficiency (Table 4), its evaluation still relies mainly on diagnosis-label accuracy and text-only LLM judgments, which cannot verify whether acquired multimodal evidence genuinely supports the predicted diagnoses. Evidence sensitivity experiments show that retrieved information matters, but not that evidence-to-answer reasoning is clinically valid, especially for conditions not directly identifiable from imaging. The gains may therefore reflect correlated clinical cues or imperfect generated reports. An independent grounding evaluation on a representative subset would better support the paper’s central claim.
2. Limited OPSD Validation
Appendix D clarifies how OPSD differs from existing methods, and Table 2 shows its effectiveness, but the observed gains remain insufficiently attributed to answer-privileged conditioning. Appendix F proposes relevant controls (e.g., neutral hints and shuffled answers), but their quantitative results are not reported. Furthermore, the experiments do not clearly distinguish improvements in evidence acquisition from those in synthesis stability and repetition suppression. These analyses would help substantiate the proposed OPSD mechanism.
Questions for authors
Q1. Evidence-Grounding Assessment
How do the authors validate whether the acquired evidence genuinely supports the predicted diagnoses, beyond label agreement? An independent grounding assessment, as discussed in Table 4, would help substantiate the evidence-grounded reasoning claims. Quantitative results across different clinical task categories would help validate the evidence-grounding claims.
Q2. OPSD Mechanism
To better understand the effectiveness of OPSD, could the authors report the neutral-hint, shuffled-answer, and evidence-prefix swapping analyses proposed in Appendix F? These results would help clarify the role of answer-privileged conditioning and distinguish improvements in evidence acquisition from those in synthesis quality.
Q3. Training and Evaluation Consistency
The manuscript reports different trajectory counts and notes differences in Consultant tool configurations across training stages. Could the authors clarify how the final training samples were selected and how Consultant's tool access was handled across SFT and RL? Additionally, was skill memory enabled in the main results (Tables 1 and 12), or only in the separate variant reported in Table 14?