acceptodds
Withdrawn from ICLR 2027

UroMed-Agents: An Interactive Multi-Agent Framework for Clinical Decision-Making Training in Urology

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Effective and standardized clinical decision-making (CDM) training is essential for preparing clinicians to diagnose and treat urological diseases. However, traditional approaches are constrained by limited access to longitudinal cases and expert supervision, making it difficult to cover traceable decision-making processes. Recent advances in large language models (LLMs) enable interactive medical environments, but most methods either evaluate autonomous artificial intelligence (AI) clinicians or focus on isolated history taking tasks, providing limited support for end-to-end, multi-turn training of human clinicians. To address this, we propose UroMed-Agents, an interactive multi-agent framework for CDM training in urology. The framework centers on human clinicians and employs four agents for patient interaction, examination result delivery, process guidance, and final evaluation. To reduce role overreach and information leakage across turns, we introduce a stateful Case Engine that maintains interaction state, enforces information boundaries for each role, and releases examination results only when requested. Furthermore, the framework separates training and testing scenarios and evaluates clinician performance based on interaction records and final reports, while diagnostic performance at the system level is assessed using metrics based on the International Classification of Diseases (ICD-10). Experimental results on 239 real longitudinal urology cases illustrate the effectiveness of our method, which achieves better performance than existing interactive approaches.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
Multi-Agent Systems, Urology, Medical Education, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_v5HZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes UroMed-Agents, a human-centered multi-agent framework for clinical decision-making training in urology. The system coordinates four agents (Patient, Examiner, Preceptor, Evaluator) through a stateful Case Engine that maintains dialogue state, controls examination evidence release, and enforces role-specific information boundaries. The key design insight is separating training (with Preceptor guidance) from testing (independent decision-making), evaluated via both CDM scores and ICD-10 diagnostic metrics. Experiments cover agent reliability, a longitudinal student study, comparisons with prompting and multi-agent baselines, and ablations.

Critical strengths
  1. The end-to-end framing of clinical decision-making is well-motivated. The explicit training/testing separation is a reasonable design decision.

  2. The formal treatment of examination gating is motivated by real failure modes, and the ablation showing the Case Engine contributes more than role initialization.

Critical weaknesses
  1. My central concern is the circularity problem that runs through most of the key evaluations. The Preceptor generates feedback, the Evaluator scores outputs, the rubrics are case-specific (and presumably generated or refined with LLM assistance), and the report-generation module used in the comparison experiment also uses GPT-5.4. When the same model family participates in both producing the training signal and assigning the scores, it becomes genuinely difficult to tell whether the reported gains reflect the students learning clinical reasoning or the system generating outputs that score well under a stylistically compatible evaluator. The paper mentions a 49-case calibration against expert consensus, but reports only aggregate agreement figures. I'd want to see per-dimension ICC values, the number of annotations per case, and whether the expert annotators scored blind to condition.

  2. The statistical treatment of the longitudinal study is the paper's most serious methodological weakness. Five students per cell is a very small sample even for an exploratory study, and the choice to run a participant-level Wilcoxon rank-sum test treats each student's entire six-round trajectory as a single data point, discarding essentially all of the longitudinal structure. I'd also want effect sizes and confidence intervals; given the sample size, a p < 0.05 result could correspond to a substantively small effect. It's also unclear whether total time-on-task and the number of Preceptor interventions received were matched across conditions, which matters a lot for interpreting the observed differences.

  3. Something in Table 1 doesn't add up. Within the GPT-5.4 row, Patient RF = 95.5 and Evaluator HA = 95.5 are identical. I notice suspiciously frequent near-identical or exactly identical values across different metrics and agents within the same backbone row. Eight conceptually distinct metrics (role fidelity, memory consistency, result specificity, request matching, intervention accuracy, non-overreach, rubric alignment, human agreement) across four different agents should not routinely produce the same numbers.

  4. The ICD-10 single-canonical-code-per-case design is a significant simplification that the paper doesn't adequately justify. In urology, patients frequently present with comorbid conditions (e.g., nephrolithiasis with secondary infection, bladder tumor with chronic retention), and a single-label evaluation scheme will systematically penalize or reward models in ways that don't map onto actual clinical diagnostic reasoning.

Questions for authors
  1. On Table 1: Please provide the raw counts underlying each metric (cases? turns? manually annotated behavior units?), the denominator definition for each of the eight metrics, and a direct explanation of why exact or near-exact numerical coincidences appear so frequently across different metrics within the same backbone.

  2. Did the Evaluator agent have access to the case-specific rubric when scoring student reports? Were those rubrics written by urologists or generated with LLM assistance? For the expert calibration on the 49-case subset, please report per-dimension ICC and MAE, the number of cases annotated per expert, and whether annotators were blind to training condition and the source of the report.

  3. Please reanalyze the training data using a repeated-measures mixed-effects model that treats student and case as random effects. Provide exact p-values, effect sizes (e.g., Cohen's d at the participant level), and 95% confidence intervals. Also clarify whether time-on-task and number of Preceptor interventions were tracked.

  4. What fraction of the 239 cases had clinically meaningful secondary diagnoses that were collapsed under the single-canonical-code scheme? How were borderline cases handled during the ICD-10 mapping? Given that the comparison experiment reports single-run P/R/F1 on 49 cases without variance estimates, it's hard to interpret differences of 1–3 points between methods.

Reviewer_brQ3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents UroMed-Agents, a framework for clinical decision-making training in urology. It combines Patient, Examiner, Preceptor, and Evaluator agents with a stateful CaseEngine that controls information access and evidence release. Evaluation includes 239 real cases, agent reliability assessments, system comparisons and ablations, and a randomized study involving 30 medical students with delayed testing. The results suggest benefits over static case-based training.

Critical strengths
  • The framework addresses a meaningful educational need. Integrating history taking, examination requests, guidance, and assessment into a complete training workflow offers practical value. Explicit role boundaries and controlled evidence disclosure are well aligned with sequential clinical reasoning.

  • The evaluation includes actual learners. Stratified randomization, separate training and testing cases, and a two-week retention assessment provide preliminary evidence of educational effectiveness beyond system-level diagnostic performance (Table 2).

Critical weaknesses
  • The link between the technical mechanisms and educational gains remains insufficiently established. The static-case control lacks both interactive information gathering and process guidance, making their contributions inseparable from those of the CaseEngine. The ablations evaluate only system-level diagnostic F1 and therefore do not establish additional learning benefits from the core technical design (Table 4).

  • Independent validation of the primary educational outcome is limited. Student performance is scored by a frozen GPT evaluator. Although agreement with experts is reported, this validation uses scripted system-test records; blinded expert assessment of actual student responses is not reported. This leaves uncertainty about the reliability of the measured learning gains.

Questions for authors

The following questions seek clarification of existing work and could materially affect the assessment:

  • Were training time, feedback exposure, and testing interfaces matched across conditions? Do existing controls or analyses distinguish the contributions of interactive guidance and the CaseEngine?

  • Were actual student test records independently scored by blinded experts? If so, do human scores reproduce the between-group differences in Table 2?

  • Could Appendices A–G, which are referenced but absent from the provided PDF, be supplied to verify implementation details, scoring rubrics, and baseline configurations?

Reviewer_EnQ1OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UroMed-Agents provides an interactive environment for urology training through patient, examiner, preceptor, and evaluator agents. A stateful Case Engine controls information access and examination-result release. The study uses 239 clinical cases, a randomized training comparison involving 30 fourth-year medical students, and a separate system evaluation on 49 cases.

Critical strengths
  • The paper evaluates actual learners over multiple training rounds and includes a delayed retention test. Randomization within prior-knowledge strata and participant-level statistical tests strengthen this evaluation.
Critical weaknesses
  • The training comparison changes several factors together: interaction, information order, process guidance, and feedback. It supports a benefit of the complete teaching package, but does not establish the educational contribution of the Case Engine. The system-level ablations measure diagnostic outputs rather than student learning, so they do not resolve this attribution problem.
  • Evidence of improved decision-making remains tied to the study's assessment environment. Expert agreement on the 49 scripted system cases is useful, but does not establish unbiased scoring of actual student responses across training arms. The separate diagnostic comparison uses a fixed clinician script and a shared report generator. It therefore provides limited additional evidence of transfer to independently assessed clinical reasoning.
Questions for authors
  • Is there a comparison that preserves interactive practice and feedback while removing the Case Engine's information controls? If so, what changes in learner performance?
  • Were actual student responses independently scored by clinicians blinded to training condition? Please distinguish this validation from the expert agreement reported for scripted system cases.
  • Can you provide participant-level effect sizes and confidence intervals, together with training exposure time and any assessment obtained before the first training round?
  • How were immediate and delayed test cases matched for difficulty, and were any outcomes assessed outside the simulator's own scoring procedure?
Reviewer_rtcdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UroMed-Agents, a clinical decision-making training framework for urology. It employs four types of intelligent agents and a stateful case engine to manage information disclosure, provide guidance, and assign scores. Based on 239 clinical cases, a study involving 30 medical students, and experiments across multiple models, the authors report on training efficacy, long-term retention, and improvements in diagnostic performance.

Critical strengths
  1. The application has a clear objective. By establishing a closed-loop process—encompassing patient inquiry, clinical examination, guidance, and assessment—and distinguishing between guided training and independent testing, the framework offers genuine pedagogical value.
  2. The system design is comprehensive. It clearly defines role-based information boundaries, state updates, and evidence-gating mechanisms. The inclusion of prompts, pseudocode, and scoring rubrics in the appendix facilitates a clear understanding of the implementation.
  3. The evaluation is multi-faceted. It goes beyond simple case demonstrations by incorporating expert case reviews, verification of agent reliability, a stratified randomized educational experiment, delayed post-testing, and cross-model ablation studies.
Critical weaknesses
  1. The process for generating diagnoses in the experiments is unclear. The agents in the scripts do not provide diagnoses, and the reporting module is restricted from adding new ones. Please clarify which component is responsible for generating the final diagnostic predictions in both the system comparisons and the ablation studies.
  2. The paper emphasizes clinical decision-making training for humans. However, the pedagogical experiment in Section 4.3 compares the proposed approach only against static case-based learning. Is it possible to conduct a comparison—using equivalent learning time and resources—against existing interactive educational systems, with student performance evaluated by independent, blinded experts?
  3. Although the paper describes role filtering and evidence-checking gating, the "basic information" and "history of present illness" sections in Figure 5 still contain information regarding disease categories and confirmed diagnoses. Please explain how these fields are handled when constructing the actual model inputs, and provide an example of a fully populated "Patient" input.
Questions for authors
  1. Diagnostic prediction: Which component generates the diagnostic predictions in the comparison and ablation experiments, given that Table 8 contains no explicit diagnosis-submission step and Figure 11 prohibits adding diagnoses not stated by the clinician? Please provide one end-to-end example.
  2. Educational evaluation: Were learning time and resources comparable across the training conditions? Beyond the static-case comparison and Evaluator validation on system test cases, is there additional evidence from an interactive educational baseline or independent, blinded expert assessment of actual student outcomes?
  3. Information filtering: How are disease labels, examination findings, and confirmed diagnoses embedded in Figure 5’s “Basic Information” and “History of Present Illness” handled before reaching the Patient Agent? Please provide a fully instantiated, post-filtering input for this case.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.