- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes UroMed-Agents, a human-centered multi-agent framework for clinical decision-making training in urology. The system coordinates four agents (Patient, Examiner, Preceptor, Evaluator) through a stateful Case Engine that maintains dialogue state, controls examination evidence release, and enforces role-specific information boundaries. The key design insight is separating training (with Preceptor guidance) from testing (independent decision-making), evaluated via both CDM scores and ICD-10 diagnostic metrics. Experiments cover agent reliability, a longitudinal student study, comparisons with prompting and multi-agent baselines, and ablations.
Critical strengths
-
The end-to-end framing of clinical decision-making is well-motivated. The explicit training/testing separation is a reasonable design decision.
-
The formal treatment of examination gating is motivated by real failure modes, and the ablation showing the Case Engine contributes more than role initialization.
Critical weaknesses
-
My central concern is the circularity problem that runs through most of the key evaluations. The Preceptor generates feedback, the Evaluator scores outputs, the rubrics are case-specific (and presumably generated or refined with LLM assistance), and the report-generation module used in the comparison experiment also uses GPT-5.4. When the same model family participates in both producing the training signal and assigning the scores, it becomes genuinely difficult to tell whether the reported gains reflect the students learning clinical reasoning or the system generating outputs that score well under a stylistically compatible evaluator. The paper mentions a 49-case calibration against expert consensus, but reports only aggregate agreement figures. I'd want to see per-dimension ICC values, the number of annotations per case, and whether the expert annotators scored blind to condition.
-
The statistical treatment of the longitudinal study is the paper's most serious methodological weakness. Five students per cell is a very small sample even for an exploratory study, and the choice to run a participant-level Wilcoxon rank-sum test treats each student's entire six-round trajectory as a single data point, discarding essentially all of the longitudinal structure. I'd also want effect sizes and confidence intervals; given the sample size, a p < 0.05 result could correspond to a substantively small effect. It's also unclear whether total time-on-task and the number of Preceptor interventions received were matched across conditions, which matters a lot for interpreting the observed differences.
-
Something in Table 1 doesn't add up. Within the GPT-5.4 row, Patient RF = 95.5 and Evaluator HA = 95.5 are identical. I notice suspiciously frequent near-identical or exactly identical values across different metrics and agents within the same backbone row. Eight conceptually distinct metrics (role fidelity, memory consistency, result specificity, request matching, intervention accuracy, non-overreach, rubric alignment, human agreement) across four different agents should not routinely produce the same numbers.
-
The ICD-10 single-canonical-code-per-case design is a significant simplification that the paper doesn't adequately justify. In urology, patients frequently present with comorbid conditions (e.g., nephrolithiasis with secondary infection, bladder tumor with chronic retention), and a single-label evaluation scheme will systematically penalize or reward models in ways that don't map onto actual clinical diagnostic reasoning.
Questions for authors
-
On Table 1: Please provide the raw counts underlying each metric (cases? turns? manually annotated behavior units?), the denominator definition for each of the eight metrics, and a direct explanation of why exact or near-exact numerical coincidences appear so frequently across different metrics within the same backbone.
-
Did the Evaluator agent have access to the case-specific rubric when scoring student reports? Were those rubrics written by urologists or generated with LLM assistance? For the expert calibration on the 49-case subset, please report per-dimension ICC and MAE, the number of cases annotated per expert, and whether annotators were blind to training condition and the source of the report.
-
Please reanalyze the training data using a repeated-measures mixed-effects model that treats student and case as random effects. Provide exact p-values, effect sizes (e.g., Cohen's d at the participant level), and 95% confidence intervals. Also clarify whether time-on-task and number of Preceptor interventions were tracked.
-
What fraction of the 239 cases had clinically meaningful secondary diagnoses that were collapsed under the single-canonical-code scheme? How were borderline cases handled during the ICD-10 mapping? Given that the comparison experiment reports single-run P/R/F1 on 49 cases without variance estimates, it's hard to interpret differences of 1–3 points between methods.