- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces , a reinforcement learning framework for multimodal differential diagnosis (DDx) reranking. A single shared policy simulates an entire multidisciplinary consultation across four stages: initial triage, specialist selection, parallel specialist consultation, and final evidence-based revision. To resolve coarse credit assignment and inefficient rollout allocation, the authors propose: (1) Modular Stage Training (MST), which co-trains full trajectories with isolated stage tasks using revision-aware credit (rewarding successful rescue and penalizing corruption); and (2) Mastery-Driven Sampling (MDS), which prioritizes tasks at intermediate mastery via an EMA-tracked Gaussian kernel with a positive floor. The authors also construct AcuteM$^2$DDx (4,225 cases / 22,208 images), featuring an independently collected, leakage-free temporal holdout test set. Across six benchmarks, fine-tuned Qwen3.5-4B and 9B variants achieve state-of-the-art results (up to 62.93% Hit@1 / 77.12% MRR), outperforming frontier commercial MLLMs, larger open models (27B/32B), and prompted multi-agent baselines.
Critical strengths
-
AcuteM2DDx is a substantial and thoughtfully constructed resource. The benchmark contains 4,225 cases and 22,208 images across 14 modalities, with 537 post-March-2026 temporal-holdout cases, explicit diagnostic-timepoint control, leakage-aware SFT/RL partitioning, MONDO-based concept normalization and alias handling, and dual-model verification with selective physician review.
-
The empirical study is broad and internally consistent. Two backbones × six benchmarks, consistent component ablations (Outcome-only 57.10 → +MST 61.23 → +MST+MDS 63.00 → +SFT 65.51), and a good 4B result (60.89 macro Hit@1) that exceeds 27B/32B general and medical models.
Critical weaknesses
W1. Unrealistic Closed-Set Setup and Risk of Synthetic Artifacts
- Clinical gap: Real-world emergency differential diagnosis (DDx) is inherently open-ended; assuming a predefined five-candidate set severely oversimplifies clinical decision-making.
- Distractor artifacts: Distractors were generated by LLMs (Gemini/GPT) and mixed with true labels from clinical reports. Models may achieve high accuracy merely by exploiting stylistic, linguistic, or granularity differences between AI-generated distractors and human-written gold-standard diagnoses, rather than by performing genuine clinical reasoning.
W2. Lack of Statistical Significance and Robustness Evaluation
- All experimental results (Tables 1 and 2) report only point estimates. Given the inherent randomness of RL exploration and LLM generation, the paper lacks error bars, standard deviations across multiple random seeds, or 95% confidence intervals (CIs) to confirm statistical significance and rule out random variance.
W3. Inconsistencies and Editorial Oversights
- Inconsistent number of specialty roles: The main text specifies a pool of “12 specialty roles” (line 369), but the consultation prompt in Appendix F.2 explicitly lists 13 distinct roles.
- Typographical errors:
- Misspelling in the Figure 1 caption: “commmercial models” (line 046).
- Section 4.1 contains a duplicated heading: “Implementation Details. Implementation Details.” (line 295).
Questions for authors
Q1: Can the authors respond to the concerns raised in W1 and W2? Specifically, can they provide empirical evidence to mitigate these concerns?
Q2: Is Team Collaboration Really Necessary?
To rigorously distinguish whether the performance gains stem from the multidisciplinary team structure or merely from increased inference compute or data curation, can the authors provide the following two single-agent baselines:
-
Token-Matched Self-Consistency: Measure the average test-time token budget consumed by a full MDT consultation, and evaluate a single-agent baseline using self-consistency methods (e.g., majority voting), scaled to roughly match the above inference token cost.
-
Symmetrically Trained Single Agent (SFT + RL): Train a single-agent model under the same data recipe:
- SFT: Fine-tune on high-quality single-physician diagnostic reasoning trajectories distilled from the 27B model series.
- RL: Use outcome-based reinforcement learning with a binary (0/1) reward based on Top-1 diagnostic correctness.
Response to MJuB — candidate artifacts, uncertainty, and single-physician controls
W1 and Q1. Clinical scope and candidate artifacts. Our task is to rerank a supplied differential diagnosis list. Open-ended generation of that list is a separate clinical task. To test your concern about the candidates themselves, we removed the history, images, and image identifiers from 1,461 cases in AcuteM²DDx, MedReaMM, and MRareBench. Each case had one source-derived reference diagnosis and four model-generated distractors. The ranking prompts and scoring rules were retained.
Candidate-only macro Hit@1 was 22.81% and 19.66% for the direct 4B and 9B backbones, and 57.69% and 56.69% for the corresponding follow-up MDT models. Random choice gives 20%. On MedReaMM, simply choosing the longest candidate achieved 60.59% Hit@1. Restoring the clinical inputs to the same trained checkpoints raised their three-set macro Hit@1 to 64.16% and 65.02%, with improvements for both models on every dataset. The candidate lists therefore contain predictive signals. Restoring the history and images together adds diagnostic information. This experiment does not distinguish AI authorship, diagnostic specificity, ordering, or other candidate features as sources of the candidate-only signal (Conclusion and Appendix I.7, Table 15).
W2 and Q1. Uncertainty. We used 20,000 within-dataset case-bootstrap replicates, weighting datasets equally and resampling matched cases together for comparisons. Appendix I.4 reports 95% Hit@1 intervals for every row of Tables 1 and 2, and MRR@5 intervals wherever the required ranks were retained. Six-set Hit@1 was 62.93% [61.17, 64.71] for 9B and 60.89% [59.09, 62.66] for 4B. These intervals measure test-case uncertainty. The paired Hit@1 differences for MDS and the roster reward were +1.77 [−0.05, 3.56] and +0.49 [−1.32, 2.28] points. An independent rerun of the 9B backbone gave six-set MRR@5 of 62.91% [61.59, 64.23], compared with the original 63.01%.
We also repeated the revision-credit ablation. Its five-set Hit@1 effect was +2.68 points in the original pair and −1.75 in the repeat. The two pairs used different GPU configurations. MST and MDS still have one training run per configuration (Appendix I.5).
Q2. Token-matched single-physician self-consistency. We retrained Table 2's Single-Agent RL baseline for 100 updates from base Qwen3.5-9B without SFT. Using its original single-physician prompt with full histories and images, we drew four to six samples per case at temperature 0.7 against each dataset's measured MDT token budget. Valid draws voted for their first choice, with ties broken by summed ranks. The comparator is the main 9B step-140 MDT. Scores below are percentages.
Voting consumed 23.19M input plus output tokens and MDT consumed 23.73M. The paired MDT advantages were 7.07 Hit@1 points [4.64, 9.52] and 5.41 MRR@5 points [3.86, 6.97] (Appendix I.6, Table 14). Training also differs between these systems. For your second control, the 27B teachers produced eligible single-physician demonstrations for 2,731 of the original 3,019 SFT cases. The single-physician SFT plus binary top-1 RL pipeline has no completed evaluation to report.
W3. Editorial corrections. The main text and the consultation prompt in Appendix F.2 now list the same 12 specialist roles. The ED attending is a separate role. The Figure 1 caption now reads “commercial models”, and Section 4.1 has a single “Implementation Details” heading.