- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper treats multi-agent medical QA as a learned consultation policy. A frozen LMM proposes 5–7 case-specific specialist roles; a trained router sequentially selects either an unconsulted specialist or a stop action, conditioned on a pooled embedding of the accumulated diagnostic record; a Moderator synthesizes the record into the final answer. The router is trained with a group-normalized policy gradient on a binary LLM-judge reward (multiplied by a length decay γ^T), using 300 train questions × 16 rollouts per dataset. It is evaluated on three text-only and four image-text benchmarks against single-model and adapted multi-agent baselines sharing a Qwen3-8B / Qwen3.5-9B backbone, with gains up to +14.1 (MedQA) and +34.3 (BT-MRI) percentage points.
Critical strengths
The paper identifies a real, previously unaddressed problem: the order of agent consultation.
Prior medical multi-agent work fixes the roles and the coordination protocol. MedAgents runs a round table, MDAgents adapts a discussion structure, MAM follows a fixed GP → Specialist → Director template. In all of them which specialist is consulted, in what order, and when to stop is a byproduct of prompt design or a hand-written workflow, never a decision that is made and evaluated. This paper proposes MedRoute and makes it explicit.
Critical weaknesses
W1. Substantially Under-Configured Single-Model Baselines
- Suspect Baseline Scores: Qwen3-8B is reported at 58.4% on MedQA. Independent literature reports 74.87% [1] and 79.4% [2] on MedQA. If the Qwen3-8B's true performance is ~75%, MedRoute’s 72.5% actually underperforms the vanilla base model.
- Unfair Compute Comparison: No CoT, few-shot, or self-consistency baselines are provided for single models. MedRoute uses up to 3 specialist calls + 1 moderator makes it an unfair comparison of multi-call aggregation vs. zero-shot direct answer, failing to isolate the gain from routing.
W2. Critical Omissions Regarding Reproducibility
- Undefined : The maximum consultation budget is never given in the settings.
- Unspecified Reward Judge (): The LLM acting as is never named, nor is its prompt, batch size, or judge-consistency provided—despite it being the sole supervisory signal for RL training.
W3. Mechanistic Disconnect from Real Clinical Consultation
Specialists cannot be re-consulted once visited. However, clinical consultation is inherently iterative (e.g., returning to radiology with surgical findings). Masking visited roles reduces the framework to a constrained subset selection over static opinions, undermining the claim of an evolving conversational trajectory.
W4. Missing LLM-as-a-Router Baseline
Using an LLM as a router may lead to better performance and acceptable extra cost.
[1] Chen J, Tang Q, Lu Q, et al. Mol for llms: Dual-loss optimization to enhance domain expertise while preserving general capabilities[J]. arXiv preprint arXiv:2505.12043, 2025.
[2] Zhang Z, Sun L, Qian X, et al. FaithMed: Training LLMs For Faithful Evidence-Based Medical Reasoning[J]. arXiv preprint arXiv:2607.01440, 2026.
Questions for authors
Please solve my concerns of Weakness W1-W4.