- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
MedRouter selects medical specialist LLMs using a lightweight multi-label router and passes their reasoning to a frozen answer generator. Its two-stage training first predicts specialist correctness, then optimizes routing through reinforcement learning using a generator-level reward. Experiments on eight medical QA benchmarks show higher average accuracy than the evaluated baselines.
Critical strengths
none
Critical weaknesses
-
The reward does not establish effective use of complementary specialist knowledge. PGR evaluates the selected expert set through final-answer correctness, without directly identifying individual contributions. When the generator can already answer correctly, a set containing useful, redundant, and misleading information can still receive a positive reward. Although exploration may help distinguish such experts over time, the paper does not demonstrate that this occurs reliably. This weakens the evidence supporting its claim of learning to exploit complementary specialist knowledge.
-
The experiments do not sufficiently isolate the benefit of learned routing. Despite comparisons with full-pool LLM Ensemble and training-stage ablations, the paper lacks fixed expert combinations, call-count-matched random selection, and simple Top-k selection evaluated with the same generator and aggregation procedure. Consequently, the improvements may partly reflect providing an appropriate amount of expert information rather than learning superior question-specific selections. This missing attribution substantially limits the strength of the empirical contribution.
Questions for authors
-
How does the routing policy distinguish useful experts from redundant or misleading ones when the generator already answers correctly without expert information? What evidence shows that PGR learns these distinctions rather than reinforcing incidental expert selections?
-
Although the paper compares against full-pool LLM Ensemble and includes training-stage ablations, it lacks several important controls using the same generator and aggregation prompt: a) Fixed expert combinations applied to every question. b) Random expert selection matched to MedRouter’s number of expert calls per question. c) Simple Top-k selection based on the Stage-I router scores. Without these controls, it remains unclear whether the gains come from learned question-specific routing or simply from providing the generator with an appropriate amount of expert information. This limits the evidence supporting the routing method’s contribution.
-
Why choose six expert groups? Given the vast number of specialized fields in medicine, relying solely on text and images to construct the expert pool remains insufficient. Can this approach be generalized to different expert pools? One could test performance after removing, replacing, or adding experts.