acceptodds
Under review as a conference paper at ICLR 2027

MedRouter: Demystifying Knowledge Differences Across Medical LLMs for Routing-Based Reasoning

Abstract

Medical question answering spans diverse specialties and modalities, and individual medical large language models (LLMs) exhibit distinct strengths across tasks and domains. This heterogeneity suggests that combining specialists may enable broader coverage of medical questions than relying on any single model. However, existing LLM routing methods primarily seek to balance answer quality and inference cost, leaving open how to exploit differences in specialist competence to improve medical reasoning. In this paper, we introduce MedRouter, an agentic system that uses an embedding-based multi-label router to select and query specialist LLMs, then passes their responses to a generator to produce the final answer. We further propose SCALE (Specialist Competence-Aware Learning), a two-stage training framework that first trains the Router with specialist correctness supervision and then optimizes its selections through reinforcement learning. The second stage uses a Performance Gain Reward (PGR) that measures how specialist information affects the generator's answer correctness relative to answering without that information. Experiments on eight text-based and multimodal medical QA benchmarks show that MedRouter outperforms the strongest routing baseline by 8% in average accuracy. Our analysis of specialist outputs further reveals distinct strengths and complementary question-level coverage, motivating learned routing to combine these capabilities for more comprehensive medical reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Reasoning, Agent, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_qXarOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

MedRouter selects medical specialist LLMs using a lightweight multi-label router and passes their reasoning to a frozen answer generator. Its two-stage training first predicts specialist correctness, then optimizes routing through reinforcement learning using a generator-level reward. Experiments on eight medical QA benchmarks show higher average accuracy than the evaluated baselines.

Critical strengths

none

Critical weaknesses
  1. The reward does not establish effective use of complementary specialist knowledge. PGR evaluates the selected expert set through final-answer correctness, without directly identifying individual contributions. When the generator can already answer correctly, a set containing useful, redundant, and misleading information can still receive a positive reward. Although exploration may help distinguish such experts over time, the paper does not demonstrate that this occurs reliably. This weakens the evidence supporting its claim of learning to exploit complementary specialist knowledge.

  2. The experiments do not sufficiently isolate the benefit of learned routing. Despite comparisons with full-pool LLM Ensemble and training-stage ablations, the paper lacks fixed expert combinations, call-count-matched random selection, and simple Top-k selection evaluated with the same generator and aggregation procedure. Consequently, the improvements may partly reflect providing an appropriate amount of expert information rather than learning superior question-specific selections. This missing attribution substantially limits the strength of the empirical contribution.

Questions for authors
  1. How does the routing policy distinguish useful experts from redundant or misleading ones when the generator already answers correctly without expert information? What evidence shows that PGR learns these distinctions rather than reinforcing incidental expert selections?

  2. Although the paper compares against full-pool LLM Ensemble and includes training-stage ablations, it lacks several important controls using the same generator and aggregation prompt: a) Fixed expert combinations applied to every question. b) Random expert selection matched to MedRouter’s number of expert calls per question. c) Simple Top-k selection based on the Stage-I router scores. Without these controls, it remains unclear whether the gains come from learned question-specific routing or simply from providing the generator with an appropriate amount of expert information. This limits the evidence supporting the routing method’s contribution.

  3. Why choose six expert groups? Given the vast number of specialized fields in medicine, relying solely on text and images to construct the expert pool remains insufficient. Can this approach be generalized to different expert pools? One could test performance after removing, replacing, or adding experts.

Reviewer_1sQLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MedRouter, a system that uses a frozen multimodal embedding encoder and a trainable multi-label routing head to select medical specialist LLMs, then passes their reasoning to a frozen answer generator. Its SCALE framework first trains the router with individual specialist correctness labels and then optimizes its selections using reinforcement learning and a Performance Gain Reward (PGR). Experiments on eight text-based and multimodal medical QA benchmarks show that MedRouter outperforms the strongest routing baseline by 8% in average accuracy.

Critical strengths
  • The question-level analysis of specialist correctness is useful. Figure 1 provides a clear empirical motivation for selecting specialists per question.
  • The separation of routing and generation is straightforward and practical. Only the routing head and its value head are trained, while the encoder, specialists, and generator remain frozen. Caching specialist responses also gives the compared routing methods access to consistent specialist outputs.
  • The experiments cover text and image questions with two generators. Training-stage ablations and the case study add useful evidence, and the reported text QA gains are substantial.
Critical weaknesses
  • Eq. (4) uses C(A⁺, A) - 0.5 C(A⁻, A). For a fixed question and frozen generator, the second term does not depend on the selected experts. Its expected policy gradient is therefore the same as that of correctness-only reward, and the subtracted term can be included in the value baseline already used in Eq. (6). Figure 2 also omits the 0.5 coefficient.

  • The comparisons do not clearly separate routing gains from the benefit of a separate generator. Prompt LLM returns an expert’s answer, while MedRouter combines expert reasoning. The generation steps for KNN, MLP, and Encoder Router are unclear. MLP Router scores 40.9%, compared with 43.6% for Stage-I-only MedRouter, despite similar router training and thresholds. Please explain this difference and compare random, fixed top-k, and learned selection using the same generator, input format, and number of expert calls.

Questions for authors
  • All answers are scored by an LLM judge without a check against human labels or exact option matching. Errors in judging could affect specialist labels, rewards, and the final evaluation.

  • The system often calls several experts, but average calls, total tokens, response time, and offline response-generation cost are not reported. Image inputs and the exclusion of text-only experts from image questions also need clarification.

  • The introduction and Figure 1(h) use different macro/micro averaging descriptions.

  • Do all routing baselines use the same generator and expert reasoning, and how do they compare at the same call count?

Reviewer_N8R5OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes MedRouter, which combines a frozen embedding encoder, a multi-label specialist router, and a frozen answer generator. SCALE trains the router using specialist correctness labels followed by REINFORCE with a generator-based reward. Experiments cover eight medical QA benchmarks, including two held-out MedXpertQA subsets.

Critical strengths

The paper studies a relevant problem and provides useful descriptive evidence of question-level complementarity among medical models. Evaluation with two generators and eight benchmarks shows consistent improvements in mean accuracy, and the training-stage ablations help characterize the proposed system.

Critical weaknesses

a. The interpretation of the central reward contribution is insufficiently justified. In Eq. (4), the generator-only correctness term is independent of the routing action. Under the stated frozen-generator setting, subtracting this term leaves the expected policy gradient unchanged relative to optimizing final-answer correctness. It may affect gradient variance and finite-sample optimization, but does not introduce a different action-dependent objective. Although Table 2 reports an improvement over correctness-only reward, the paper does not establish whether this reflects a reproducible variance-reduction benefit, particularly given the additional learned value baseline in Eq. (6). This limits the methodological significance supported by the current evidence.

b. The experimental comparisons do not sufficiently isolate the contribution of learned routing from response integration. The MLP Router in Table 1 and the Stage-I-only variant in Table 2 appear closely related in their stated architecture and supervision, yet obtain 40.9 and 43.6 average accuracy. Their aggregation protocols and the source of this difference need clarification. Moreover, the 332 correct answers outside the specialist-pool oracle do not by themselves demonstrate complementary information integration, since the generator may already solve some of these questions independently. These ambiguities weaken the attribution of the reported gains to the proposed routing mechanism.

Questions for authors
  1. Can you clarify the relationship between PGR and an action-independent baseline, and provide multi-seed evidence explaining its advantage over correctness-only reward?
  2. What exactly differs between the MLP Router and Stage-I-only pipelines? Are their generators, specialist inputs, and aggregation prompts identical?
  3. Among the 332 questions, how many are answered incorrectly by the generator alone? How does learned routing compare with fixed-best-specialist and random matched-cardinality routing using the same generator?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.