acceptodds
Withdrawn from ICLR 2027

CompareRoute: Learning Relative Expert Capability for Prefill-Based LLM Routing

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

LLM routing selects an expert before answer generation, requiring reliable estimates of relative expert capability on each query. We propose CompareRoute, a prefill-based router that uses expert-conditioned dynamic layer aggregation to read multilayer states from a frozen Observer. A shared antisymmetric comparator predicts pairwise success-rate differences from the resulting expert-specific query representations. Training combines success supervision and continuous regret with a comparison loss that gives additional weight to pairs involving the router’s current top choice, proportional to their empirical success-rate gaps. Inference requires one Observer prefill and invokes only the selected expert for answer generation. Across three benchmarks with five heterogeneous experts, in-domain evaluation shows an average success-rate improvement of 1.06 percentage points over the strongest evaluated baseline. Selected-expert token usage decreases by an average of 30.65% relative to single-expert references, excluding Observer overhead. These results highlight the potential of CompareRoute to improve response quality while reducing token usage in multi-LLM deployments.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM Routing

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oWqcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper aims to address a problem in LLM routing, where most existing methods predict the probability that each model/expert answers correctly and try to reduce this prediction error . This paper argues that an accurate prediction does not mean to select the right expert, and that the relative ordering among experts is the key to routing. Based on this intuition, the paper proposes CompareRoute, which follows the setting of SharedTrunkNet and uses a frozen Observer (Qwen3-8B in the experiments) to run a single prefill over the question, taking the hidden states of the last token as the input to the router (a small network trained on these features). On top of this, the paper makes three changes, (i) it dynamically weights the layers depending on the question and the expert, (ii) it first compares the experts pairwise to predict the difference in their success rates and then aggregates these differences into a score, and (iii) it adds a regret loss and a pairwise loss to the BCE loss during training. The router finally outputs the probability that each expert answers correctly and selects the expert with the highest probability. The paper compares against 5 baselines on 3 benchmarks (SuperGPQA, LiveCodeBench-v6, MMLU-Pro) with 5 7B-scale experts, and performs slightly better on average than the strongest baseline SharedTrunkNet.

Critical strengths

None


I cannot identify any strength of this paper. (i) Novelty: There is little novelty, and the contribution is incremental. It is essentially SharedTrunkNet + a few additional small components. (ii) Experiments: Insufficient. The comparison with SOTA baselines is inadequate, and I feel that even the related work has not been investigated in detail, e.g., the paper has <20 references. (iii) Presentation: Hard to follow, e.g., it is unclear why the relative ordering among experts is the key, and the intuition is missing.

Critical weaknesses
  1. The writing, motivation, and challenges of this paper are not clearly explained. (i) How the model is trained and used, which parts are frozen and which are trained, and what the router takes as input and produces as output are all hard to find. This information is scattered across Section 3 and the appendix, the full network architecture only appears in Table 4 of the appendix, the main text has no end-to-end description and no pseudocode, and Figure 2 does not make clear which parts are trained. (ii) The paper argues that the relative ordering among experts should be considered, but it only gives an 80%/70% example in the 2nd paragraph of Introduction to show that a smaller prediction error does not mean a better choice. This is only a phenomenon, and it is neither a motivation nor a reason. The paper does not explain why its architecture ranks the experts more accurately, nor what the intuition and the underlying mechanism are. (iii) The paper's own Direct baseline uses the same features and labels and differs only in the network architecture, yet the paper does not explain why pairwise comparison ranks better. (iv) This paper looks almost the same as SharedTrunkNet. The Observer input and the overall setting are the same, and the training procedure is largely the same. The only differences are inside the router, e.g., how the layers are weighted, the pairwise comparison structure and the loss functions.

  2. The experiments are insufficient, and the current results do not support the paper's conclusion. (i) There is only 1 expert pool (5 7B-scale models), 1 Observer and 3 benchmarks, which is even narrower than the evaluation in the SharedTrunkNet paper. (ii) The paper emphasizes the relative ordering among experts but provides no analysis of ranking at all and only reports success rates. (iii) The paper cites routing methods such as RouterDC, GraphRouter, IRT-Router and EquiRouter but compares with none of them, in particular EquiRouter, which also uses pairwise ranking supervision. (iv) Baselines such as KNN, MLP and RouteLLM use MiniLM sentence embeddings, while CompareRoute uses Observer features, so even when CompareRoute wins it is impossible to tell whether the gain comes from the method or from the features. (v) Qwen3-8B itself may be stronger than all of these 7B experts, but the paper does not report the results of letting the Observer answer directly. (vi) The improvement is small, only 1.06 pps over SharedTrunkNet on average.

Questions for authors
  1. Why does considering the relative ordering among experts lead to better selection than directly predicting each expert's success rate? What are the intuition and the underlying mechanism? Please provide strong evidence that relative ordering is better than direct prediction. This evidence should not come only from improvements in task accuracy, since such improvements can come from many factors. A more concrete metric is needed for this comparison.
  2. Full and Direct use the same features and labels and differ only in the network architecture. Why does pairwise comparison followed by averaging rank more accurately?
  3. Does the size of the Observer matter, or are there any requirements on it? Does the Observer need to be able to judge the difficulty of a question? Does the method still work if the Observer is weaker than the experts? For example, in agentic settings, the question itself may not be difficult while tool calling is very hard. Since the Observer only sees the prefill hidden states of the question, can it reflect this kind of difficulty, and how would the method behave in this case?
Reviewer_qXBKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CompareRoute, a prefill-based LLM routing framework that learns relative expert capabilities to select the most suitable model for each query. It uses a frozen Observer to extract multilayer hidden states, applies expert-conditioned dynamic layer aggregation, and predicts pairwise success-rate differences through an antisymmetric comparator. The model is trained with BCE, continuous regret, and weighted comparison losses. Experiments on three benchmarks with five expert models show an average success-rate improvement of 1.06 percentage points over the strongest evaluated baseline, along with an average reduction of 23.71% in total token usage compared with the best-single-expert reference.

Critical strengths
  1. Well-motivated routing framework. The paper highlights the importance of relative expert capability rather than absolute success-probability estimation for routing decisions. The proposed architecture provides a coherent way to learn pairwise expert comparisons from shared prefill representations.
  2. Relatively comprehensive experimental analysis. The paper evaluates several routing baselines across three benchmarks and provides controlled comparisons, module ablations, and statistical confidence intervals. The analysis of token usage and cross-dataset transfer provides additional insights into the method's effectiveness and limitations.
Critical weaknesses
  1. Unclear Justification for the Observer–Expert Configuration. CompareRoute uses Qwen3-8B as the Observer but delegates generation to similarly sized, mostly older-generation experts. Without a direct-answer baseline using Qwen3-8B, it is unclear whether this two-stage configuration provides an advantage over simply using the Observer itself.
  2. Missing Mixed-Domain Routing Evaluation. The paper trains a separate router for each benchmark but does not evaluate a unified router on mixed-domain workloads. Although the authors acknowledge the limited cross-dataset transfer, the results—four of six transfer directions underperforming the target best-single expert—raise questions about how strongly routing gains depend on domain-specific training. A mixed-domain evaluation would help establish whether the proposed approach can handle heterogeneous traffic while retaining its advantages over simpler routing methods.
  3. Limited Methodological Novelty. The contribution appears incremental, mainly refining hidden-state aggregation and comparison objectives within the existing prefill-based routing framework. Expert-conditioned representations and pairwise ranking supervision have already been explored in prior routing work, leaving the specific methodological advance over these approaches insufficiently clear.
  4. Limited Validation of Robustness to Sampling Noise. Each query–expert success rate is estimated from only five responses. The resulting sampling noise may affect both the pairwise comparison targets and the weights derived from their magnitudes. The stability of the reported gains under larger or independently generated response banks remains unexamined.
Questions for authors
  1. Could the authors compare CompareRoute with Qwen3-8B answering directly under clearly specified decoding settings, reporting both success rate and token usage? Does CompareRoute retain its advantage when Qwen3-8B is included in the expert pool?
  2. Could the authors evaluate a unified router trained on pooled data from the three benchmarks and tested on held-out queries from each domain? A comparison with a similarly trained Shared TrunkNet would help determine whether CompareRoute retains its advantage under mixed-domain training.
  3. Do the gains, particularly those from comparison weighting, persist with more response samples or an independent response bank? Are the main improvements over Shared TrunkNet stable across independent repetitions of the full training procedure?
Reviewer_Si9tOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Compare Route selects an LLM by learning pairwise differences in expert success rates from a frozen observer’s hidden states. It combines expert-dependent layer weights with comparison-based training. On 3 benchmarks with 5 experts, it improves success by 1.06 percentage points over the strongest evaluated baseline on average.

Critical strengths

The improvements are consistent across 3 benchmarks. The controlled ablations provide useful evidence for expert-dependent layer weights and pairwise comparison. The shared data splits and paired evaluation also make the results more convincing.

Critical weaknesses

The baseline comparison does not sufficiently establish the contribution over existing routing methods. RouterDC and Graphrouter are relevant multi-expert routers, and EquiRouter also learns model rankings rather than absolute performance scores. These methods are discussed but not evaluated. A comparison with EquiRouter under the same features and supervision would help show what the proposed comparison design adds beyond existing methods.

The main results use a separate router for each benchmark, with two knowledge QA benchmarks and one code benchmark. This supports routing within each dataset, but does not establish that one router can handle a broader mixture of tasks. Given the weak transfer between code and knowledge tasks, a mixed-task evaluation is important for assessing the method’s practical scope.

Questions for authors

See weakness

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.