- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper aims to address a problem in LLM routing, where most existing methods predict the probability that each model/expert answers correctly and try to reduce this prediction error . This paper argues that an accurate prediction does not mean to select the right expert, and that the relative ordering among experts is the key to routing. Based on this intuition, the paper proposes CompareRoute, which follows the setting of SharedTrunkNet and uses a frozen Observer (Qwen3-8B in the experiments) to run a single prefill over the question, taking the hidden states of the last token as the input to the router (a small network trained on these features). On top of this, the paper makes three changes, (i) it dynamically weights the layers depending on the question and the expert, (ii) it first compares the experts pairwise to predict the difference in their success rates and then aggregates these differences into a score, and (iii) it adds a regret loss and a pairwise loss to the BCE loss during training. The router finally outputs the probability that each expert answers correctly and selects the expert with the highest probability. The paper compares against 5 baselines on 3 benchmarks (SuperGPQA, LiveCodeBench-v6, MMLU-Pro) with 5 7B-scale experts, and performs slightly better on average than the strongest baseline SharedTrunkNet.
Critical strengths
None
I cannot identify any strength of this paper. (i) Novelty: There is little novelty, and the contribution is incremental. It is essentially SharedTrunkNet + a few additional small components. (ii) Experiments: Insufficient. The comparison with SOTA baselines is inadequate, and I feel that even the related work has not been investigated in detail, e.g., the paper has <20 references. (iii) Presentation: Hard to follow, e.g., it is unclear why the relative ordering among experts is the key, and the intuition is missing.
Critical weaknesses
-
The writing, motivation, and challenges of this paper are not clearly explained. (i) How the model is trained and used, which parts are frozen and which are trained, and what the router takes as input and produces as output are all hard to find. This information is scattered across Section 3 and the appendix, the full network architecture only appears in Table 4 of the appendix, the main text has no end-to-end description and no pseudocode, and Figure 2 does not make clear which parts are trained. (ii) The paper argues that the relative ordering among experts should be considered, but it only gives an 80%/70% example in the 2nd paragraph of Introduction to show that a smaller prediction error does not mean a better choice. This is only a phenomenon, and it is neither a motivation nor a reason. The paper does not explain why its architecture ranks the experts more accurately, nor what the intuition and the underlying mechanism are. (iii) The paper's own Direct baseline uses the same features and labels and differs only in the network architecture, yet the paper does not explain why pairwise comparison ranks better. (iv) This paper looks almost the same as SharedTrunkNet. The Observer input and the overall setting are the same, and the training procedure is largely the same. The only differences are inside the router, e.g., how the layers are weighted, the pairwise comparison structure and the loss functions.
-
The experiments are insufficient, and the current results do not support the paper's conclusion. (i) There is only 1 expert pool (5 7B-scale models), 1 Observer and 3 benchmarks, which is even narrower than the evaluation in the SharedTrunkNet paper. (ii) The paper emphasizes the relative ordering among experts but provides no analysis of ranking at all and only reports success rates. (iii) The paper cites routing methods such as RouterDC, GraphRouter, IRT-Router and EquiRouter but compares with none of them, in particular EquiRouter, which also uses pairwise ranking supervision. (iv) Baselines such as KNN, MLP and RouteLLM use MiniLM sentence embeddings, while CompareRoute uses Observer features, so even when CompareRoute wins it is impossible to tell whether the gain comes from the method or from the features. (v) Qwen3-8B itself may be stronger than all of these 7B experts, but the paper does not report the results of letting the Observer answer directly. (vi) The improvement is small, only 1.06 pps over SharedTrunkNet on average.
Questions for authors
- Why does considering the relative ordering among experts lead to better selection than directly predicting each expert's success rate? What are the intuition and the underlying mechanism? Please provide strong evidence that relative ordering is better than direct prediction. This evidence should not come only from improvements in task accuracy, since such improvements can come from many factors. A more concrete metric is needed for this comparison.
- Full and Direct use the same features and labels and differ only in the network architecture. Why does pairwise comparison followed by averaging rank more accurately?
- Does the size of the Observer matter, or are there any requirements on it? Does the Observer need to be able to judge the difficulty of a question? Does the method still work if the Observer is weaker than the experts? For example, in agentic settings, the question itself may not be difficult while tool calling is very hard. Since the Observer only sees the prefill hidden states of the question, can it reflect this kind of difficulty, and how would the method behave in this case?