- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper fine-tunes a query encoder to predict correctness across multiple source LLMs, then discards the prediction head and reuses the frozen encoder with downstream routers. Experiments on EmbedLLM report improvements across eight routing variants and several target-supervision budgets. Transfer to RouterBench also improves the reported quality–cost tradeoff. The main contribution is a reusable representation learned from historical model outcomes, with no candidate-model probing at inference time.
Critical strengths
Learning the representation once and reusing it across downstream routers is a useful practical direction. It moves the additional work into training and preserves a simple inference pipeline that calls only the selected LLM.
Critical weaknesses
- The main experiments are limited to EmbedLLM, with external transfer tested only on RouterBench. Prior analysis finds that routers on RouterBench tend to favor globally strong models, so gains there do not by themselves establish transferable, query-specific routing ability. Evaluation on another benchmark such as SPROUT/CARROT, together with best-single-model and cost-matched static baselines, is needed to substantiate the claim of a broadly reusable encoder.
- KNN's weak performance needs explanation. In Table 2, Raw KNN achieves 59.82%, versus 64.66% for logistic regression, whereas The Routing Plateau and Rethinking Predictive Modeling for LLM Routing find KNN highly competitive. Different model pools or data splits could explain this discrepancy, but the paper neither reconciles these differences nor specifies KNN's neighborhood-size selection and tie-breaking. This leaves uncertainty about baseline quality and whether the reported benefits persist against a well-tuned KNN router.
Questions for authors
- On RouterBench, what gains remain over a validation-selected best single model and query-independent model mixtures at matched cost? This would help establish whether transfer improves query-specific decisions rather than mainly recovering global model preferences. Can you demonstrate frozen-encoder transfer on another benchmark, such as SPROUT/CARROT, using the same strong baselines and a verified separation between source training and target test data?
- Why does KNN substantially underperform the learned routers in Table 2, given its strong performance in The Routing Plateau and Rethinking Predictive Modeling for LLM Routing? Please reconcile the model pools and splits and report KNN's neighbor count, validation search, similarity metric, and tie-breaking. Do the ranking and behavioral-pretraining gains persist against a well-tuned KNN baseline under your exact setup?
- What is the intuition for why predicting correctness on a particular source-model pool should produce a broadly reusable query representation? Two queries with similar source-model outcomes may have different preferences under another pool or query distribution. What general query properties does the objective learn beyond source-pool-specific patterns?
2 comments
Thank you for the reply!
My question still stands:
-
What is the margin of uncertainty around the increase in routing reward of +0.0108 and the quality correction improvement through the use of the Behavior-Raw routing method?
-
The confidence intervals are calculated based on the macro AUC for the KNN method. Is there any concrete evidence that supports the lower routing reward for the KNN method, which is attributed to the ties problem?
Thank you for carefully reading our work and for raising these concrete questions. We added query-independent static baselines, held-model transfer, strict family-holdout tests, and a full KNN protocol audit. We respond point by point below.
W1 + Q1 — Does RouterBench transfer reflect query-specific routing?
Yes. Query-dependent routing clearly outperforms query-independent static strategies, and Behavior adds a further representation gain.
We add two static baselines: (B1) a validation-selected best single model, and (B2) a query-independent static mixture optimized under the same cost constraint. To ensure fairness, B1/B2 are re-solved at each router's realized average cost.
Their NQCA values are nearly identical:
By comparison, Table 4 reports Behavior NQCA of 0.7617 / 0.7617 / 0.7624 for LogReg / KNN / MLP. Under matched-cost comparison, Behavior exceeds B2 by +0.0538 / +0.0586 / +0.0567 average quality, positive at all 30/30 operating points for each router.
Thus, RouterBench gains cannot be explained by selecting a globally strong model or a fixed model mixture; the main gain comes from query-specific model selection.
The representation also transfers beyond one model pool. Replacing the 89 seen models with 23 held-out models whose outcomes were never used in pretraining still gives +0.0304 macro AUC / +0.0108 reward for Behavior−Raw. Strict leave-MMLU-out training also gives 0.6072 AUC vs. 0.5818 Raw (+0.0254) on held-out MMLU.
W2 + Q2 — Why is KNN weaker than learned routers?
KNN is validation-tuned, and the Behavior gain remains significant; the weaker result is specific to top-1 selection in our 89-model binary-outcome setting.
Our KNN uses L2-normalized embeddings, cosine similarity, validation search over
k ∈ {4,...,256}, and selects k=128. Candidate-model scores are the mean binary outcomes of the top-k neighbors.Neighbor selection uses
np.argpartition, while final model selection usesnp.argmax, which returns the first maximum. AUC usesrankdata(method="average"), so ties receive average ranks.Because EmbedLLM outcomes are binary, KNN scores lie on
{0,1/k,...,1}. With 89 candidate models, neighborhood averaging produces a coarse score grid, making top-1 selection more tie-sensitive. Accordingly, Behavior KNN reaches 0.7251 macro AUC, compared with 0.7441 for LogReg; the ranking gap is much smaller than the reward gap.Most importantly, under this tuned KNN protocol, Behavior−Raw still reaches +0.0288 macro AUC [0.0235,0.0357]. Therefore KNN's lower absolute reward does not explain the behavioral-pretraining gain.
Q3 — Why can correctness on one source-model pool produce a reusable query representation?
The source models act as diverse capability probes: their joint success/failure pattern reveals which properties of a query make different models succeed, rather than merely identifying which source model is best.
The key is that supervision comes from the full outcome vector, not a source-model ID. If two queries induce similar success/failure patterns across many heterogeneous models, they are likely to share properties that affect model capability—for example difficulty or the type of knowledge required. These query-side properties can remain useful even when the candidate models change.
Our transfer results directly test this interpretation. Behavior−Raw macro AUC is +0.0260 on the 89 source models and +0.0304 on 23 held-out models whose outcomes were never used in encoder pretraining. Thus the gain does not depend on reusing the source-model identities. It also survives query-distribution shift: after removing all MMLU text and behavior from encoder training, Behavior still achieves 0.6072 vs. 0.5818 AUC (+0.0254) on held-out MMLU.
The transferable signal is also richer than scalar difficulty. A difficulty-only neighborhood reaches only 0.605–0.610 pair-preference agreement, versus 0.775–0.796 for behavioral neighborhoods. Moreover, Full−Difficulty grows to +0.0044 / +0.0035 AUC in the highest model-disagreement regions and is near zero when models behave similarly.
Together, these results support a concrete interpretation: behavioral pretraining uses the source pool to expose query properties that predict differences in model success. Because those properties belong to the query rather than to a fixed model identity, the resulting representation remains useful for new model pools and new query families.