acceptodds
Under review as a conference paper at ICLR 2027

Few-Shot Model Onboarding for LLM Routing

Abstract

Model routing reduces the inference cost of large language models (LLMs) by sending each prompt to the cheapest adequate model in a pool. Frequent model releases and costly response labeling motivate onboarding from a handful of labels, yet this setting remains largely unexplored. Recent routers can admit unseen models without retraining, but they either calibrate on a large set of randomly drawn labeled prompts or train an additional item response model to select prompts. We identify prompt selection and label extrapolation as two key factors in few-shot calibration. We propose Active Anchor Calibration (AAC), which clusters existing models' responses to select a few anchor prompts, and uses a closed-form ridge map, together with a cost prior, to extrapolate their labels to model quality and per-prompt predictions. Under a factor model with known prompt factors, we bound pool-level routing regret through an anchor design criterion. Experiments on four public benchmarks show that, with 5 labeled prompts per new model, AAC recovers 88–93% of full-calibration routing quality, against 74–83% for random calibration, which needs about 40 labels to match it.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
LLM routing, Few-shot calibration, Model onboarding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_bBSEOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This article proposes Active Anchor Calibration (AAC), a few shot calibration method for model routing update, without router retraining. It uses clustering on prior knowledge to identify a reduced set of "anchor" prompts suited to extrapolate the quality of new, untested models. This approach is supported by the derivation of a bound on the prediction quality as well as experimental results on QA datasets.

Critical strengths
  1. Method is clearly explained. Every step of the process is detailed and as are their implications.

  2. Ablations are complete, including several component of the method introduced. Boundary and failure cases are alsor reported.

  3. The derivations are consistent and well supported. Codebase is thoroughly provided, including experimental code and results.

Critical weaknesses
  1. Motivation weakness. As acknowledged by the authors, the method is most relevant when using a reduced number of inputs and a large set of prior data points. Such a setup lacks a clear usecase, specifically with regard to the experimental datasets where inference is cheap. Results outside of the binary truth case (AlpacaEval) tend to indicate the improvement provided the method does not hold there. Moreover, there is little motivation given for the design of the method, theoretical results don't directly support its construction.

  2. Much of the performance gain seems to come from the cost prior, which any router can use. It is unclear to what extent would these results hold when considering real cost values.

  3. Lack of relevant baselines. The article takes UniRoute’s problem formulation, routing rule, protocol, metrics and baselines almost unchanged, and pursues a direction explicitly listed as future work (Jitkrittum et al. (2026)). Yet there is no fair comparison to this baseline as only its aggregated results are reported. Additionnaly, elements of an item response model like Yan et al. (2026) are compared against, but not the full published method.

Questions for authors
  1. The authors describe that the response matrix RR is constructed from data "collected in service" but it appears unrealistic given its size. Could the author provide results with a smaller or sparse prior?

  2. Could the authors report each router with the cost prior separately, in particular UniRoute (LearnedMap), and add the learned-cluster-map variant of UniRoute with AAC's anchors and the prior?

  3. The authors mention the addition of unforesseen models to a routing setup but only consider the case where one batch of new models is added. What would happen if we were to consider a stream of models instead? Does the extrapolation based on existing models hold? To what scale can we onboard models before performance starts degrading?

Reviewer_eAMTOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles the problem of onboarding new models in model routing, where it is necessary to get enough understanding of the model to select models when a future prompt arrives. It proposes a method called Active Anchor Calibration to onboard the model with few, rather than a large number of, labels. Its main idea is prompt selection and label extrapolation.

Critical strengths

The paper is motivated by a very concrete problem which is understanding how a new model should be routed. The problem is clearly defined within the domain of using a set of prompt / response to predict how that new model will perform.

Then, a valid methodology is proposed to reduce this onboarding cost by using fewer prompts than the full set. Experimental results show that this method is non-trivial, and random and other common clustering techniques fall short of this method. Section 6.3 gives a nice analysis where we came to understand which parts are sources of the gain.

Critical weaknesses

Given the paper’s contribution as a highly applicational method, a main weakness is situating this method. It is unclear whether reducing the number of onboarding prompts needed while (mildly) reducing the accuracy area is effective or where this might apply.

Given the applicational nature, understanding experimental setups is especially important. First, I acknowledge that the paper includes recent few-shot onboarding baselines like Jitkrittum, et al (2026). However, is the 5-label setup behind the Table in Figure 4 a norm in the literature or otherwise has a reason to be a default setup? Would the advantage maintain for higher labels (Figure 4 shows advantage over random calibration maintains across higher label counts, but it doesn’t use other baselines for that particular experiment)?

Questions for authors

My main questions are regarding the label count - what range is default in the literature, what range is of interest in applications, and on what range does the paper’s method maintains an advantage over baselines - in the Weakenss section.

Reviewer_9q6WOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper studies how to incorporate a new LLM into a routing system using only a few labeled responses. Its method selects representative questions from historical model evaluations using K-means, predicts overall model quality with a simple ridge regression, and combines this estimate with a cost-based prior and transferred prompt-specific predictions adjustment.

Critical strengths

The approach is simple and practical, and I appreciate the careful ablations, uncertainty reporting, and discussion of failure cases. The observation that overall-quality estimation dominates routing errors at very small calibration budgets is useful.

Critical weaknesses

My main concern is the limited methodological novelty. The approach combines established anchor selection, ridge regression, and shrinkage techniques. Its contribution therefore rests largely on the empirical finding that these components work well for few-shot routing, and I am not convinced that the experiments establish this finding broadly enough.

In particular, the motivation emphasizes frequent new-model releases, but the main evaluation holds out models from an existing collection. The organization holdouts are helpful, yet they do not establish transfer to later model generations or substantially different capability profiles. The relationship between a few anchor responses and overall quality could change in precisely these settings. The theory does not resolve this concern because it assumes shared factor structure and leaves the learned-map estimation error as an uncontrolled term.

The practical scope is also limited by the assumption of extensive, consistently graded historical responses and the use of parameter count as a cost proxy.

Questions for authors

see comments

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.