- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This article proposes Active Anchor Calibration (AAC), a few shot calibration method for model routing update, without router retraining. It uses clustering on prior knowledge to identify a reduced set of "anchor" prompts suited to extrapolate the quality of new, untested models. This approach is supported by the derivation of a bound on the prediction quality as well as experimental results on QA datasets.
Critical strengths
-
Method is clearly explained. Every step of the process is detailed and as are their implications.
-
Ablations are complete, including several component of the method introduced. Boundary and failure cases are alsor reported.
-
The derivations are consistent and well supported. Codebase is thoroughly provided, including experimental code and results.
Critical weaknesses
-
Motivation weakness. As acknowledged by the authors, the method is most relevant when using a reduced number of inputs and a large set of prior data points. Such a setup lacks a clear usecase, specifically with regard to the experimental datasets where inference is cheap. Results outside of the binary truth case (AlpacaEval) tend to indicate the improvement provided the method does not hold there. Moreover, there is little motivation given for the design of the method, theoretical results don't directly support its construction.
-
Much of the performance gain seems to come from the cost prior, which any router can use. It is unclear to what extent would these results hold when considering real cost values.
-
Lack of relevant baselines. The article takes UniRoute’s problem formulation, routing rule, protocol, metrics and baselines almost unchanged, and pursues a direction explicitly listed as future work (Jitkrittum et al. (2026)). Yet there is no fair comparison to this baseline as only its aggregated results are reported. Additionnaly, elements of an item response model like Yan et al. (2026) are compared against, but not the full published method.
Questions for authors
-
The authors describe that the response matrix is constructed from data "collected in service" but it appears unrealistic given its size. Could the author provide results with a smaller or sparse prior?
-
Could the authors report each router with the cost prior separately, in particular UniRoute (LearnedMap), and add the learned-cluster-map variant of UniRoute with AAC's anchors and the prior?
-
The authors mention the addition of unforesseen models to a routing setup but only consider the case where one batch of new models is added. What would happen if we were to consider a stream of models instead? Does the extrapolation based on existing models hold? To what scale can we onboard models before performance starts degrading?