- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper studies how to route queries among LLMs while balancing answer quality and cost. Model Metropolis maintains capability- and difficulty-specific success estimates using discounted Beta updates, then combines quality thresholds, evidence requirements, exploration, and cost-aware selection (lines 181–245). It also extends the framework to escalation stopping and activation-based pre-generation routing (lines 253–295). Evaluation includes a repeated-task online gateway experiment, held-out public benchmarks, and tests of the two extensions (lines 298–305, 1250–1304, 1306–1320, and 1350–1357). The reported hard-task gains are promising, while transfer results depend on calibration coverage (lines 1215–1231 and 1268–1285).
Critical strengths
S1. A practical approach to an important routing problem. Tracking actual success by task type and difficulty is a useful basis for model selection, particularly when aggregate model scores hide differences in reliability (lines 181–201). The reported hard-task delivery improves from 73% for Cost-min and 70% for the difficulty-agnostic bandit to 91% for the full method, with 91.2 ± 1.0% across three repeats (lines 1215–1231). These results provide encouraging evidence for the overall design, although the contribution of each component remains unclear.
S2. Held-out evaluation reveals when calibration transfers. The paper includes disjoint calibration/evaluation splits (lines 1250–1252). On the same 30 held-out LiveCodeBench hard problems, adding separate, task-family-matched calibration evidence raises delivery from 10.0% to 46.7% (lines 1268–1285). This is informative because it identifies calibration coverage as an important condition for transfer, while the mixed benchmark results make the limits visible (lines 1254–1263 and 1277–1285).
Critical weaknesses
W1. The experiments do not systematically isolate the proposed mechanisms. The central design combines difficulty-dependent thresholds, a mean-based quality gate, evidence requirements, and targeted exploration (lines 812–852). Although the difficulty-agnostic comparison is useful (lines 1216–1218), the component ablation in Table 12 reports only qualitative routing behavior, without delivery or cost measurements (lines 1242–1249). It therefore remains unclear how much each mechanism contributes beyond shifting more hard queries to strong models. This limits the evidence for the specific algorithmic contribution.
The evaluation protocols also leave a gap: the main learning results are reported after five rounds on the 115-task stream (lines 303–305 and 1179–1181), whereas the deterministic public-benchmark replay disables Thompson sampling and targeted exploration (lines 812–816). Repeated-task learning is a valid setting, but these experiments do not establish the full online policy’s performance on fresh requests. Quantitative component ablations and learning curves on an unseen task stream, with matched feedback and budgets, would directly address this concern.
W2. The cost and feedback assumptions limit the practical conclusions. The headline 0.44× relative cost uses the mean selected-model output-token unit price, rather than actual token usage (lines 854–863 and 1329–1343). It therefore does not establish an equivalent reduction in total spending. Later benchmarks use token-based costs (lines 861–863), but the main savings claim still needs accounting for input/output tokens, calibration, exploration, and retries. Comparisons at matched quality or matched total cost would make the practical benefit easier to assess.
The general-benchmark escalation experiment also uses an oracle verdict that is explicitly unavailable during normal inference (lines 1017–1019). This is useful as a controlled test, but it leaves the benefit under deployable feedback uncertain. Evaluation with an available verifier, including its errors and cost, is needed to support that broader claim. Together, these limitations weaken the evidence for the paper’s central quality–cost benefit in deployment.
Questions for authors
Q1. Which mechanisms drive the gains, and how are the learning results aggregated? Does the reported 91% hard delivery refer to the final round or another aggregation of the five rounds (lines 303–305 and 1179–1181)? Can you provide quantitative delivery/cost ablations for the components in Table 12, together with results on a fresh held-out stream using matched feedback and total budgets (lines 1242–1249)? These results would help establish what the full online policy contributes.
Q2. Does the cost advantage remain under complete accounting? What are the overall and hard-only costs when actual input/output tokens, calibration, exploration, and retries are included, and which costs are amortized (lines 854–863 and 1329–1343)? At matched delivery or matched total cost, does the claimed advantage remain? This would clarify what the 0.44× figure implies for practical savings.
Q3. What feedback supports escalation in normal deployment? For the general-benchmark setting, what usable verifier would replace the oracle verdict in Table 7, and how do its errors, availability, and cost affect the reported gains (lines 1017–1024)? Evidence under such feedback would materially change the assessment of the escalation extension’s practical value.