- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 3(+1)
- Presentation
- 2(+1)
- Soundness
- 2
Read review
Paper summary
This paper proposes Route-then-Probe (RTP), a two-stage framework for cost-effective LLM routing. RTP first selects a suitable small model based on query–model compatibility, then probes its prefill hidden states to determine whether escalation is needed. Experiments across diverse benchmarks demonstrate superior cost–accuracy trade-offs over existing baselines.
Critical strengths
- The paper presents a practical two-stage routing framework that combines the complementary strengths of multiple small models with inference-time correctness signals for selective escalation.
- The experiments cover diverse tasks and competitive baselines, demonstrating consistent improvements in cost–accuracy trade-offs and the applicability of the proposed framework to existing routing methods.
Critical weaknesses
- Insufficient justification and analysis of behavioral fingerprints. The first-stage router represents each candidate model using a behavioral fingerprint constructed from 216 randomly sampled training queries. However, the paper does not explain why this particular number is chosen or whether random sampling provides sufficient coverage of diverse model capabilities. Since the datasets vary substantially in size and task characteristics, randomly selected anchors may underrepresent certain specialized capabilities. Moreover, the model representation combines the fingerprint with a learnable model-specific embedding, making it unclear how much the fingerprint itself contributes to routing performance. Additional ablations on fingerprint size, sampling strategies, and the contribution of model identity embeddings would strengthen this component.
- Insufficient evidence for the necessity of the L2 probe. A central motivation of the paper is that inference-time hidden states provide more reliable correctness signals than query-side predictions. However, the L1 router already produces per-model correctness scores, which could potentially serve as confidence estimates for escalation decisions. The paper does not directly compare escalation based on calibrated L1 confidence with escalation based on L2 probe confidence under the same routing framework. Consequently, it remains unclear how much additional information the hidden-state probe provides beyond the first-stage predictions, particularly considering its additional prefill overhead for escalated requests. Such a comparison is important for establishing the necessity and effectiveness of the proposed two-stage design.
- Limited justification and evaluation of the cost-aware escalation rule. The L2 escalation rule compares the estimated failure probability with a threshold proportional to the ratio between small-model and flagship costs. However, the theoretical motivation for this particular formulation is insufficiently discussed. For example, increasing the flagship cost reduces the threshold and can make escalation more likely, which appears counterintuitive from a cost-optimization perspective. Furthermore, the rule assumes that the flagship always answers correctly, although its reported accuracy is below 100%. While the paper reports cost–accuracy curves and APGR, additional comparisons at fixed cost budgets or fixed accuracy targets, together with ablations of alternative escalation rules, would better establish its practical effectiveness.
Questions for authors
Here I list the questions I am most concerned about. Please see the Weaknesses section for more details.
- Why were 216 randomly sampled queries selected, and how sensitive is L1 performance to the number and sampling strategy of anchor queries, particularly when model identity embeddings are removed?
- How does escalation using calibrated L1 confidence compare with the proposed L2 probe under identical routing and cost constraints, and do the two signals provide complementary information?
- What motivates the proposed cost-ratio threshold, and how does it compare with alternative decision rules based on expected accuracy improvement and incremental inference cost?
I would be willing to raise my score if the authors can adequately address these concerns, as I believe the proposed two-stage framework is practically meaningful and shows promising empirical results, but stronger evidence is needed to justify its key design choices and establish the contributions of individual components.
4 comments
Response to Reviewer 6SnM, part 2 of 2.
Weakness 3 and Q3. Cost-aware escalation.
Thanks for pointing this out. There was a mistake in our original escalation rule. We have therefore revised the escalation condition and added a flagship output-length regressor (Eq. (G1) in the general response). The revision makes two changes:
- Incremental cost instead of a cost ratio. When L2 decides, the selected small model has already completed its prefill. This prefill is a sunk cost shared by both branches. The decision should therefore compare the cost of calling the flagship with the remaining decoding cost of the small model, rather than the ratio of their total costs.
- Per-request estimate of the flagship cost. The flagship's output price is three times its input price. Its output length also varies widely across requests, so a fixed constant cannot capture the escalation cost of each request. The new ModernBERT regressor therefore predicts this length from the request. Its inference cost is included in the total cost.
With these changes, for a fixed , a more expensive flagship or cheaper small-model decoding makes escalation harder. The rule is now consistent with cost optimization.
To evaluate the rule at fixed cost budgets, we read the accuracy and escalation rate from RTP's cost–accuracy curve. Each budget caps the average cost at a fraction of the flagship's average cost:
| Average-cost cap (relative to the flagship) | Accuracy | Escalation rate |
|---|---|---|
| 10% | 81.98% | 9.3% |
| 20% | 85.60% | 22.4% |
| 30% | 88.28% | 37.0% |
| 50% | 90.18% | 51.2% |
| Always calling the flagship | 92.26% | 100% |
At no more than half of the flagship's average cost, RTP reaches 90.18% accuracy. This is close to the 92.26% obtained by always calling the flagship. RTP's overall APGR is 0.7185.
Finally, we also tried several alternative rules that trade off expected accuracy improvement against incremental cost. Their performance was similar. In each rule, controls this tradeoff and can be adjusted to reach the desired accuracy and escalation rate.
Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts in addressing my concerns.
One minor concern remains: I hope the authors can clarify how the revised escalation rule accounts for the fact that the flagship model is not always correct, and provide the quantitative comparison with alternative escalation rules in the final version.
Overall, I believe the paper has been substantially improved, and I have decided to raise my score from 2 to 3.
We thank the reviewer for the comments. The term also represents the selected model's failure risk. We will clarify this and provide quantitative comparisons with alternative escalation rules in the revised paper.
Response to Reviewer 6SnM, part 1 of 2.
We thank the reviewer for the thoughtful comments and suggestions. Our general response, "Method Revisions and Updated Results", summarizes the revised method and the updated results. We address your specific questions below.
Weakness 1 and Q1. Anchor fingerprints.
Thanks for this comment. With the revised L1, we find that model embeddings learned during training provide good routing performance without anchor fingerprints. We therefore remove the anchor queries and behavioral fingerprints and represent each model with a learnable model token.
With this simpler representation, RTP achieves the highest overall APGR in Tables 1 and 2 (0.7185 [0.6926, 0.7446]). It also performs best in all 13 pool states of Table 3. As a result, the final method no longer depends on the number of anchors, or their sampling strategy. We will update the method description accordingly.
To show the revised system's overall performance, Table 1 compares RTP with the baselines under their native policies:
Weakness 2 and Q2. L2 probes versus L1 confidence.
Thanks for this question. We first examine which signal suits each decision in general. We then compare the two escalation signals directly within RTP.
Signal analysis. On our benchmark, we compare request-side methods, which read only the request text, with inference-side methods, which read the prefill hidden states, on both decisions of a cascade:
The per-model BERT predictor is a separate classifier for each small model. It predicts that model's correctness from the request text and is trained on historical correctness labels. The self-probe reads the model's own prefill hidden states. The PIKA utility router prefills all seven candidates and selects the model with the highest predicted success. AUROC is computed within each dataset and then averaged over the datasets and the seven models.
The results show that self-probes predict correctness much more accurately than request-side predictors. However, using them for model selection requires prefilling every candidate, which incurs substantial overhead. Even with this extra cost, the PIKA utility router does not outperform request-side routing.
These findings motivate our two-stage design. L1 selects a small model from request-side signals, while L2 uses the selected model's probe to decide whether to escalate. We test the benefit of probe-based escalation directly within RTP below.
Direct comparison within RTP. The three configurations below separate the training objective of L1 from the escalation signal: