acceptodds
Under review as a conference paper at ICLR 2027

Route-then-Probe: A Two-Stage Cascade for Cost-Effective LLM Routing

Abstract

Routing algorithms reduce the cost of LLM service by selecting a suitable model for each request from a pool of models with varying sizes, capabilities, and prices. We observe that a well-matched small model can answer many requests at a fraction of the cost of a flagship model. However, even a well-matched small model can fail, so realizing these savings requires two decisions: which small model to select and whether to escalate to a stronger model. We propose Route-then-Probe (RTP), a two-stage cascade that first selects exclusively among small models and then uses the inference-time signals of the selected model to decide whether to escalate. In a single forward pass, a lightweight router jointly encodes each request and the features of all candidate small models to select the model most likely to answer correctly. A calibrated probe gate then examines the hidden states of the selected model and determines, before decoding, whether the estimated accuracy gain from escalation justifies its cost. Existing routers typically select directly from a mixed pool of small and flagship models using request-side evidence alone, which may be insufficient to predict per-request success. Hidden-state probes provide model-specific correctness signals, but probing every candidate incurs substantial inference overhead and requires access to the internals of each model. RTP exploits complementarity between small models while probing only the selected model, avoiding the overhead of probing the entire pool. Experiments with seven self-deployed small models and an API-based flagship model on a mixed dataset spanning more than ten domains show that RTP achieves better cost–accuracy trade-offs than existing routing and cascading methods across the evaluated cost range. Its modular architecture also accommodates existing routers as first-stage selectors, extending the benefits of selective probing to other routing algorithms.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, model cascades, cost-efficient inference

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_6SnMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
3(+1)
Presentation
2(+1)
Soundness
2
Read review
Paper summary

This paper proposes Route-then-Probe (RTP), a two-stage framework for cost-effective LLM routing. RTP first selects a suitable small model based on query–model compatibility, then probes its prefill hidden states to determine whether escalation is needed. Experiments across diverse benchmarks demonstrate superior cost–accuracy trade-offs over existing baselines.

Critical strengths
  1. The paper presents a practical two-stage routing framework that combines the complementary strengths of multiple small models with inference-time correctness signals for selective escalation.
  2. The experiments cover diverse tasks and competitive baselines, demonstrating consistent improvements in cost–accuracy trade-offs and the applicability of the proposed framework to existing routing methods.
Critical weaknesses
  1. Insufficient justification and analysis of behavioral fingerprints. The first-stage router represents each candidate model using a behavioral fingerprint constructed from 216 randomly sampled training queries. However, the paper does not explain why this particular number is chosen or whether random sampling provides sufficient coverage of diverse model capabilities. Since the datasets vary substantially in size and task characteristics, randomly selected anchors may underrepresent certain specialized capabilities. Moreover, the model representation combines the fingerprint with a learnable model-specific embedding, making it unclear how much the fingerprint itself contributes to routing performance. Additional ablations on fingerprint size, sampling strategies, and the contribution of model identity embeddings would strengthen this component.
  2. Insufficient evidence for the necessity of the L2 probe. A central motivation of the paper is that inference-time hidden states provide more reliable correctness signals than query-side predictions. However, the L1 router already produces per-model correctness scores, which could potentially serve as confidence estimates for escalation decisions. The paper does not directly compare escalation based on calibrated L1 confidence with escalation based on L2 probe confidence under the same routing framework. Consequently, it remains unclear how much additional information the hidden-state probe provides beyond the first-stage predictions, particularly considering its additional prefill overhead for escalated requests. Such a comparison is important for establishing the necessity and effectiveness of the proposed two-stage design.
  3. Limited justification and evaluation of the cost-aware escalation rule. The L2 escalation rule compares the estimated failure probability with a threshold proportional to the ratio between small-model and flagship costs. However, the theoretical motivation for this particular formulation is insufficiently discussed. For example, increasing the flagship cost reduces the threshold and can make escalation more likely, which appears counterintuitive from a cost-optimization perspective. Furthermore, the rule assumes that the flagship always answers correctly, although its reported accuracy is below 100%. While the paper reports cost–accuracy curves and APGR, additional comparisons at fixed cost budgets or fixed accuracy targets, together with ablations of alternative escalation rules, would better establish its practical effectiveness.
Questions for authors

Here I list the questions I am most concerned about. Please see the Weaknesses section for more details.

  1. Why were 216 randomly sampled queries selected, and how sensitive is L1 performance to the number and sampling strategy of anchor queries, particularly when model identity embeddings are removed?
  2. How does escalation using calibrated L1 confidence compare with the proposed L2 probe under identical routing and cost constraints, and do the two signals provide complementary information?
  3. What motivates the proposed cost-ratio threshold, and how does it compare with alternative decision rules based on expected accuracy improvement and incremental inference cost?

I would be willing to raise my score if the authors can adequately address these concerns, as I believe the proposed two-stage framework is practically meaningful and shows promising empirical results, but stronger evidence is needed to justify its key design choices and establish the contributions of individual components.

4 comments

Response to Reviewer 6SnM, part 1 of 2.

We thank the reviewer for the thoughtful comments and suggestions. Our general response, "Method Revisions and Updated Results", summarizes the revised method and the updated results. We address your specific questions below.

Weakness 1 and Q1. Anchor fingerprints.

Thanks for this comment. With the revised L1, we find that model embeddings learned during training provide good routing performance without anchor fingerprints. We therefore remove the anchor queries and behavioral fingerprints and represent each model with a learnable model token.

With this simpler representation, RTP achieves the highest overall APGR in Tables 1 and 2 (0.7185 [0.6926, 0.7446]). It also performs best in all 13 pool states of Table 3. As a result, the final method no longer depends on the number of anchors, or their sampling strategy. We will update the method description accordingly.

To show the revised system's overall performance, Table 1 compares RTP with the baselines under their native policies:

Method (native policy)APGR [95% CI]
RTP (ours)0.7185 [0.6926, 0.7446]
Avengers-Pro0.6004 [0.5681, 0.6317]
PrefillRouter0.5696 [0.5306, 0.6080]
PIKA0.5637 [0.5278, 0.5968]
FrugalGPT0.5583 [0.5245, 0.5899]
CascadeRouting0.5222 [0.4791, 0.5643]
kNN-Router0.5152 [0.4750, 0.5557]
RouteLLM0.5082 [0.4720, 0.5431]
IRT-Router0.4803 [0.4437, 0.5153]

Weakness 2 and Q2. L2 probes versus L1 confidence.

Thanks for this question. We first examine which signal suits each decision in general. We then compare the two escalation signals directly within RTP.

Signal analysis. On our benchmark, we compare request-side methods, which read only the request text, with inference-side methods, which read the prefill hidden states, on both decisions of a cascade:

Task (metric)Request-side methodResultInference-side methodResult
Correctness prediction (AUROC)Per-model BERT predictor0.6518Self-probe0.7777
Model selection (accuracy of the selected model)Single BERT router78.69%PIKA utility router (Lugoloobi et al., 2026)78.27%

The per-model BERT predictor is a separate classifier for each small model. It predicts that model's correctness from the request text and is trained on historical correctness labels. The self-probe reads the model's own prefill hidden states. The PIKA utility router prefills all seven candidates and selects the model with the highest predicted success. AUROC is computed within each dataset and then averaged over the datasets and the seven models.

The results show that self-probes predict correctness much more accurately than request-side predictors. However, using them for model selection requires prefilling every candidate, which incurs substantial overhead. Even with this extra cost, the PIKA utility router does not outperform request-side routing.

These findings motivate our two-stage design. L1 selects a small model from request-side signals, while L2 uses the selected model's probe to decide whether to escalate. We test the benefit of probe-based escalation directly within RTP below.

Direct comparison within RTP. The three configurations below separate the training objective of L1 from the escalation signal:

L1 training objectiveEscalation signalAPGR [95% CI]
Lbce+Llist\mathcal{L}_{\mathrm{bce}}+\mathcal{L}_{\mathrm{list}}L1 correctness confidence0.6360 [0.6091, 0.6634]
Lbce+Llist\mathcal{L}_{\mathrm{bce}}+\mathcal{L}_{\mathrm{list}}Probe of the selected model0.6858 [0.6569, 0.7119]
Lbce+Llist+Lroute\mathcal{L}_{\mathrm{bce}}+\mathcal{L}_{\mathrm{list}}+\mathcal{L}_{\mathrm{route}}Probe of the selected model0.7185 [0.6926, 0.7446]
  • Same model selections and escalation rule. The first two rows use the same L1. Their selections, escalation rule, and cost accounting are identical. Only the escalation signal differs. Even after the L1 correctness confidence is calibrated with isotonic regression, the probe improves APGR by 4.98 APGR points. The hidden-state probe therefore provides information beyond the first-stage predictions.
  • Complementarity of the two signals. The third row brings probe feedback into L1 training (Eqs. (G2) and (G3) in the general response). This further increases APGR to 0.7185. In this design, L1's routing output is a selection distribution, not a correctness probability. It is therefore not used as escalation confidence. Instead, the two signals play complementary roles. Request-side signals make selection cheap. Inference-side probes provide more reliable correctness evidence. Training coordinates the two stages, and only the selected small model is run and probed online.

Response to Reviewer 6SnM, part 2 of 2.

Weakness 3 and Q3. Cost-aware escalation.

Thanks for pointing this out. There was a mistake in our original escalation rule. We have therefore revised the escalation condition and added a flagship output-length regressor (Eq. (G1) in the general response). The revision makes two changes:

  • Incremental cost instead of a cost ratio. When L2 decides, the selected small model has already completed its prefill. This prefill is a sunk cost shared by both branches. The decision should therefore compare the cost of calling the flagship with the remaining decoding cost of the small model, rather than the ratio of their total costs.
  • Per-request estimate of the flagship cost. The flagship's output price is three times its input price. Its output length also varies widely across requests, so a fixed constant cannot capture the escalation cost of each request. The new ModernBERT regressor therefore predicts this length from the request. Its inference cost is included in the total cost.

With these changes, for a fixed λ\lambda, a more expensive flagship or cheaper small-model decoding makes escalation harder. The rule is now consistent with cost optimization.

To evaluate the rule at fixed cost budgets, we read the accuracy and escalation rate from RTP's cost–accuracy curve. Each budget caps the average cost at a fraction of the flagship's average cost:

Average-cost cap (relative to the flagship)AccuracyEscalation rate
10%81.98%9.3%
20%85.60%22.4%
30%88.28%37.0%
50%90.18%51.2%
Always calling the flagship92.26%100%

At no more than half of the flagship's average cost, RTP reaches 90.18% accuracy. This is close to the 92.26% obtained by always calling the flagship. RTP's overall APGR is 0.7185.

Finally, we also tried several alternative rules that trade off expected accuracy improvement against incremental cost. Their performance was similar. In each rule, λ\lambda controls this tradeoff and can be adjusted to reach the desired accuracy and escalation rate.

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts in addressing my concerns.

One minor concern remains: I hope the authors can clarify how the revised escalation rule accounts for the fact that the flagship model is not always correct, and provide the quantitative comparison with alternative escalation rules in the final version.

Overall, I believe the paper has been substantially improved, and I have decided to raise my score from 2 to 3.

We thank the reviewer for the comments. The term 1−v~m∗(x)1-\tilde{v}_{m^{\ast}}(x) also represents the selected model's failure risk. We will clarify this and provide quantitative comparisons with alternative escalation rules in the revised paper.

Reviewer_qXgnOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Route-then-Probe combines request-side routing with inference-time probing. A first-stage router uses the request and model fingerprints to select one small model. A calibrated probe then reads that model's prefill hidden state and decides whether to escalate to an API-based flagship. This avoids probing the entire model pool while exploiting small-model complementarity. Experiments span 11 datasets, seven small models, and one flagship; the method achieves an APGR of 0.6463 versus 0.6026 for the strongest baseline, and the probe stage also improves other routers.

Critical strengths
  1. A practical and modular architecture. Selecting a small model before probing it is a sensible way to combine inexpensive request-side signals with model-specific evidence. The design avoids full-pool probing and does not require access to the flagship's hidden states, making it compatible with API-based models.

  2. Useful empirical coverage and component comparisons. The paper evaluates several routing and cascading baselines, substitutes alternative first-stage routers, and considers 13 model-pool states. Table 2 supports the usefulness of the probe stage beyond the proposed first-stage router, which strengthens the practical contribution.

Critical weaknesses
  1. The monetary cost comparison needs a sensitivity check for routing overhead. Appendix D already measures added latency: the five-member first-stage ensemble runs in parallel on five GPUs and takes about 5.0 ms. However, Table 6 approximates the BERT-based routing charge as zero. Low parallel latency does not quantify the compute cost per query, so the effect of this approximation on the cost-accuracy comparison remains unclear. Several baselines also use ensembles; the issue is consistent accounting across methods, not evidence of an unfair comparison. A representative nonzero routing-cost sensitivity check would clarify whether the reported advantage is robust to this approximation.

  2. Uncertainty in the reported performance differences is not quantified. Tables 1-3 give APGR point estimates. Averaging five trained routers into one deployed ensemble, as described in Appendix B, does not provide an uncertainty interval for the comparison. This matters particularly for smaller component gains, such as 0.6463 versus 0.6377 in Table 2. Paired uncertainty estimates would help distinguish reliable improvements from differences that could depend on the test sample. The current results demonstrate positive point estimates, not that the method is unstable.

Questions for authors
  1. How would the cost-accuracy comparison change under a representative nonzero charge for the first-stage router, applied consistently to the relevant baselines? The existing latency measurements are useful; GPU time per query or a simple cost sensitivity range would address the remaining accounting question.

  2. Can the authors provide paired uncertainty estimates for the principal APGR differences and the smaller component gains, for example by resampling test requests within datasets? This would complement, rather than replace, the existing five-seed ensemble results.

3 comments

We thank the reviewer for the thoughtful comments and suggestions. Our general response, "Method Revisions and Updated Results", summarizes the revised method and the updated results. We address your specific questions below.

Weakness 1 and Q1. Nonzero routing cost.

Thanks for this suggestion. We agree that the routing overhead should be charged rather than approximated as zero. In the revised evaluation, we charge L1 and the flagship output-length regressor according to their GPU time. We then add this charge to the cost side of APGR. On the same hardware, L1 takes 4.74 ms per request on average, including tokenization. The length regressor takes 4.55 ms. Because these charges are small, including them has little effect on the results. The main conclusions remain unchanged. The updated Table 1 with confidence intervals is as follows:

Method (native policy)APGR [95% CI]
RTP (ours)0.7185 [0.6926, 0.7446]
Avengers-Pro0.6004 [0.5681, 0.6317]
PrefillRouter0.5696 [0.5306, 0.6080]
PIKA0.5637 [0.5278, 0.5968]
FrugalGPT0.5583 [0.5245, 0.5899]
CascadeRouting0.5222 [0.4791, 0.5643]
kNN-Router0.5152 [0.4750, 0.5557]
RouteLLM0.5082 [0.4720, 0.5431]
IRT-Router0.4803 [0.4437, 0.5153]

The intervals are obtained from 2,000 paired bootstrap resamples stratified by dataset, as described below.

Weakness 2 and Q2. Paired uncertainty estimates.

Thanks for this suggestion. We follow the resampling scheme you suggested. Keeping the trained models fixed, we resample the test requests with replacement within each dataset 2,000 times. All methods share the same resampled requests. On each resample, we recompute the cost–accuracy curves and APGR. The table below reports the paired differences between RTP and the second-best method in each table:

ResultSecond-best methodAPGR difference [paired 95% CI]
Table 1: native policiesAvengers-Pro+0.1181 [+0.0878, +0.1481]
Table 2: each L1 with the same L2PrefillRouter+0.0137 [+0.0019, +0.0260]
Table 3: mean over 13 pool statesAvengers-Pro+0.1930 [+0.1716, +0.2149]

For Table 2, the non-deployable domain oracle is not counted as the second-best method.

Thank you for the response and additional experiments. The paired bootstrap analysis addresses my statistical uncertainty concern, and including routing overhead substantially addresses my cost concern. Please clarify the GPU pricing and parallel ensemble accounting in the final paper, and distinguish the revised method from the original submission. I maintain my overall rating.

We thank the reviewer for the comments. We will improve our paper following the suggestions.

Reviewer_KAGDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Route-then-Probe (RTP), a two-stage framework for cost-effective LLM routing. The first stage employs a lightweight encoder to select a suitable small language model based on request features and model-specific representations. The second stage uses a calibrated probe on the selected model's prefill hidden states to determine whether the request should be escalated to a stronger model. Experiments across eleven datasets show that RTP achieves better cost–accuracy trade-offs than several existing routing and cascading methods.

Critical strengths

The paper addresses a practically relevant problem in LLM deployment, particularly balancing inference costs and response quality across heterogeneous model pools.

Critical weaknesses

Limited methodological novelty and insufficient fundamental contributions. The proposed framework mainly combines two existing techniques: request-based model routing and hidden-state probing for correctness prediction. Both have been extensively investigated in prior work. The first-stage router uses conventional joint encoding with binary cross-entropy and listwise ranking losses, while the second stage relies on PCA, logistic regression, and probability calibration. The two-stage integration represents a reasonable engineering design, but the paper does not introduce a fundamentally new learning mechanism or theoretical insight. The authors should more clearly explain what novel knowledge this work contributes to the machine learning community beyond combining existing routing and probing methods.

The experimental evidence does not sufficiently establish the advantage of the proposed components. Although RTP achieves an APGR of 0.6463 compared with 0.6026 for the strongest baseline under its native routing policy, the performance gap becomes substantially smaller when existing routers are incorporated into the same two-stage architecture. In Table 2, PrefillRouter combined with the proposed L2 gate achieves an APGR of 0.6377, compared with 0.6463 for RTP. This difference is relatively small, yet the paper provides no confidence intervals or statistical significance analysis for this comparison. Moreover, the individual contributions of the proposed model representations, joint encoder, and training objectives are not adequately isolated. Stronger ablation studies are necessary to demonstrate that the proposed design provides meaningful improvements over simpler alternatives.

Poor figure presentation and insufficient attention to manuscript formatting. The presentation quality does not meet the level of clarity expected for a conference submission. Several figures contain small labels and crowded visual elements that make the results difficult to interpret. These presentation issues unnecessarily increase the effort required to assess the work and suggest insufficient attention to manuscript preparation.

Questions for authors

What are the fundamental methodological contributions of RTP beyond integrating existing request-based routing and hidden-state probing techniques? Can the authors demonstrate advantages that cannot be achieved through a straightforward combination of existing methods?

Given the relatively small performance difference between RTP and PrefillRouter with the same L2 gate, can the authors provide statistical significance tests and more comprehensive ablations to isolate the contribution of the proposed L1 architecture?

2 comments

Response to Reviewer KAGD, part 1 of 2.

We thank the reviewer for the thoughtful comments and suggestions. Our general response, "Method Revisions and Updated Results", summarizes the revised method and the updated results. We answer your questions below.

Weakness 1 and Q1. Methodological contribution.

Thanks for this question. Our design is based on an analysis of request-side and inference-side signals. The revised L1 further incorporates probe-aware supervision to better coordinate model selection with the subsequent probe-based escalation decision.

Signal analysis. Serving requests with small models involves two decisions: which small model to select and whether to escalate to the flagship. Each decision can rely on request-side signals, which read only the request text, or on inference-side signals, which read the prefill hidden states of a small model. To find out which signal suits each decision, we compare representative methods of both kinds on our benchmark:

Task (metric)Request-side methodResultInference-side methodResult
Correctness prediction (AUROC)Per-model BERT predictor0.6518Self-probe0.7777
Model selection (accuracy of the selected model)Single BERT router78.69%PIKA utility router (Lugoloobi et al., 2026)78.27%

The per-model BERT predictor is a separate classifier for each small model. It predicts that model's correctness from the request text and is trained on historical correctness labels. The self-probe reads the model's own prefill hidden states. The PIKA utility router prefills all seven candidates and selects the model with the highest predicted success. AUROC is computed within each dataset and then averaged over the datasets and the seven models.

The results show that self-probes predict correctness much more accurately than request-side predictors. However, using them for model selection requires prefilling every candidate, which incurs substantial overhead. Even with this extra cost, the PIKA utility router does not outperform request-side routing.

This asymmetry motivates our architecture. L1 selects a small model from request-side signals. L2 then uses the probe of the selected model to decide whether to escalate. The selected model prefills the request anyway to answer it. Reading its probe therefore adds almost no cost. In this way, RTP uses inference-side signals where they help most, without prefilling the whole pool.

Probe-aware training. In this architecture, the model chosen by L1 also determines which probe makes the escalation decision. A router trained only on correctness labels ignores this coupling. We therefore add the routing loss to the original BCE and ListNet objectives (Eqs. (G2) and (G3) in the general response). When some models answer correctly, the target favors correct models whose probes are more confident. When all models fail, it favors models whose probes more clearly recognize the failure. As a result, L1 considers both model capability and probe reliability when routing. The probe feedback is used only during offline training. At inference, L1 still reads only the request.

Component ablation. To isolate these components, we compare three configurations. They differ in the training objective of L1 and in the escalation signal:

L1 training objectiveEscalation signalAPGR [95% CI]
Lbce+Llist\mathcal{L}_{\mathrm{bce}}+\mathcal{L}_{\mathrm{list}}L1 correctness confidence0.6360 [0.6091, 0.6634]
Lbce+Llist\mathcal{L}_{\mathrm{bce}}+\mathcal{L}_{\mathrm{list}}Probe of the selected model0.6858 [0.6569, 0.7119]
Lbce+Llist+Lroute\mathcal{L}_{\mathrm{bce}}+\mathcal{L}_{\mathrm{list}}+\mathcal{L}_{\mathrm{route}}Probe of the selected model0.7185 [0.6926, 0.7446]

The first two rows use the same L1, so their selections are identical. The L1 correctness confidence is calibrated with isotonic regression. All rows share the same escalation rule and cost accounting.

  • Effect of probe verification. Replacing only the escalation signal raises APGR from 0.6360 to 0.6858, a gain of 4.98 APGR points.
  • Effect of probe-aware routing. With the probe-aware L1, APGR further increases to 0.7185, 3.27 points higher than with the BCE + ListNet router.

The second row corresponds to a straightforward combination of an independently trained router and a probe. The third row goes one step further. It feeds the verification behavior of the probe back into router training. This provides selection supervision that the straightforward combination lacks, without any online computation on candidate models.

Response to Reviewer KAGD, part 2 of 2.

Weakness 2 and Q2. Component gains and statistical significance.

Thanks for this suggestion. For statistical significance, we use 2,000 paired bootstrap resamples stratified by dataset. All methods share the same resampled requests. The cost–accuracy curves and APGR are recomputed on each resample. The updated Table 1 reports each method under its native policy:

Method (native policy)APGR [95% CI]
RTP (ours)0.7185 [0.6926, 0.7446]
Avengers-Pro0.6004 [0.5681, 0.6317]
PrefillRouter0.5696 [0.5306, 0.6080]
PIKA0.5637 [0.5278, 0.5968]
FrugalGPT0.5583 [0.5245, 0.5899]
CascadeRouting0.5222 [0.4791, 0.5643]
kNN-Router0.5152 [0.4750, 0.5557]
RouteLLM0.5082 [0.4720, 0.5431]
IRT-Router0.4803 [0.4437, 0.5153]

Under native policies, RTP outperforms the strongest baseline by a clear margin. The gap becomes smaller when other routers are placed in our two-stage architecture. Table 2 reports this setting, where each L1 selects among the seven small models and then uses the same L2:

L1 (with the same L2)APGR [95% CI]≤10%≤20%≤30%≤50%
RTP (ours)0.7185 [0.6926, 0.7446]0.29970.43690.53250.6522
PrefillRouter0.7049 [0.6787, 0.7304]0.28240.42010.51300.6364
kNN-Router0.6981 [0.6705, 0.7232]0.25990.40490.50350.6282
EmbedLLM0.6949 [0.6680, 0.7198]0.24700.39420.49350.6220
Avengers-Pro0.6946 [0.6682, 0.7214]0.27300.41020.50250.6242
IRT-Router0.6878 [0.6620, 0.7127]0.22640.38030.48630.6155
Model-SAT0.6816 [0.6553, 0.7062]0.20030.36390.47200.6063
ICL-Router0.6582 [0.6325, 0.6836]0.04090.26450.40720.5721
Domain oracle (not deployable)0.7141 [0.6878, 0.7401]0.30350.43530.52690.6465

RTP still improves APGR over the strongest deployable alternative, PrefillRouter + L2, by +0.0137 (paired 95% CI [+0.0019, +0.0260]). The smaller gap in Table 2 mainly reflects the benefit of the architecture itself. The two-stage architecture gives existing routers consistent and substantial gains. When PrefillRouter, kNN-Router, Avengers-Pro, and IRT-Router select only among the small models and let L2 decide escalation, their APGR increases by about 13.5, 18.3, 9.4, and 20.8 APGR points over their native policies. Probe-aware training then brings a further gain on top of this architecture.

Weakness 3. Figures and presentation.

Thanks for this suggestion. We will improve the figures and the presentation in the revised paper as soon as possible.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Method Revisions and Updated Results

We thank all reviewers for their thoughtful comments and suggestions. Based on their feedback and new experimental findings, we have refined the L2 escalation rule and the L1 training objective. We have re-evaluated RTP with these changes and summarize the revisions and results below. These results will be added to our paper very soon.

1. Revised L2 escalation rule with flagship output-length prediction. Under the revised rule, Eq. (G1), we escalate when the selected model's estimated failure probability exceeds λ\lambda times the incremental cost:

\tag{G1}$$ where $\delta_m(x)=u(x)\\,c_F^{\mathrm{in}}+\hat{\ell}_F(x)\\,c_F^{\mathrm{out}}-\hat{\ell}_m(x)\\,c_m^{\mathrm{out}}$ is the estimated incremental cost. Here, $m^{\ast}(x)$ is the small model selected by L1 for request $x$. $\tilde{v}_m(x)$ is the probability that model $m$ answers $x$ correctly, as predicted by its L2 probe. $u(x)$ is the input token count. $\hat{\ell}_F(x)$ is the flagship's output token count predicted by a new ModernBERT regressor. $\hat{\ell}_m(x)$ is the estimated output token count of small model $m$ and we use its training-set mean. $c^{\mathrm{in}}$ and $c^{\mathrm{out}}$ are the LLM's input and output prices. $\lambda\ge 0$ is the cost preference. The incremental cost is **the flagship's cost minus the small model's remaining decoding cost**. The selected model has already completed prefill. This is a sunk cost shared by both branches. It remains in the actual bill but does not affect the decision. For a fixed $\lambda$, a more expensive flagship or cheaper small-model decoding makes escalation harder. **2. Probe-aware routing objective for L1.** **L1 consider both model capability and probe reliability when routing.** We keep the original BCE and ListNet objectives while adding a routing head that determines the model choice. Its target is given by Eq. (G2): $$s_{i,m}= \begin{cases} \dfrac{y_{i,m}\exp(q_{i,m}/T)}{\sum_{j=1}^{M}y_{i,j}\exp(q_{i,j}/T)}, & i\in\mathcal{P},\\\\[12pt] \dfrac{\exp(-q_{i,m}/T)}{\sum_{j=1}^{M}\exp(-q_{i,j}/T)}, & i\notin\mathcal{P}. \end{cases} \tag{G2}$$ Here, $M$ is the number of candidate small models. $y_{i,m}\in\\{0,1\\}$ indicates whether model $m$ answers training request $x_i$ correctly. $\mathcal{P}$ contains the indices of requests that at least one small model answers correctly. $q_{i,m}$ is the probe's uncalibrated out-of-fold correctness probability for model $m$, computed with five grouped folds so the predicting probe never sees $x_i$ or its duplicates. $T$ is a temperature parameter that sets the target's sharpness. $s_{i,m}$ is the normalized target probability assigned to model $m$ for $x_i$. - **When some small models answer correctly**, the target assigns mass only to correct models. It favors those whose probes are more confident. This reduces unnecessary escalation. - **When all small models fail**, the target favors models whose probes more clearly recognize failure. This makes escalation more likely. The routing loss, Eq. (G3), is the soft-target cross-entropy: $$\mathcal{L}_{\mathrm{route}}=-\frac{1}{N}\sum_{i=1}^{N}\sum_{m=1}^{M}s_{i,m}\log\frac{\exp(r_{i,m})}{\sum_{j=1}^{M}\exp(r_{i,j})}, \tag{G3}$$ where $N$ is the number of training requests. $r_{i,m}$ is the routing head's score for model $m$ on $x_i$. The total loss is $\mathcal{L}=\mathcal{L}_{\mathrm{bce}}+\mathcal{L}_{\mathrm{list}}+\mathcal{L}_{\mathrm{route}}$. $\mathcal{L}_{\mathrm{bce}}$ and $\mathcal{L}_{\mathrm{list}}$ are the original BCE and ListNet losses. All three losses update the encoder and model tokens. **At inference, L1 reads only the request. It does not run any candidate model in advance. Its routing output is a selection distribution, not a correctness probability.** L2 reads only the selected model's prefill hidden states to estimate correctness. **3. Updated results.** The revised RTP achieves an APGR of **0.7185 [0.6926, 0.7446]**. Tables 1–3 include the computation costs of L1 and the length regressor. We also add paired confidence intervals. We address each reviewer's questions in the individual responses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.