acceptodds
Under review as a conference paper at ICLR 2027

Probe Before You Route: LLM Selection via Test-Time Probing

Abstract

Large language models (LLMs) vary substantially in capability and inference cost, making it important to select an appropriate model for each query. Conventional LLM routers select a model before any candidate model generation, potentially lowering routing performance because they overlook informative signals in the generated tokens. We therefore propose \bf Probe-then-Route, which first probes each candidate LLM by allowing it to generate a small number of output tokens for the query and then selects one model based on the query together with the generated tokens. To reduce the additional inference cost of probing every candidate, we further introduce a variant, dubbed \bf Selective Probe-then-Route, that first selects a candidate subset, probes only those candidates, and routes among them using the same trained router without retraining. We evaluate both methods using eight heterogeneous LLMs on 12 benchmarks spanning six domains, with one in-distribution (ID) and one out-of-distribution (OOD) benchmark per domain. Probing all eight candidates with up to eight output tokens per candidate improves average ID accuracy by 2.73% over the strongest baseline and outperforms all baselines on three of six ID benchmarks. \bf Probe-then-Route also generalizes effectively out of distribution, achieving the highest accuracy on two of six OOD benchmarks, including one tie, and remaining competitive on the others. With \bf Selective Probe-then-Route, additional inference cost can be reduced by 56.4% on ID benchmarks while retaining 83.3% of \bf Probe-then-Route's accuracy gain over the strongest baseline and still outperforming all baselines across three ID benchmarks.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM routing, large language models, LLM selection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_89YgOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes running candidate LLMs for a short prefix before choosing which response to complete. A router predicts correctness and the remaining generation cost based on the query and candidate prefixes. A selective variant uses a query-only router to reduce the number of candidates probed. Experiments cover eight models and twelve benchmarks. The reported ID gain over Lookahead is 2.03 percentage points at 8 tokens per candidate, while OOD average accuracy remains below that of the strongest baselines.

Critical strengths

Short-prefix probing is a practical way to acquire response information without generating every candidate answer in full. The paper extensively investigates probe length and candidate pruning, includes a useful no-prefix ablation, and reports both probing costs and unfavorable OOD results rather than presenting only the best cases.

Critical weaknesses

The accuracy gain does not yet justify the extra inference cost. Even an eight-token probe requires processing the prompt on every retained model. The paper includes input-token charges but does not show how the trade-off varies with prompt length or pool size when multiple prefills are required for the models. Full and selective probing improve ID accuracy by 2.03 and 1.69 points over Lookahead, while increasing total cost by 69% and 29%, respectively. Both main configurations are less accurate and more expensive on OOD data. The selective variant's headline saving of over 50% is offset by additional cost; its total ID saving over full probing is 24%. Given the incremental advance over response-informed routing, comparisons at matched total budgets and against simpler selectors using the same prefixes are central to establishing the contribution.

Questions for authors
  1. Does “selected on the ID test set” in Appendix A.2.5 accurately describe the implementation? If so, what results remain when temperature, tolerance, and probe length are selected exclusively on validation data and then fixed for an untouched evaluation, with paired uncertainty?
  2. Does probing still improve accuracy over a tuned routing or cascade baseline at matched total inference budgets? Please show how the trade-off changes with prompt length and the number of models probed, separating input, probe output, completion, and router costs, and report end-to-end latency and caching assumptions.
  3. What is the substantive advance over the cited Lookahead, FrugalGPT, and AutoMix? In particular, does the proposed architecture outperform Lookahead's actual-response classifier when both receive the same short prefixes, or simpler prefix-based answer selection?
  4. Were the experiments performed with actual interrupted/resumed API calls or by truncating stored complete responses? Please clarify prompts, decoding settings, correctness grading, and how continuation preserves the generation state.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 89Yg: Inference Cost, Latency, and Same-Prefix Comparisons

Thank you for asking about the selection protocol, inference costs, and comparisons using the same prefixes.

Selection protocol. Appendix A.2.5 should say that the temperature and δ\delta settings were selected on the ID validation set. We held them fixed for ID and OOD testing. The supplementary material shows the sweeps.

Accuracy at shared spending caps. We used the reported k=8k=8 ID API costs of selective and full probing as shared spending targets. Costs are USD per 1,000 queries. Our reply to Reviewer 2 describes the RadialRouter cascade.

ID spending targetMethodID macro accuracyRealized ID cost
0.342Selective Probe-then-Route76.07%0.342
0.342RadialRouter performance-first cascade73.67%0.320
0.450Probe-then-Route76.41%0.450
0.450RadialRouter performance-first cascade74.10%0.321

On ID, selective probing leads the cascade by 2.40 percentage points (pp) at the 0.342 target, and full probing leads by 2.31 pp at the 0.450 target. On OOD, using the same ID-tuned policies, selective probing reaches 69.17% vs. 68.95% for the cascade at the 0.342 target, and full probing reaches 69.04% vs. 68.88% at the 0.450 target.

Prompt length, probe count, and cost. The supplementary material shows input API cost by prompt length for all eight models. For continuation, the API charges for processing the query and prefix again and for generating the remaining tokens. We verified zero cache hits and used uncached input prices.

We report the cost of generating the selected response in one call and the additional API cost of probing and continuation. These aggregates do not itemize prompt-input, probe-output, and continuation charges for the original runs. At k=8k=8, the two components are 0.243 and 0.207 for full probing, and 0.252 and 0.090 for selective probing, in USD per 1,000 ID queries. The locally hosted controller incurs no separate API charge.

We reused two k=8k=8 selective-probing results to show how accuracy and cost change with fewer probes.

ConfigurationModels probed per ID queryID macro accuracyID additional costID total cost
Full Probe-then-Route876.41%0.2070.450
Top-7 selection776.13%0.1610.389
Adaptive Selective Probe-then-Route3.47 on average76.07%0.0900.342

Adaptive selection reduces additional cost by 56.4% and total cost by 24.0% relative to full probing, with a 0.34 pp drop in ID macro accuracy.

Latency. Estimated end-to-end latency for full probing at k=8k=8 was 13.646 s with warm-up and 14.484 s without warm-up. Our reply to Reviewer 3 describes both timing protocols.

Contribution and same-prefix comparison. Lookahead predicts response representations before candidate models generate tokens at inference. Its preliminary experiment tests a classifier that reads actual responses. FrugalGPT and AutoMix use completed answers to decide whether to invoke another model. Probe-then-Route uses short prefixes to choose which model finishes the answer. Like self-consistency (Wang et al., ICLR 2023) and multi-agent debate (Du et al., ICML 2024), it spends additional inference compute to improve accuracy. Shortlisting limits the roughly linear growth in probing cost with pool size.

Following Lookahead's response-classification procedure, we trained SmolLM2-135M and ModernBERT classifiers for four epochs on all ID training queries, using the same eight-token prefixes as Probe-then-Route. We chose checkpoints by ID validation macro accuracy. Prefix voting falls back to a random candidate (seed 42) when no usable answer appears.

Method given eight-token prefixesID macro accuracyOOD macro accuracy
Prefix voting with random fallback68.70%64.89%
Lookahead classifier (SmolLM2-135M)70.50%64.96%
Lookahead classifier (ModernBERT)73.08%68.34%
Probe-then-Route76.41%69.04%

Probe-then-Route leads the stronger Lookahead classifier by 3.33 pp on ID and 0.70 pp on OOD, and prefix voting by 7.71 pp on ID and 4.15 pp on OOD.

API procedure and caching. We prompted each candidate with the benchmark question, with no extra system prompt, at temperature 0 and seed 42. For continuation, we sent a separate request with the original question and generated prefix. We waited ten days between probing and continuation and confirmed zero cache hits and no reuse of cached decoding state. We used GPT-5.6 Luna with high reasoning effort to extract answers as given, without solving or correcting them, and then grade them using benchmark-specific criteria.

Reviewer_UvwAOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tests whether short observed outputs from candidate LLMs improve routing. It predicts correctness and continuation length from the query and candidate prefixes, then continues a low-cost model within a predicted-correctness tolerance; a selective variant probes a smaller pool. The evaluation uses eight models across six paired ID/OOD domains, including query-only routers and FrugalGPT.

Critical strengths

First, acquiring actual early candidate behavior is a meaningful design point between query-only selection and completed-answer cascading. It offers a concrete way to observe candidate-specific information before paying for complete answers. The prefix-processing ablation supports an ID benefit (76.41% versus 73.63% without prefixes, p.25), while its architecture changes limit the causal conclusion.

Second, the selective-probing and answer-exposure analyses reveal where that signal may be useful. The probe-length and subset sweeps, OOD tests, and separate selected/additional-cost accounting let readers inspect the tradeoff rather than relying on accuracy alone. Including FrugalGPT among the baselines also makes the comparison more informative.

Critical weaknesses

First, the evidence does not yet establish a favorable, general quality–cost regime. At k=8, full probing improves ID accuracy over Lookahead by 2.03 points but increases reported total token cost from USD 0.266 to USD 0.450/1,000 queries; selective probing reaches 76.07% at USD 0.342, while FrugalGPT reaches 74.30% at USD 0.322 (Tables 1–2). Selective Probe therefore gains 1.77 macro points over FrugalGPT at about 6.2% higher reported token cost, a tradeoff worth showing across budgets. On the reported k=8 OOD average, Lookahead reaches 69.37% at USD 0.151, compared with 69.04% at USD 0.345 for full Probe and 69.17% at USD 0.227 for selective Probe.

FrugalGPT is present, so the needed comparison is its competitive, tuned frontier and matched total-cost operating points, including controller and continuation costs. FrugalGPT's original work assesses task performance across spending budgets; one reimplemented point does not establish the relative frontier.

The result also depends on the target workload: equal-benchmark macro averaging gives 91-question GPQA the same weight as 4,500-question MATH (Table 12). Reweighting the rounded Table 1 values by test-query count leaves full Probe about 0.74 points above Lookahead and 0.12 above FrugalGPT. Equal domain weighting is defensible, but both views should be reported and their intended use explained. Other probe settings in Table 3 prevent a method-wide dominance claim. This practical concern does not establish statistical insignificance.

Second, the main operating point and gain need firmer validation. A.2.5 (p.22) describes ID-test-set selection over 742 temperature–delta pairs per k, whereas supplied notebook text specifies validation-based selection without saved run outputs. Please reconcile the exact protocol behind Tables 1–2/13 and use frozen validation selection if test labels chose the reported settings. Paired uncertainty matters because GPQA's 7.69-point difference is seven net answers among 91 questions and strongly influences the macro average; training-seed variability for the main comparison is also unreported. The code discrepancy is unresolved provenance, not proof of leakage.

Third, the mechanism remains uncertain. The p.25 no-prefix ablation also changes cross-attention, pooling, and the predictor MLP, so it cannot isolate the information in the actual prefix. Appendix A.3's complete-answer exposure analysis is useful, but correlational: a fixed-architecture shuffled-prefix or answer-masked control would test whether query-specific early behavior matters beyond format or answer exposure. Positioning the method against routing-plus-cascading approaches would also clarify when paying for partial outputs is preferable to evaluating completed candidates.

Questions for authors
  1. Which run artifacts and splits produced Tables 1–2/13? Under fixed validation selection, what are the paired intervals, both equal-benchmark and query-weighted summaries, and total-cost-matched curves against Lookahead and the existing FrugalGPT baseline?

  2. How are unfinished prefixes continued in inference, and what controller cost and latency does this require beyond the already charged candidate-token costs? At which matched budgets does partial probing outperform query-only routing or completed-answer cascading?

  3. With the predictor architecture fixed, would shuffled or answer-masked prefixes retain the gain beyond the answer-exposure analysis in A.3?

1 comment
AuthorsOpenReview ↗

Response to Reviewer UvwA: Evaluation Protocol, Paired Uncertainty, and Prefix Controls

Thank you for asking us to clarify the evaluation, compare costs and accuracy, and test what the prefixes contribute.

Run artifacts and splits. Tables 1 and 2 report 9,913 ID and 5,152 OOD test queries across six benchmarks per split. The ID dataset also has 31,457 training and 2,302 validation queries. Table 13 lists the inference settings. Both k=8k=8 methods use T=12.903900T=12.903900 and δ=0.001\delta=0.001. Appendix A.2.5 contains a typo. The sweep over temperature and δ\delta used the ID validation set.

Paired intervals and accuracy weighting. We ran 10,000 paired bootstrap replicates with seed 42, sampling with replacement within each benchmark. Each replicate preserved benchmark sizes and used the same query draws for both methods. We computed gaps by weighting each benchmark equally (macro) or each query equally. The 95% CIs use the 2.5th and 97.5th percentiles. The table compares full Probe-then-Route with each baseline.

SplitBaselineMacro accuracy (ours / baseline)Macro gap, pp [95% CI]Query-weighted accuracy (ours / baseline)Query-weighted gap, pp [95% CI]
IDLookahead76.41% / 74.38%+2.02 [−1.24, +5.20]85.65% / 84.91%+0.74 [−0.24, +1.69]
IDFrugalGPT76.41% / 74.30%+2.11 [+1.05, +3.31]85.65% / 85.52%+0.12 [−0.08, +0.32]
OODLookahead69.04% / 69.37%−0.34 [−2.10, +1.46]66.34% / 66.28%+0.06 [−1.40, +1.49]
OODFrugalGPT69.04% / 68.76%+0.28 [−0.03, +0.64]66.34% / 66.07%+0.27 [+0.06, +0.50]

GPQA. Probe-then-Route answers seven more of the 91 questions correctly than Lookahead (+7.69 pp). The paired 95% CI is [−7.69, +21.98] pp against Lookahead and [+5.49, +19.78] pp against FrugalGPT. Only the latter interval excludes zero.

We chose ID macro accuracy as our primary metric to give each domain equal weight. A router may serve a math tutor in one deployment and a science assistant in another. MATH and GPQA count equally in this metric despite having 4,500 and 91 test queries, respectively. Against FrugalGPT, the ID macro and OOD query-weighted CIs exclude zero. All four OOD point estimates are within 0.5 pp of zero.

Training-seed results. At k=8k=8, seeds 0, 1, and 2 give 76.461%, 76.404%, and 76.341% ID macro accuracy (mean 76.402%, sample SD 0.060 pp, range 0.120 pp). We reused seeds from the online-pruning experiment. These results measure variation across training runs. The paired CIs above measure variation across test queries. Our reply to Reviewer 1 gives the MMLU and GPQA scores.

On ID, full probing gains 0.12 query-weighted pp over FrugalGPT and spends more. Selective probing gains 0.29 pp (85.81% vs. 85.52%) with lower additional API cost (USD 0.090 vs. 0.107 per 1,000 queries). Against RadialRouter performance-first, selective probing gains 2.08 query-weighted pp on ID and trails by 0.52 pp on OOD.

Continuation and controller overhead. If the selected model has not finished, we send a second API request with the original query and prefix as an unfinished assistant message, then append the returned text. For Llama-3.1-8B-Instruct, we used stage_one_request. Continuation starts from the prefix text with a fresh decoding state. We waited ten days between probing and continuation to let any cache expire and verified zero cache hits.

The locally hosted controller incurs no separate API charge. Reported costs cover candidate-model API calls. On an NVIDIA A100 at k=8k=8, mean controller latency was 18.05 ms for full probing and 28.94 ms for selective probing with warm-up, and 856.14 ms and 943.13 ms without it. For warm-up, we shuffled all 9,913 ID test queries with seed 42, ran the first query once untimed, then timed all queries, including the first.

What the prefixes contribute. The no-prefix experiment on p. 25 is a query-only ablation. Both predictors see only the query and the candidate model's identity. We removed the prefix branch and narrowed the MLP input because even a blank prefix can attend to the query. We also retrained both predictors at k=8k=8 with the architecture fixed.

Prefix conditionID macro accuracyOOD macro accuracy
Original prefixes (reported reference)76.41%69.04%
Answer-masked prefixes75.77%68.61%
Shuffled prefixes74.20%67.90%

Masking annotated answer spans lowers macro accuracy by 0.64 pp on ID and 0.43 pp on OOD. Shuffling whole eight-model bundles within each benchmark and split preserves their distribution and model order while breaking the query–prefix match. Accuracy falls by 2.21 pp on ID and 1.14 pp on OOD. These controls indicate that query-specific prefixes provide routing signals beyond visible answers.

Reviewer_9qhCOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduce a new routing method named Probe-then-Route, which allows model to generate the prefix of rollout as additional input in routing process. Two variants of Probe-then-Route, including vanilla Probe-then-Route and selective Probe-then-Route are evaluated on 12 benchmarks in 6 domains with 8 frontier LLMs. For each domain, this work evaluates the performance of methond on two benchmarks, one is The evaluation results show vanilla method obtain the best performance in overall average and in three benchmarks, denote the effectiveness of Probe-then-Route. To evaluate the performance of Authors further deduct several ablation studies to learn the influences of several factors, including hyper parameter selection,

Critical strengths
  1. The paper proposes a new routing method, Probe-then-Route, which provides a useful middle ground between query-only routing and selection after full response generation. Compared with eight baselines, the method achieves the highest average accuracy on ID datasets and competitive performance on OOD datasets. This makes the basic idea interesting and potentially useful.

  2. This work also proposes a variant, Selective Probe-then-Route, which addresses an important practical limitation of the original approach by reducing the additional inference cost of probing.

Critical weaknesses
  1. The roubustness and generalization capability of proposed method is still unclear. Comparing with previous methods, the improvement of proposed method is mainly contributed by GPQA dataset in ID dataset and even worse than the baselines with less computational cost in OOD dataset. What's more, for the important hyperparameter kk, this work do not give a clear clafication on why select kk=8 and why such hyperparameter works.
  2. Limited contribution. Prior work such as Lookahead already motivates the use of response-related information for routing. Observing actual short prefixes from candidate models is a distinct and reasonable extension, but the current analysis does not sufficiently establish what additional generalizable insight this design provides.
Questions for authors
  1. Under a same cost budget, where do the proposed methods improve on the strongest baselines on ID and OOD data?
  2. Can authors provide further studies on the relationship between the improvement of this method and the generated prefixes? Especially on clarifying how much gains came from visible results in prefixes and how much gains came from predictive signals.
1 comment
AuthorsOpenReview ↗

Response to Reviewer 9qhC: Contribution, Prefix Length, and Accuracy-Cost Trade-offs

Thank you for asking where the gains come from and how they compare with routing and cascading baselines at similar budgets.

Contribution beyond Lookahead. Lookahead predicts response representations before candidates generate tokens. Probe-then-Route reads short candidate prefixes before selecting which model finishes the answer. It improves on our no-prefix ablation and gains 2.03 percentage points (pp) in ID macro accuracy over Lookahead at k=8k=8. Probing every candidate makes cost grow roughly linearly with pool size. Shortlisting cuts additional API cost by 56.4% relative to full probing. Future work could adapt the probe length to each query.

ID gains beyond GPQA and OOD performance. Full probing improves on Lookahead on five of six ID benchmarks. Across both probing variants, the best non-oracle scores are 83.80% on MMLU, 62.64% on GPQA, and 82.78% on LogicBench. Of the 73 additional ID queries full probing answers correctly over Lookahead, seven are from GPQA and 55 from LogicBench.

MethodID macro (%)ID query-weighted (%)OOD macro (%)OOD query-weighted (%)
Probe-then-Route76.4185.6569.0466.34
Selective Probe-then-Route76.0785.8169.1766.21
Lookahead74.3884.9169.3766.28
RadialRouter (performance-first)73.9183.7369.5366.73
FrugalGPT74.3085.5268.7666.07

Selective probing leads on ID query-weighted accuracy at 85.81%, 0.90 pp above Lookahead and 0.29 pp above FrugalGPT. Relative to RadialRouter performance-first, which leads on OOD query-weighted accuracy, selective probing gains 2.08 pp on ID and trails by 0.52 pp on OOD. Both probing variants are within 0.07 pp of Lookahead on OOD query-weighted accuracy.

Excluding GPQA leaves five ID benchmarks and 9,822 queries.

MethodID macro (%)ID query-weighted (%)Correct queries
Probe-then-Route79.1685.868,433
Selective Probe-then-Route79.2086.048,451
Lookahead78.2785.198,367
EmbedLLM78.5184.808,329
RadialRouter (performance-first)78.1584.028,252
FrugalGPT79.0585.858,432

Full and selective probing exceed EmbedLLM, the strongest routing baseline here, by 0.65 and 0.69 pp in ID macro accuracy. Selective probing retains the query-weighted lead at 86.04%, with 19 more correct answers than FrugalGPT. Full probing answers one more correctly.

Why eight tokens work. Of 13 probe lengths from 1 to 4,096 tokens, k=8k=8 gave the highest ID macro accuracy. Mean answer exposure across the six ID benchmarks rose from 5.36% at k=4k=4 to 20.06% at k=8k=8, giving the router more direct evidence about candidate answers. Longer prefixes exposed more answers and lowered training loss without consistent ID test gains, suggesting that overfitting may offset the benefit.

What the prefixes tell us. At k=2k=2, answer exposure is 4.91% ID and 0.75% OOD, versus 20.06% and 29.50% at k=8k=8. Full probing still reaches 75.52% ID / 68.97% OOD macro accuracy. Shortlisting with Lookahead before probing raises this to 75.93% / 69.39%, versus 74.38% / 69.37% for Lookahead alone. Masking annotated answer spans during training and testing at k=8k=8 with the same architecture retains 75.77% ID / 68.61% OOD, still 1.39 pp ahead of Lookahead on ID. These results indicate that short prefixes provide routing signals beyond visible answers.

Accuracy at shared API spending caps. We adapted the cascading approach of Dekoninck et al. (ICML 2025). The cascade starts with the top-ranked model from RadialRouter performance-first and uses FrugalGPT correctness predictors to accept or escalate. On escalation, it selects the highest-ranked remaining model estimated to fit the budget, or the cheapest remaining model if none fits. We estimate call cost using the no-prefix length predictor and model API token prices. Thresholds were selected on ID validation and held fixed for both test splits. Costs are USD per 1,000 queries.

MethodID capID macro (%)ID costOOD macro (%)OOD cost
RadialRouter (performance-first)—73.910.23369.530.172
Probe-then-Route0.45076.410.45069.040.345
RadialRouter cascade0.45074.100.32168.880.233
Selective Probe-then-Route0.34276.070.34269.170.227
RadialRouter cascade0.34273.670.32068.950.229

Full and selective probing lead the cascade on ID by 2.31 and 2.40 pp at their respective caps. At the 0.342 cap, selective probing also has higher OOD macro accuracy and slightly lower OOD cost than the cascade. Standalone RadialRouter remains more accurate and cheaper on OOD than either k=8k=8 probing variant. On OOD, the cascade lost more correct answers through escalation than it recovered.

Reviewer_eTgbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Probe-then-Route, where each candidate LLM first generates a few tokens (8 in the main results) and a ModernBERT-based predictor then picks the cheapest model among those predicted to be most accurate. A selective variant prunes candidates first so fewer models are probed. It improves average in-distribution accuracy by about 2 points over the best baseline, Lookahead. I think the question is reasonable, but the idea is a modest extension of Lookahead in my opinion - however I feel the paper has good practical value

Critical strengths
  • The paper addresses a practical question: can generating just a few tokens from each model improve routing without the cost of full responses? It builds on Lookahead and explores a cheaper way to use this information.

  • The ablation studies are detailed and useful. The results show that generated tokens improve accuracy (76.41% vs. 73.63%), and the authors test several architectural components. They also highlight cases where certain components, like cross-attention, do not help.

  • The evaluation is thorough, covering 12 benchmarks across six domains, eight models, and different token lengths. The authors are transparent that adding more tokens does not always improve accuracy and that OOD performance remains competitive rather than consistently better.

  • The cost analysis is practical and well presented. It considers real token prices, separates probing costs from model usage costs, and clearly shows the trade-off between accuracy and efficiency. The selective approach is also a useful addition, as it aims to reduce unnecessary model calls.

  • The paper is clear about its limitations. The authors avoid making unsupported causal claims and acknowledge the extra costs, weaker OOD improvements, and the lack of joint training for model selection and routing.

Critical weaknesses
  • In a lot of the cases, the gains look like noise, since there are no seeds, intervals or significance tests. Several wins are tiny (0.04 to 0.18 points on MMLU, and a 7-question gain on GPQA with n = 91). Can the authors report the seeds / confidence intervals? This will add more reliability to the results.

  • With 68% of MMLU answers already visible at k = 8, how much of the gain is just cross-model voting? Would a masking control or a majority-vote baseline show this? I would like the authors to answer this during the rebuttal.

  • I feel that the the OOD claims are a bit overstated, and it is unclear what the router is learning. PtR’s average OOD accuracy (69.04) is below Lookahead (69.37), RadialRouter (69.53), and Qwen3-14B (69.26). Can the authors comment on it?

Questions for authors

Please see Weakness.

1 comment
AuthorsOpenReview ↗

Response to Reviewer eTgb: Reliability, Answer Exposure, and OOD Performance

Thank you for asking us to quantify uncertainty, test whether visible answers explain the gains, and narrow our OOD claims.

Reliability across training seeds and test queries. The main Probe-then-Route run used training seed 42. For a separate k=8k=8 check, we trained with seeds 0, 1, and 2 and report all three runs. We reused the seed values from our Selective Probe-then-Route online-pruning experiment.

Training seedGPQA Main (91 queries)MMLU (2,846 queries)
056/91 (61.54%)2,361/2,846 (82.96%)
154/91 (59.34%)2,382/2,846 (83.70%)
255/91 (60.44%)2,393/2,846 (84.08%)
Mean ± sample SD60.44% ± 1.10 pp83.58% ± 0.57 pp

Separately, we estimated uncertainty in the main k=8k=8 comparison with 10,000 paired bootstrap replicates (resampling seed 42). We matched the observed correctness outcomes by query. For each replicate, we sampled the original number of test queries within each benchmark with replacement, used the same draws for both methods, and averaged the six benchmark gaps for the ID macro result. The table gives the 2.5th and 97.5th percentiles in percentage points.

ID benchmark or summaryProbe-then-Route − Lookahead (95% CI)Probe-then-Route − FrugalGPT (95% CI)
MMLU[−1.90, +1.97][+0.91, +1.72]
GPQA Main[−7.69, +21.98][+5.49, +19.78]
ID macro[−1.24, +5.20][+1.05, +3.31]

Probe-then-Route answered seven more GPQA questions correctly than Lookahead. The intervals against Lookahead include zero. The intervals against FrugalGPT are above zero.

Visible answers do not fully explain the result. Voting on answers visible in the same eight-token candidate prefixes reaches 80.24% on MMLU, versus 83.77% for Probe-then-Route, and 68.70% ID / 64.89% OOD macro accuracy, versus 76.41% / 69.04% for Probe-then-Route. We averaged vote ties uniformly and used a random candidate (seed 42) when no usable answer was visible. With the predictor architecture fixed, masking annotated answer spans during training and testing retains 75.77% ID / 68.61% OOD, drops of 0.64 and 0.43 points. On MMLU, the masked run scores 84.05%, up 0.28 percentage points from the unmasked reference of 83.77%. Finally, only 0.89% of MMLU candidate prefixes expose an answer at k=2k=2, versus 68.81% at k=8k=8, yet MMLU accuracy is comparable (83.84% vs. 83.77%). These results indicate that short prefixes provide routing signals beyond visible answers.

What the OOD evidence supports. We selected k=8k=8 based on ID macro accuracy. At k=8k=8, full Probe-then-Route scores 69.04% OOD, below Lookahead (69.37%), RadialRouter performance-first (69.53%), and Qwen3-14B (69.26%), while gaining 2.03, 2.50, and 3.52 points on ID, respectively. Selective Probe-then-Route gains 1.69 points on ID over Lookahead (76.07% vs. 74.38%) and trails by 0.20 points on OOD (69.17% vs. 69.37%). Full Probe-then-Route leads on ARC-C (94.97%) and ties for the best score on MMLU-CF (70.80%). Relative to Lookahead, it gains on those benchmarks and MathBench-A and loses on HumanEval, LogiQA-en, and SimpleQA. Shorter selective probes reach 75.41% ID / 69.48% OOD at k=1k=1 and 75.93% / 69.39% at k=2k=2. Both improve on Lookahead on ID and are numerically close on OOD. These results show competitive transfer to unseen benchmarks within the six evaluated domains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.