- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes running candidate LLMs for a short prefix before choosing which response to complete. A router predicts correctness and the remaining generation cost based on the query and candidate prefixes. A selective variant uses a query-only router to reduce the number of candidates probed. Experiments cover eight models and twelve benchmarks. The reported ID gain over Lookahead is 2.03 percentage points at 8 tokens per candidate, while OOD average accuracy remains below that of the strongest baselines.
Critical strengths
Short-prefix probing is a practical way to acquire response information without generating every candidate answer in full. The paper extensively investigates probe length and candidate pruning, includes a useful no-prefix ablation, and reports both probing costs and unfavorable OOD results rather than presenting only the best cases.
Critical weaknesses
The accuracy gain does not yet justify the extra inference cost. Even an eight-token probe requires processing the prompt on every retained model. The paper includes input-token charges but does not show how the trade-off varies with prompt length or pool size when multiple prefills are required for the models. Full and selective probing improve ID accuracy by 2.03 and 1.69 points over Lookahead, while increasing total cost by 69% and 29%, respectively. Both main configurations are less accurate and more expensive on OOD data. The selective variant's headline saving of over 50% is offset by additional cost; its total ID saving over full probing is 24%. Given the incremental advance over response-informed routing, comparisons at matched total budgets and against simpler selectors using the same prefixes are central to establishing the contribution.
Questions for authors
- Does “selected on the ID test set” in Appendix A.2.5 accurately describe the implementation? If so, what results remain when temperature, tolerance, and probe length are selected exclusively on validation data and then fixed for an untouched evaluation, with paired uncertainty?
- Does probing still improve accuracy over a tuned routing or cascade baseline at matched total inference budgets? Please show how the trade-off changes with prompt length and the number of models probed, separating input, probe output, completion, and router costs, and report end-to-end latency and caching assumptions.
- What is the substantive advance over the cited Lookahead, FrugalGPT, and AutoMix? In particular, does the proposed architecture outperform Lookahead's actual-response classifier when both receive the same short prefixes, or simpler prefix-based answer selection?
- Were the experiments performed with actual interrupted/resumed API calls or by truncating stored complete responses? Please clarify prompts, decoding settings, correctness grading, and how continuation preserves the generation state.
Response to Reviewer 89Yg: Inference Cost, Latency, and Same-Prefix Comparisons
Thank you for asking about the selection protocol, inference costs, and comparisons using the same prefixes.
Selection protocol. Appendix A.2.5 should say that the temperature and δ settings were selected on the ID validation set. We held them fixed for ID and OOD testing. The supplementary material shows the sweeps.
Accuracy at shared spending caps. We used the reported k=8 ID API costs of selective and full probing as shared spending targets. Costs are USD per 1,000 queries. Our reply to Reviewer 2 describes the RadialRouter cascade.
On ID, selective probing leads the cascade by 2.40 percentage points (pp) at the 0.342 target, and full probing leads by 2.31 pp at the 0.450 target. On OOD, using the same ID-tuned policies, selective probing reaches 69.17% vs. 68.95% for the cascade at the 0.342 target, and full probing reaches 69.04% vs. 68.88% at the 0.450 target.
Prompt length, probe count, and cost. The supplementary material shows input API cost by prompt length for all eight models. For continuation, the API charges for processing the query and prefix again and for generating the remaining tokens. We verified zero cache hits and used uncached input prices.
We report the cost of generating the selected response in one call and the additional API cost of probing and continuation. These aggregates do not itemize prompt-input, probe-output, and continuation charges for the original runs. At k=8, the two components are 0.243 and 0.207 for full probing, and 0.252 and 0.090 for selective probing, in USD per 1,000 ID queries. The locally hosted controller incurs no separate API charge.
We reused two k=8 selective-probing results to show how accuracy and cost change with fewer probes.
Adaptive selection reduces additional cost by 56.4% and total cost by 24.0% relative to full probing, with a 0.34 pp drop in ID macro accuracy.
Latency. Estimated end-to-end latency for full probing at k=8 was 13.646 s with warm-up and 14.484 s without warm-up. Our reply to Reviewer 3 describes both timing protocols.
Contribution and same-prefix comparison. Lookahead predicts response representations before candidate models generate tokens at inference. Its preliminary experiment tests a classifier that reads actual responses. FrugalGPT and AutoMix use completed answers to decide whether to invoke another model. Probe-then-Route uses short prefixes to choose which model finishes the answer. Like self-consistency (Wang et al., ICLR 2023) and multi-agent debate (Du et al., ICML 2024), it spends additional inference compute to improve accuracy. Shortlisting limits the roughly linear growth in probing cost with pool size.
Following Lookahead's response-classification procedure, we trained SmolLM2-135M and ModernBERT classifiers for four epochs on all ID training queries, using the same eight-token prefixes as Probe-then-Route. We chose checkpoints by ID validation macro accuracy. Prefix voting falls back to a random candidate (seed 42) when no usable answer appears.
Probe-then-Route leads the stronger Lookahead classifier by 3.33 pp on ID and 0.70 pp on OOD, and prefix voting by 7.71 pp on ID and 4.15 pp on OOD.
API procedure and caching. We prompted each candidate with the benchmark question, with no extra system prompt, at temperature 0 and seed 42. For continuation, we sent a separate request with the original question and generated prefix. We waited ten days between probing and continuation and confirmed zero cache hits and no reuse of cached decoding state. We used GPT-5.6 Luna with high reasoning effort to extract answers as given, without solving or correcting them, and then grade them using benchmark-specific criteria.