- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This work studies the problem of test-time scaling. The authors show that existing TTS methods typically adopt a uniform task-level strategy across all queries and are confined to a single model and a limited scaling space, whereas different queries should be matched with different combinations of model and scaling strategy. To address this, they propose TTS-Route, in which a TTS-Router performs action selection based on the query, a low-cost probe, and the verifier score, while a TTS-Planner constructs the TTS graph on the fly according to verifier feedback and decides when to expand and when to stop. Experiments show that TTS-Route improves the performance-cost frontier over strong baselines and generalizes to out-of-distribution tasks and alternative budget metrics.
Critical strengths
- The design of the method is supported by clear experiments and analysis, and is solid.
- The proposed method achieves good performance and efficiency.
- Multiple budget settings and several performance-cost metrics are considered.
Critical weaknesses
-
The preparation of training queries and scoring also appears to incur a non-trivial cost. The inference cost of this part should be analyzed separately, and both the training cost and the inference cost should be taken into account when computing the overall cost.
-
There is no complete logical loop between the theoretical analysis and the method design. Proposition 1 demonstrates the limitation of directly predicting the oracle, but it does not explain why prediction based on success probabilities can break this ceiling, or at least outperform direct oracle prediction.
-
Too many components and operations limit readability. A pseudocode covering the complete pipeline should be included to aid understanding.
Questions for authors
Please refer to the Weakness part.
6 comments
Response Part II
Reply 2 (W2): Theoretical Motivation for Success-Probability Prediction
W2. There is no complete logical loop between the theoretical analysis and the method design. ... it does not explain why prediction based on success probabilities can break this ceiling, or at least outperform direct oracle prediction.
We thank the reviewer for identifying this gap. We clarify the advantages, theoretical limitations, and motivation of success-probability prediction, supported by empirical evidence.
What are the advantages of success-probability prediction over direct oracle-label prediction? For query and action , let denote the outcome of a stochastic execution and its success probability. The oracle label denotes the lowest-cost successful action, with . Success-probability prediction learns , whereas direct oracle-action prediction learns and selects the most likely action. The former offers two advantages:
(1) Alignment with the optimization objective and sufficient information for decision-making. Our objective (Eq. (2)) is to maximize expected task success subject to an average compute budget: Given action costs, the success probabilities provide sufficient information to determine an optimal budget-aware routing policy. In contrast, maximizing oracle-label prediction accuracy does not necessarily optimize the accuracy–cost trade-off.
(2) More informative supervision and improved learning efficiency. Oracle labels suffer from information censoring: the outcome of a higher-cost action is directly revealed only when all cheaper actions fail. For profiled queries, action-level supervision provides observations per action. Under independent outcomes and fixed success probabilities, oracle labels directly reveal the outcome of action in approximately observations.
Can success-probability prediction overcome the theoretical ceiling? No. The ceiling in Eq. (3) bounds the maximum accuracy of predicting stochastic oracle labels from query-only information. Changing the prediction target cannot eliminate the inherent randomness of oracle labels. To improve oracle-label predictability, additional informative evidence is needed. Accordingly, we introduce the probe response and its verifier score (Insight 3) to reduce uncertainty.
How does Insight 2 motivate success-probability prediction? Insight 2 reveals a fundamental instability in direct oracle-label prediction: the lowest-cost successful action for the same query may vary across stochastic executions. If oracle labels were sufficiently stable, directly predicting the oracle action could reasonably balance accuracy and cost. However, Insight 2 shows that their inherent uncertainty limits prediction reliability and learning efficiency on expensive actions. We therefore predict each action's conditional success probability , which better aligns with the accuracy–cost trade-off in our optimization objective and mitigates the learning inefficiency associated with oracle-label prediction. This does not eliminate execution randomness. Instead, learning expected action performance avoids reliance on unstable oracle labels and more directly supports the budget-aware optimization in Eq. (2).
Empirical Validation Our ablation study (Appendix F.1) shows that replacing success-probability prediction with direct oracle-action prediction decreases / by 11 / 17 percentage points on average, further supporting the effectiveness of our design.
Reply 3 (W3): Improving Presentation Clarity
W3. Too many components and operations limit readability. Pseudocode covering the complete pipeline should be included to aid understanding.
We appreciate this suggestion. The current submission includes pseudocode for graph execution (Algo. 1) and online inference (Algo. 2) in the appendix, but lacks a unified description of the offline and online stages. To improve readability, we will include a consolidated pipeline algorithm in the main text, e.g.,
Offline stage (executed once)
1: Train proxy verifier V via teacher distillation. (Eq. 5)
2: For each training query q and action a = (M, S):
Execute a using TTS-Planner;
record Y_{q,a} and C_{q,a}. (Eq. 6)
3: Train TTS-Router on D_router to predict p_q(a). (Eq. 7)
4: Calibrate lambda(b) on D_cal for each target budget b. (Eq. 9)
Online stage (for each query q and budget b)
... (see details in Algo.2 in the paper)
We thank the reviewer again for the valuable feedback and will incorporate these improvements into the revised manuscript. We welcome any further feedback.
Thanks for the authors' response. It appears that the offline cost of TTS-Route is noticeably higher than that of FrugalGPT (1.72x), yet its APGR does not surpass FrugalGPT's by that much even when scaled to a larger . For fairness and rigor, could the authors provide an APGR comparison under aligned offline costs?
Response Part I
We sincerely thank the reviewer for the constructive feedback. We address each concern below.
Reply 1 (W1): Offline Training and Overall Compute Costs
We agree and provide a separate analysis of offline training costs, followed by an evaluation of the overall cost after amortization.
Offline Cost. We account for four components across 4,604 training queries, measured in normalized compute units:
The following table reports the offline cost per training query.
TTS-Route's offline cost is comparable to RouteLLM's, approximately 1.7× that of BEST-Route, FrugalGPT, AutoMix, BoN, and BG-MCTS, and 3.2× that of HybridLLM. This additional cost arises from profiling a broader action space, including width-lean and depth-lean strategies. We therefore evaluate whether the improved inference efficiency offsets this upfront cost.
Overall Cost. Offline training incurs a one-time cost. For a deployment serving T queries, the amortized overhead per query is Coffline/T. We incorporate this overhead into each operating point and recompute APGR.
Each entry reports APGR1b / APGR8b.
Key findings: (i) Small deployments: At T=105, TTS-Route is less cost-effective, with HybridLLM, FrugalGPT, AutoMix, and BEST-Route achieving higher APGR1b. (ii) Break-even point: TTS-Route becomes the best method on APGR8b after approximately 1.3×105 queries and on both metrics after 3.3×105 queries, when the amortized overhead is approximately 10 units per query. (iii) Large deployments: At T≥106, TTS-Route leads on both metrics, approaching inference-only performance at T=107. These results clarify that TTS-Route's higher offline cost is offset by its inference efficiency at sufficiently large deployment scales.