- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates when additional computation is beneficial in adaptive LLM inference and how it should be allocated across inputs. The authors formulate routing between cheaper and more expensive inference actions using conditional incremental advantage, which measures the expected utility improvement from selecting the more expensive action given available information. Under a static allocation model with a fixed expected upgrade rate, the paper characterizes the optimal allocation value relative to random routing, its dependence on the budget, and information required to preserve optimal decisions. It then considers learning the conditional advantage from paired observations, derives regret bounds in terms of estimation error, and extends the analysis to routing-information acquisition costs and heterogeneous computation costs. The proposed framework is empirically evaluated on SWE-Gym and RouterBench, examining learned allocation gains, cross-repository transfer, sample efficiency, and within-task routing performance. Additional experiments investigate the costs of acquiring routing signals, inexpensive alternative actions, and sequential answer sampling. The results demonstrate that learned routing scores can improve fixed-budget allocation, while the practical value of additional computation depends on information costs, available alternatives, and deployment conditions.
Critical strengths
- Comprehensive experimental evaluation with meaningful controls. The paper provides a systematic empirical evaluation across multiple model pairs and tasks, supported by carefully designed control experiments. In particular, the cross-repository transfer experiments, within-task allocation controls, class-balanced evaluations, and training-data ablations strengthen the credibility of the reported allocation gains.
Critical weaknesses
-
Limited theoretical novelty and contribution. The theoretical results in Sections 3 and 4 largely follow from standard decision-theoretic and resource allocation principles, including optimal top-r selection, information requirements for preserving optimal decisions, estimation-error regret bounds, and cost-aware allocation. While these results provide a formal characterization, it is unclear what substantial new theoretical insights are obtained beyond existing frameworks. Moreover, several results are not meaningfully used to explain the empirical observations or derive new predictions. Most of the main experimental findings could be established without these theoretical developments, which raises concerns about the actual contribution of the proposed theoretical framework.
-
Insufficient justification of the problem formulation. The theoretical analysis is built primarily on a simplified static allocation model with a fixed pair of inference actions and an expected upgrade budget. However, the paper does not sufficiently justify these modeling assumptions or clarify the extent to which the resulting theoretical conclusions apply to practical adaptive inference settings. Furthermore, the cost-sensitive experiments in Section 6.5 rely on assumed relative call prices and fixed upgrade rates rather than measured inference costs. These limitations raise concerns about the practical relevance and generalizability of the theoretical framework.
Questions for authors
-
Theoretical Novelty. The results in Sections 3 and 4 appear closely related to established principles in decision theory and resource allocation, such as top-r selection, information preservation, and estimation-error regret bounds. Could the authors clarify which theoretical results, particularly Theorem 3.1 and Proposition 3.2, provide substantive new insights beyond existing frameworks, including related treatment-prioritization and Qini analysis?
-
Justification of the Problem Formulation. The theoretical framework primarily considers a simplified static allocation problem with a fixed pair of inference actions and an expected upgrade budget. Could the authors explain the motivation for these modeling choices and clarify the practical settings in which the resulting theoretical conclusions are expected to apply? A clearer justification of these assumptions would help assess the practical relevance of the framework.
-
Practical Implications of the Regret Bound. Proposition 4.1 bounds the policy regret by the expected absolute advantage estimation error. However, the paper also emphasizes that preserving the ordering of conditional advantages is sufficient for optimal fixed-budget allocation, even without accurate numerical estimation. Could the authors clarify what additional insights this bound provides about routing performance, particularly when estimation error is large but the ordering is preserved? Is there evidence that the bound is informative for the learned routing policies evaluated in the paper?
-
Connection Between Cost-Aware Theory and Experiments. Equation (12) provides a cost-aware selection rule under heterogeneous computation costs, while the experiments in Section 6.5 mainly consider fixed upgrade rates and assumed relative call prices. Could the authors clarify how this theoretical decision rule informs practical LLM routing and which experimental findings directly support its effectiveness? In particular, how well do the assumed costs reflect actual inference costs?
-
Much of the proposed framework appears applicable to general decision problems involving conditional action values and resource constraints. Could the authors clarify which properties of LLM inference are specifically captured by the theoretical analysis, and what new insights the framework provides beyond general resource allocation principles? This distinction would help establish the significance of the theoretical contribution in the context of adaptive LLM inference.
-
In Table 1 and the experiments in Sections 6.2 and 6.3, cheap-error prediction often achieves performance comparable to direct advantage estimation. Could the authors clarify under what conditions direct advantage estimation offers a meaningful advantage over simpler prediction targets? Considering the additional cost of acquiring paired supervision, what evidence supports its practical benefit?