acceptodds
Under review as a conference paper at ICLR 2027

The Conditional Value of Compute: When Is Adaptive Inference Worth It?

Abstract

More test-time compute can improve LLM performance, but a fixed budget makes this benefit selective: which inputs should receive the extra computation? Average quality-cost curves do not answer this, and difficulty or confidence can be misleading because inputs that are hard for a cheap action may also be hard for a more expensive one. We therefore study the expected improvement from additional computation conditioned on information available before the decision-the conditional value of compute. We derive the optimal allocation rule across budgets, characterize what routing information must preserve to support these decisions, and quantify losses from estimation error and from paying to acquire the routing signal. On four SWE-Gym judge pairs, learned routing improves held-out trajectory-judgment accuracy by 3.92-8.35 percentage points over random allocation when 30% of inputs receive the more expensive judge, with positive gains in all 44 leave-one-repository-out evaluations. On RouterBench, question-text routing improves actual answer correctness by 3.32-6.02 points on three representative model pairs; 1.37-2.64-point gains remain when the upgrade budget is fixed separately within each task. Complementary experiments reinforce a second distinction: a signal can improve targeting yet may not be worth acquiring once its own cost and competing cheap actions are counted. Together, these results support a simple principle: allocate compute according to predicted incremental return, not difficulty alone.

Primary area:
foundation or frontier models, including LLMs
Keywords:
adaptive inference, test-time compute, LLM routing, conditional value of compute, compute allocation, model cascades, value of information

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_N5wUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates when additional computation is beneficial in adaptive LLM inference and how it should be allocated across inputs. The authors formulate routing between cheaper and more expensive inference actions using conditional incremental advantage, which measures the expected utility improvement from selecting the more expensive action given available information. Under a static allocation model with a fixed expected upgrade rate, the paper characterizes the optimal allocation value relative to random routing, its dependence on the budget, and information required to preserve optimal decisions. It then considers learning the conditional advantage from paired observations, derives regret bounds in terms of estimation error, and extends the analysis to routing-information acquisition costs and heterogeneous computation costs. The proposed framework is empirically evaluated on SWE-Gym and RouterBench, examining learned allocation gains, cross-repository transfer, sample efficiency, and within-task routing performance. Additional experiments investigate the costs of acquiring routing signals, inexpensive alternative actions, and sequential answer sampling. The results demonstrate that learned routing scores can improve fixed-budget allocation, while the practical value of additional computation depends on information costs, available alternatives, and deployment conditions.

Critical strengths
  1. Comprehensive experimental evaluation with meaningful controls. The paper provides a systematic empirical evaluation across multiple model pairs and tasks, supported by carefully designed control experiments. In particular, the cross-repository transfer experiments, within-task allocation controls, class-balanced evaluations, and training-data ablations strengthen the credibility of the reported allocation gains.
Critical weaknesses
  1. Limited theoretical novelty and contribution. The theoretical results in Sections 3 and 4 largely follow from standard decision-theoretic and resource allocation principles, including optimal top-r selection, information requirements for preserving optimal decisions, estimation-error regret bounds, and cost-aware allocation. While these results provide a formal characterization, it is unclear what substantial new theoretical insights are obtained beyond existing frameworks. Moreover, several results are not meaningfully used to explain the empirical observations or derive new predictions. Most of the main experimental findings could be established without these theoretical developments, which raises concerns about the actual contribution of the proposed theoretical framework.

  2. Insufficient justification of the problem formulation. The theoretical analysis is built primarily on a simplified static allocation model with a fixed pair of inference actions and an expected upgrade budget. However, the paper does not sufficiently justify these modeling assumptions or clarify the extent to which the resulting theoretical conclusions apply to practical adaptive inference settings. Furthermore, the cost-sensitive experiments in Section 6.5 rely on assumed relative call prices and fixed upgrade rates rather than measured inference costs. These limitations raise concerns about the practical relevance and generalizability of the theoretical framework.

Questions for authors
  1. Theoretical Novelty. The results in Sections 3 and 4 appear closely related to established principles in decision theory and resource allocation, such as top-r selection, information preservation, and estimation-error regret bounds. Could the authors clarify which theoretical results, particularly Theorem 3.1 and Proposition 3.2, provide substantive new insights beyond existing frameworks, including related treatment-prioritization and Qini analysis?

  2. Justification of the Problem Formulation. The theoretical framework primarily considers a simplified static allocation problem with a fixed pair of inference actions and an expected upgrade budget. Could the authors explain the motivation for these modeling choices and clarify the practical settings in which the resulting theoretical conclusions are expected to apply? A clearer justification of these assumptions would help assess the practical relevance of the framework.

  3. Practical Implications of the Regret Bound. Proposition 4.1 bounds the policy regret by the expected absolute advantage estimation error. However, the paper also emphasizes that preserving the ordering of conditional advantages is sufficient for optimal fixed-budget allocation, even without accurate numerical estimation. Could the authors clarify what additional insights this bound provides about routing performance, particularly when estimation error is large but the ordering is preserved? Is there evidence that the bound is informative for the learned routing policies evaluated in the paper?

  4. Connection Between Cost-Aware Theory and Experiments. Equation (12) provides a cost-aware selection rule under heterogeneous computation costs, while the experiments in Section 6.5 mainly consider fixed upgrade rates and assumed relative call prices. Could the authors clarify how this theoretical decision rule informs practical LLM routing and which experimental findings directly support its effectiveness? In particular, how well do the assumed costs reflect actual inference costs?

  5. Much of the proposed framework appears applicable to general decision problems involving conditional action values and resource constraints. Could the authors clarify which properties of LLM inference are specifically captured by the theoretical analysis, and what new insights the framework provides beyond general resource allocation principles? This distinction would help establish the significance of the theoretical contribution in the context of adaptive LLM inference.

  6. In Table 1 and the experiments in Sections 6.2 and 6.3, cheap-error prediction often achieves performance comparable to direct advantage estimation. Could the authors clarify under what conditions direct advantage estimation offers a meaningful advantage over simpler prediction targets? Considering the additional cost of acquiring paired supervision, what evidence supports its practical benefit?

Reviewer_1pXmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper investigates when adaptive test time computation is beneficial under a fixed inference budget. Instead of allocating additional computation based only on input difficulty or uncertainty from a cheaper model, the authors propose using the conditional incremental advantage g(F)=E[UB−UA∣F]g(F)=\mathbb{E}[U_B-U_A\mid F]. Here, UAU_A and UBU_B represent the utilities of cheap and expensive inference actions, respectively. This approach estimates how much a query is expected to benefit from additional computation. The main contributions focus on compute budget allocation, learning when additional computation is useful, and accounting for inference costs.

Critical strengths
  1. Well motivated decision objective: The paper clearly explains why predicting the benefit of additional computation is more useful than simply predicting whether a cheaper model will fail. Equation 2 and the example in Section 3.3 show that input difficulty alone is not enough, since cheap and expensive inference actions may have different strengths.

  2. Strong empirical evaluation with multiple controls: The paper evaluates four SWE-Gym model pairs and reports positive allocation gains across 44 evaluations, each leaving out one repository. It also uses within task routing on RouterBench to control for differences between task categories. These experiments provide evidence that the benefits of adaptive computation are not simply due to repository identity or differences between benchmark categories.

Critical weaknesses
  1. Limited methodological novelty: The proposed approach builds on existing ideas from treatment effect prioritization, Qini/TOC curves, and learning to defer. Theorem 3.1 and Proposition 4.1 mainly adapt established principles to adaptive inference rather than introducing a fundamentally new algorithm.

  2. Limited practical improvement: The SWE-Gym results mainly show improvements over random upgrading (Table 1). A text only classifier already achieves 79.35% balanced accuracy without requiring judge calls, while additional routing methods do not show statistically significant improvements. This raises questions about the practical benefits of the proposed approach over simpler alternatives.

  3. Incomplete cost evaluation: The cost comparisons rely on assumed model cost ratios and do not fully account for feature extraction overhead or hardware latency, as discussed in Section 6. As a result, it remains unclear whether the proposed approach provides meaningful efficiency improvements in real world deployment.

  4. Weak generalization across tasks: The held out RouterBench evaluations show negative allocation gains for several combinations of model pairs and task families, including MBPP and HellaSwag. These results raise concerns about how well the approach generalizes to unseen tasks.

  5. Unclear benefit of sequential decision making: Myopic stopping achieves the same accuracy as continuation while using fewer tokens. At λ=0.005\lambda=0.005, it uses 7,750.0 tokens compared with 8,204.2 on MATH500 and 2,394.3 compared with 2,408.2 on GSM8K, without any loss in accuracy. This makes the additional benefit of more complex sequential decision making unclear.

  6. Limited robustness evaluation: The experiments use fixed model pairs without examining how the approach performs when models are updated or inference costs change. This limits confidence in its robustness under changing deployment conditions.

Questions for authors
  1. Novelty compared with prior work: Since Eq. 1 connects allocation value to TOCi curves and describes the top budget allocation rule as a standard approach, what is the main new theoretical or algorithmic contribution beyond applying existing treatment effect prioritization methods to inference computation?

  2. Improvement over simpler alternatives: Table 14 shows that additional judge calls do not provide a statistically significant improvement in balanced accuracy over the text only classifier, which already achieves 79.35%. Can the authors demonstrate statistically significant improvements in overall performance compared with this simpler and less expensive baseline, while accounting for the full cost of routing and verification?

  3. Practical inference efficiency: The cost analysis does not include feature extraction overhead or hardware latency. Can the authors report actual end to end latency, FLOPs, and throughput for direct routing and cheap first cascading? This would help determine whether the predicted benefits in compute allocation translate into meaningful savings in practice.

  4. Robustness of routing decisions: How do changes in data distribution, model upgrades, or relative inference costs affect the learned advantage rankings, fixed calibration thresholds, and overall allocation gains? Can the authors evaluate whether the routing approach remains effective under these changing conditions?

Reviewer_PFwgOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper propose a new formulation to test time compute budget allocation based on the expected improve for the allocated compute given the information available before the decision. The paper introduces some theoretical arguments followed by an empirical study to validate the claims.

Critical strengths

The theorems and empirical studies are sound in the sense that the ask the right question, assuming an ideal selector that knows the conditional expected benefit of the additional computations.

Critical weaknesses

Although the theorems and experiments are sound, they fail to provide any results of significant impact. The theorem relies on the existence of an ideal selector (conditional expected benefit) but makes no conclusion about the complexity of computing the optimal (or any) decision. In that sense the main statement of the theorem simply easily follows from the strict assumptions without providing much mathematical contributions here. The experimental study then evaluates how different ML methods can estimate the conditional incremental advantage and propose an "advantage based selector" based on this estimate. The experiments weaknesses are 1) have little connection to the theoretical contributions (ie simply estimating g) and 2) improvements of advantage-based selection only hold compared to random baselines while fail to make significant gains over existing ones, and 3) the selected benchmarks and models are relatively outdated, making it not clear if the are transferable to more recent models and tasks. Overall, while sounds, the paper's contribution have questionable impact to the ML research community.

Questions for authors

under what conditions can g be sufficiently estimated and under which not? same goes for the advantage based selection, when should this outperform other predictors and when not?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.