acceptodds
Under review as a conference paper at ICLR 2027

EXACT: OPTIMIZING ADAPTIVE INFERENCE UTILITY VIA EXECUTION COMPILATION

Abstract

Adaptive inference trades terminal answer quality against the cost of acquiring responses. We introduce EXACT, a method for optimizing the expected utility of a specified adaptive inference procedure. Given IID response banks and controllers with finite sufficient state, it compiles shared continuation states to average subset selection, reveal order, and stopping conditional on each bank. With the controller fixed, the resulting credits weight generator sequence-score updates (EXACT-G); with the generator fixed, differentiating the compiled value trains the stopping controller (EXACT-C). The generator estimator is the conditional expectation of its causal trajectory counterpart, eliminating conditional execution variance while retaining response sampling. Across three paired Qwen3-4B seeds under a 200-update budget, development-selected EXACT-G checkpoints improve mean six-benchmark deployment utility by 0.0140 and adaptive accuracy by 1.19 percentage points over correctness GRPO, with 1.54% fewer expected rollouts. A separate Qwen3-1.7B rank-eight LoRA diagnostic attributes 51% of sampled-gradient energy to execution variation under aggressive stopping. On frozen Qwen3-4B banks, EXACT-C computes exact conditional controller gradients and reaches comparable observed held-out utility to Order-MC-64 with 19.5% less controller-training CPU.

Primary area:
reinforcement learning
Keywords:
adaptive inference, test-time scaling, reinforcement learning, policy gradient, language model post-training, inference-time compute, stopping policy, Rao-Blackwellization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Mt6HOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how to train language models and stopping controllers to balance answer quality and sampling cost in adaptive inference. EXACT-G uses dynamic programming to average over executions on a sampled response bank. EXACT-G computes response-level credits for training the generator under a fixed stopping policy. EXACT-C freezes the generator and trains the stopping controller. Experiments on mathematical reasoning tasks with Qwen3 models show the effectiveness of the proposed methods.

Critical strengths
  • The dynamic programming formulation is interesting.
  • The theoretical analysis provides a clear justification for averaging execution randomness and explicitly distinguishes conditional exactness from uncertainty due to response sampling.
Critical weaknesses
  • The improvements in experiments are moderate. For example, Table 3 shows that EXACT-C achieves comparable accuracy and slightly lower mean utility than Order-MC-16, while requiring substantially more CPU time (389.8 versus 225.3 seconds).
  • The ablation studies do not sufficiently isolate the contribution of exact compilation.
  • Experiments are limited to one model family (Qwen) and one task (math reasoning).
Questions for authors
  • Why choose β=0.2\beta=0.2? Could the authors provide an ablation study or sensitivity analysis on this parameter?

  • Compared with GRPO, EXACT-G changes both the training objective and gradient estimation. Could the authors provide an ablation study to distinguish the effects of these two changes?

  • Could the authors provide statistical significance tests for the reported improvements in Table 2 and Table 3, and so on?

  • Could the authors provide more results on other model families or other tasks?

  • What if EXACT-G and EXACT-C are trained jointly? Are there any difficulties with joint training, and could it lead to further improvements?

Reviewer_LxHkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes EXACT, a framework for optimizing adaptive inference utility through exact marginalization of execution randomness. It introduces EXACT-G for training language model generators and EXACT-C for optimizing stopping controllers. Experiments demonstrate reduced gradient variance and improvements in deployment utility.

Critical strengths

S1. Principled formulation of adaptive inference optimization.

The paper develops a unified framework for computing execution-averaged training signals for both generators and stopping controllers. The theoretical analysis of conditional gradient variance reduction is a meaningful technical contribution.

Critical weaknesses

W1. Insufficient experimental validation and inadequate baseline comparisons.

The experimental evaluation is insufficient to support the paper's central contributions. The main generator-training experiments are conducted only on Qwen3-4B, with evaluation restricted to mathematical reasoning tasks. Although a supplementary Qwen3-1.7B study is provided, validation across different model scales, architectures, and task domains remains limited. More importantly, EXACT-G is primarily compared against correctness-based GRPO, which optimizes a different objective. The lack of stronger and more comprehensive baseline comparisons makes it difficult to determine whether the reported improvements originate from the proposed method itself or simply from optimizing a deployment-aware objective.

W2. Limited practical improvements relative to the proposed method's complexity.

The experimental results in Table 2 and Table 3 do not demonstrate sufficiently compelling performance advantages over existing baselines. Despite the considerable algorithmic complexity introduced by EXACT, the observed improvements in deployment utility, accuracy, and computational efficiency remain relatively modest. In particular, the controller experiments do not establish a clear advantage over computationally cheaper alternatives. These results raise concerns about whether the additional complexity of the proposed method is justified by its practical benefits.

W3. Limited scalability and restrictive assumptions.

The proposed method relies on independently sampled responses and a finite sufficient-state representation, which limits its applicability to more general adaptive reasoning settings. Furthermore, the number of dynamic-programming states can grow exponentially with answer diversity, potentially undermining the claimed computational efficiency. However, the experiments primarily consider response banks of only 16 candidates, without systematically evaluating scalability under larger sampling budgets or more diverse response distributions. Consequently, the practical applicability and scalability of EXACT remain insufficiently demonstrated.

Questions for authors

Q1. What is the theoretical motivation behind the design of the response-level causal credit coefficients qiq_i in Equation (8)? Could the authors provide a clearer explanation of why this particular formulation appropriately captures each response's contribution to the overall adaptive inference utility?

Q2. Why is EXACT-G primarily compared against correctness-based GRPO? Can the authors provide comparisons with stronger and more comprehensive baselines, particularly methods that optimize the same deployment-aware objective, to better substantiate the contribution of EXACT?

Q3. Can the authors provide additional experiments across different model scales, architectures, and task domains to demonstrate the generalizability of EXACT beyond the current mathematical reasoning setting?

Q4. Given the relatively modest improvements reported in Table 2 and Table 3, what practical advantages justify the additional complexity of EXACT? Can the authors provide more comprehensive end-to-end computational cost analyses and scalability evaluations?

Reviewer_V58WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Training for adaptive inference requires a method for assigning reward for each response, for their effects on answer quality as well as the cost to generate. This paper introduces a new procedure "EXACT" that assigns credit for each response under a particular training procedure. It uses a dynamic programming based procedure to average over subset selection, reveal order and shared states. Experiments over 2 qwen models and 6 mathematical reasoning datasets show consistent (if mild) gains on adaptive accuracy and mean deployment utility.

Critical strengths
  1. Pool augmentation as an analysis framework. I like the formulation of an ambient superpool of sampled responses, where an adaptive inference procedure can be viewed as sequentially revealing indices from this pool. This provides a clean and intuitive framework for analyzing different execution orders and stopping decisions.

  2. Order-invariant state representation. The idea of representing different response sequences that lead to the same answer counts using a shared state is elegant. It allows the algorithm to reuse computations across different execution paths, rather than enumerating each possible ordering separately.

  3. Computational complexity analysis. I appreciate that Section 3 explicitly analyzes the algorithm's computational complexity, including how the number of reachable states depends on answer diversity. This helps clarify both the computational advantages of the proposed approach and the regimes in which those advantages may diminish.

Critical weaknesses
  1. Limited model and task diversity. The experiments use only two models from the same Qwen family, and all six evaluation benchmarks focus on mathematical reasoning. This somewhat limits confidence in the broader applicability of the proposed procedure, particularly across different model architectures and task domains.

  2. Modest gains relative to computational complexity. The empirical improvements are relatively small given the additional computational cost. For instance, in Table 3, EXACT-C incurs more than five times the CPU training cost of PPO (389.8 versus 73.3 seconds), while improving expected utility by less than 1%. Although the exact gradient computation is theoretically appealing, the practical benefits relative to simpler and cheaper alternatives are less convincing.

  3. Unclear applicability to open-ended generation. The proposed algorithm relies heavily on grouping responses into discrete answer classes, which is relatively straightforward for mathematical reasoning tasks with well-defined final answers. It is less clear how this framework could extend to open-ended generation, where responses may not admit natural or unambiguous equivalence classes. I would be interested in the authors' thoughts on whether and how the procedure could be adapted to such settings.

  4. Scope of the theoretical contributions. The mathematical grounding is a strength of the paper, particularly the exact gradient estimation and variance-reduction guarantees. However, much of the theoretical analysis builds on established techniques, including dynamic programming, exchangeability, and Rao–Blackwellization. While their combination in the proposed framework is elegant, I would have liked to see broader theoretical insights or guarantees extending beyond the specific adaptive inference setting studied here.

Questions for authors

See my weaknesses above.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.