acceptodds
Under review as a conference paper at ICLR 2027

Annealed Predictor-Corrector Sampling for Inference-Time Scaling of LLMs

Abstract

Autoregressive large language models (LLMs) have demonstrated strong performance on various reasoning tasks, and when a reward model is available, inference-time scaling can steer them further toward high-reward responses. Best-of-N sampling draws responses independently from the base model and selects the highest-reward response, but reward information from one candidate is not used to guide the generation of others. Recent Sequential Monte Carlo (SMC) approaches instead guide generation along the token axis, requiring intermediate estimates of partial-response quality that may prematurely discard promising trajectories. We propose the Annealed Predictor-Corrector (APC) sampler, which performs SMC along the reward-tilting axis by constructing a sequence of intermediate distributions with gradually increasing reward strength. APC maintains complete responses as particles and alternates between predictor and corrector steps. The predictor progressively favors higher-reward responses through reweighting, while the corrector refines particles by regenerating response suffixes from the base model and selecting among candidates using a transition that preserves the current intermediate target distribution. Applying this correction along the annealing path enables responses to be progressively refined before the target distribution becomes strongly concentrated. We further introduce an adaptive stopping rule that avoids unnecessary computation once a response stabilizes. Across reasoning benchmarks, APC provides a favorable accuracy-compute trade-off compared with independent sampling and token-axis inference-time scaling baselines.

Primary area:
generative models
Keywords:
inference-time scaling, large language models, sequential Monte Carlo, reward-tilted sampling

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AvcDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents a new LLM test-time scaling algorithm. The authors formulate a monte-carlo process in the sequence-space and derive a method (APC) that moves along the reward-tilting rather than token-generating axis using two components: a predictor and a corrector, to control resampling and re-generating particles. The design is supplemented by a mathematical framework that describes the resulting sampling distributions. The authors then conduct experiments to test the FLOP-efficiency of their method compared to popular test-time scaling baselines, such as best-of-n. The authors conclude from the experiments that APC achieves a favorable compute-quality tradeoff.

Critical strengths
  1. The method is well grounded theoretically, and the resulting algorithm is easy to follow.
  2. The experiments establish a good flop-to-quality tradeoff compared to the other methods, ablations help isolating the role of different components.
Critical weaknesses
  1. The ablation results presented in table 3, suggest that removing the corrector, which is one of the main algorithmic novelties of APC, leads to an only modest decrease of the metric, but cuts down on the FLOPs substantially.
  2. The experiments show good scaling of the score with FLOPs, however the structure of the method means that it requires many sequential generations, which could be detrimental to the time-scaling of this method. Although the authors acknowledge this limitation, there is no dedicated time-scaling study in the paper, limiting the practicality of the method.
Questions for authors
  1. Given the substantial FLOPs cost of the corrector and relatively modest quality regression from removing it, would an existing correction kernel, such as QAlign's MH kernel within the same annealing framework, reach better or comparable results given the same FLOPs? Could a predictor-only setup with a larger number of particles N recover the performance?
  2. How does the method scale on the time axis? For instance, how does it compare with the other methods, how does the comparison change with different sequence lengths, particle counts and annealing steps?
  3. Could you provide a larger ablation for the role of the corrector? The ablation in table 3 uses a single seed, and the improvement from the corrector is modest. A larger study would help understand the significance of this improvement.
  4. Could you clarify the novel contributions of APC and their significance, particularly compared to the SMC framework of SMCEvolve and the correction procedure of QAlign?

I believe that this method is reasonable and well grounded in literature. The mathematical framework is sound and the experiments establish good FLOPs scaling. However, the novelty of the method is limited, since most of the components were already established previously. The novel construction of the corrector is not sufficiently tested against previous art, and ablations reveal that the overall role of the corrector in the success of the method could be limited. I am open to increasing the score if the authors can demonstrate the improvements that their novel components bring over previous work and provide quantitative analysis of the time-to-performance scaling of the method.

5 comments
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer AvcD (1/3)

We thank the reviewer for the detailed review and valuable feedback. We appreciate the positive assessment of the theoretical foundation and accuracy-compute trade-off of APC. We address each point below and have incorporated the feedback into the revised manuscript, with the corresponding changes highlighted in blue for clarity.


W1 & Q1 & Q3: [Role of the corrector] (Weaknesses 1) The ablation results presented in table 3, suggest that removing the corrector, which is one of the main algorithmic novelties of APC, leads to an only modest decrease of the metric, but cuts down on the FLOPs substantially.

(Questions 1) Given the substantial FLOPs cost of the corrector and relatively modest quality regression from removing it, would an existing correction kernel, such as QAlign's MH kernel within the same annealing framework, reach better or comparable results given the same FLOPs? Could a predictor-only setup with a larger number of particles N recover the performance?

(Questions 3) Could you provide a larger ablation for the role of the corrector? The ablation in table 3 uses a single seed, and the improvement from the corrector is modest. A larger study would help understand the significance of this improvement.

We expand the ablation study to three random seeds and additionally compare APC with MH corrector from QAlign and larger-population predictor-only variants. We evaluate the ablations without adaptive stopping and use λ=3\lambda=3, the cut scale adopted in the main APC configuration, instead of λ=1\lambda=1 used in the original ablation. All variants with correction perform ten corrector updates per particle.

[Table 4 in the revised manuscript: Ablation study of APC on MATH-500. Results are from three random seeds.]

ConfigurationN# CorrectorAccuracy (%)TFLOPs
No annealing81084.4 ± 0.51,075
No corrector8086.8 ± 0.1168
No corrector16087.4 ± 0.2337
No corrector32087.7 ± 0.1673
No corrector64087.8 ± 0.51,347
APC with QAlign’s MH corrector81088.3 ± 0.41,261
APC with i-SIR corrector81088.4 ± 0.71,075

Removing the corrector substantially reduces computation, but also lowers accuracy. Increasing the predictor-only population improves accuracy, but it remains below APC throughout the tested population sizes. Even with N=64N=64, the predictor-only (no corrector) variant achieves 87.8% at 1,347 TFLOPs, compared with 88.4% at 1,075 TFLOPs for APC. These results suggest that using additional inference computation for refinement can be more effective than using it solely to generate more initial candidates.

Replacing i-SIR with MH corrector from QAlign yields similar accuracy. The results suggest robustness across these two invariant kernel choices. We use i-SIR as a simple tractable implementation of target-preserving refinement, while the framework can also accommodate other invariant correction kernels.

Finally, applying all corrector updates directly at the final reward strength achieves 84.4% accuracy, compared with 88.4% when distributing them along the annealing path, at the same compute of 1,075 TFLOPs. This supports the importance of applying refinement along the annealing path. The corresponding analysis is included in Section 5.3 of the revised manuscript.

AuthorsOpenReview ↗

Official Comment by Authors for Reviewer AvcD (2/3)

W2 & Q2: [Time-scaling study] (Weaknesses 2) The experiments show good scaling of the score with FLOPs, however the structure of the method means that it requires many sequential generations, which could be detrimental to the time-scaling of this method. Although the authors acknowledge this limitation, there is no dedicated time-scaling study in the paper, limiting the practicality of the method.

(Questions 2) How does the method scale on the time axis? For instance, how does it compare with the other methods, how does the comparison change with different sequence lengths, particle counts and annealing steps?

As discussed in Section 5.3, the computational efficiency of APC comes with additional sequential generation steps. To examine the implications of this trade-off, we conduct a wall-clock time study under single-request and batched execution. Batched execution is relevant to throughput-oriented LLM serving, where multiple requests can be processed together.

All timing experiments use a single NVIDIA A100 GPU. We consider single-request execution, where MATH-500 problems are processed one at a time; and batched execution, where 100 problems are served concurrently. In batched execution, each problem follows the original sampling procedure, while generation and reward evaluation are batched across problems.

[Comparison with other methods]

The comparison below reports accuracy, compute, sequential calls, and execution time for baselines and APC. Accuracy, TFLOPs, and sequential calls correspond to the configuration reported in Tables 1 and 5. Single-request latency is the mean wall-clock time per problem. Batched time is the total wall-clock time divided by the number of problems.

[Table 5 in the revised manuscript: Wall-clock time analysis on MATH-500.]

MethodAccuracy (%)TFLOPsSequential callsSingle-request latency (s)Batched time (s/problem)
Best-of-N87.91,3471.034.522.3
Self-Consistency88.06431.028.416.5
QAlign87.89567.073.313.8
Particle Filtering88.61,8049.145.021.4
APC (λ=1,κ=3\lambda=1, \kappa=3)88.03354.241.56.9
APC (λ=3,κ=5\lambda=3, \kappa=5)88.86826.469.212.4

For single-request execution, the results confirm the latency cost of sequential refinement. At approximately 88% accuracy, APC (λ=1,κ=3\lambda=1, \kappa=3) takes 41.5 s per problem, compared with 28.4 s for self-consistency and 34.5 s for Best-of-N, while remaining faster than the other reward-guided samplers.

Under batched execution, both APC configurations achieve lower processing times than all compared baselines. At the same accuracy of 88%, APC (λ=1,κ=3\lambda=1, \kappa=3) takes 6.9 s per problem, a 58% reduction relative to self-consistency at 16.5 s. The higher-accuracy APC configuration achieves 88.8% at 12.4 s per problem, requiring about 25% less processing time than self-consistency. These results show a practical benefit under the evaluated serving load, where requests can be batched. The Computation Analysis paragraph in Section 5.3 discusses the trade-off between single-request latency and batched processing time.

[Effect of inference configuration]

We also analyze the effects of particle count and annealing stages on execution time. Increasing the particle count raises execution time, although single-request latency grows sublinearly as particles are processed in parallel. Accuracy initially improves, but further increasing the population provides no additional gain among the evaluated configurations. Increasing the number of annealing stages improves accuracy at the cost of more sequential generation calls and longer execution time.

[Table 11 in the revised manuscript: Analysis of particle count NN and annealing stages MM]

HyperparameterValueAccuracy (%)TFLOPsSequential callsSingle-request latency (s)Batched time (s/problem)
NN487.03241161.17.4
888.86451186.411.8
1688.61,31911122.625.5
MM488.2471759.88.5
888.86451186.411.8

[Effect of response length]

We also compare batched processing times across response-length quartiles. Problems are grouped by their mean response length under best-of-N (N=64), with Q1 and Q4 containing the shortest and longest responses, respectively. Processing time increases with response length for all methods. Both APC configurations retain lower batched processing times than Best-of-N and self-consistency across all groups, although the relative advantage is smaller for the longest responses.

[Table 12 in the revised manuscript: Batched processing time (s/problem) by response-length quartile]

Response-length quartileQ1 (shortest)Q2Q3Q4 (longest)
Mean response length (tokens)2714156071,239
Best-of-N12.916.323.336.8
Self-Consistency9.712.016.826.6
APC (λ=1,κ=3\lambda=1, \kappa=3)4.04.66.813.6
APC (λ=3,κ=5\lambda=3, \kappa=5)7.18.212.421.6
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer AvcD (3/3)

Q4: [Clarification of novel contributions] (Questions 4) Could you clarify the novel contributions of APC and their significance, particularly compared to the SMC framework of SMCEvolve and the correction procedure of QAlign?

APC builds on established SMC and suffix-regeneration MCMC principles. Our contribution is a concrete construction for reward-guided LLM inference that combines annealed refinement of complete-response populations, a tractable target-preserving suffix corrector, and adaptive stopping. The expanded experiments examine the significance of these design choices.

[Relation to SMCEvolve]

SMCEvolve formulates program evolution through annealed reweighting, resampling, and mutation. APC shares this SMC framework but differs in its refinement transition and stopping criterion for LLM response generation.

SMCEvolve uses context-conditioned program edits and rewrites, with a reward-only approximation to the MH acceptance ratio in its practical implementation. APC generates suffixes from the base LM conditional on the retained prefix, allowing the LM probability terms to cancel while retaining the cut selection probabilities. Theorem 1 establishes invariance of the resulting corrector without approximating additional model-density ratios.

The adaptive controls also serve different purposes. SMCEvolve uses ESS to adapt the annealing schedule until the final reward strength is reached. APC monitors task-level answer stability and can terminate the remaining refinement before completing the annealing path.

[Relation to QAlign]

QAlign uses base-model suffix regeneration and cancellation of LM probability terms. APC incorporates this sampling principle into an interacting particle population with incremental reweighting and resampling across intermediate reward strengths, applying refinement throughout the annealing path rather than only at a fixed final target.

The corrector in APC accommodates the chosen response-dependent cut distribution and multiple suffix candidates through an i-SIR update. We use i-SIR as a tractable implementation of target-preserving refinement. The similar accuracies obtained with i-SIR and the MH corrector from QAlign support robustness across these two invariant kernel choices. Therefore, we emphasize the annealed population-refinement construction rather than attributing the improvement specifically to the i-SIR corrector.

[Empirical significance]

The expanded three-seed ablations in revised Table 4 connect these design choices to their empirical effects. Distributing the same corrector updates along the annealing path achieves 88.4% accuracy, compared with 84.4% when applying them directly at the final reward strength, at the same compute. This supports the importance of applying refinement at intermediate reward strengths.

We additionally compare answer-stability-based stopping with stopping every problem at a fixed annealing stage, using the same full trajectories. As detailed in our response to Reviewer DrVg Q2 and shown in the revised Figure 4, adaptive stopping achieves 88.7% accuracy at 589 TFLOPs, compared with 87.6% at 554 TFLOPs for fixed-stage stopping. It also exceeds the best observed fixed-stage accuracy of 88.5%, obtained at 814 TFLOPs. These results support using answer stability to adapt the amount of refinement to each problem.

We revise the Related Work to distinguish the inherited SMC and MCMC principles from the specific construction of APC and connect the contributions more explicitly to these controlled analyses.

Reviewer_DrVgOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Annealed Predictor-Corrector (APC), a method designed to improve the inference efficiency of large language models. Starting from a set of complete responses, APC gradually increases the influence of reward guidance, adjusts candidate responses through reweighting and resampling, and explores new solutions through suffix regeneration. The authors prove that the correction step preserves the current target distribution and develop an early stopping strategy based on answer consistency across consecutive iterations. Experiments on MATH-500, AMC23, and IFEval demonstrate that APC reduces inference-time computational costs while maintaining high accuracy.

Critical strengths
  1. APC directly uses the rewards of complete responses to update candidates, without needing to estimate the rewards of incomplete responses. This makes it applicable to tasks where intermediate rewards are difficult to define. Furthermore, the IFEval experiments in the paper demonstrate its applicability beyond mathematical reasoning.

  2. The authors account for the dependence of the cut-selection distribution on the response when deriving the suffix update rule and prove that the correction step preserves the current target distribution with the corresponding adjustment. Furthermore, generating suffixes using the base model allows the relevant probability terms to cancel out, eliminating the need to train an additional proposal model.

  3. APC demonstrates computational efficiency advantages in both mathematical reasoning and instruction-following tasks. The authors further compare settings with and without annealing under matched correction counts and comparable computational budgets, providing empirical evidence for the effectiveness of the annealing design itself.

Critical weaknesses
  1. The experimental baselines are primarily drawn from works published in or before 2025, with comparisons against more recent relevant methods, such as Sampling for Quality [1] and Energy-Based Decoding [2], notably absent. These methods also focus on training-free reward-guided decoding. Including comparisons with these recent approaches would help better assess APC's relative advantages over state-of-the-art methods.

  2. The experiments cover only two mathematical reasoning datasets and one instruction-following dataset, providing insufficient evidence of APC's effectiveness across a broader range of tasks. Furthermore, on MATH-500, APC achieves an accuracy improvement of only 0.8 percentage points over Self-Consistency, despite incurring higher computational costs.

  3. Although Table 1 states that "All results are averaged over three random seeds," the paper does not provide standard deviations or confidence intervals.

[1] Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo

[2] The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates

Questions for authors
  1. The paper defines APC as a method for sampling from a reward-tilted distribution. However, in the IFEval experiments, the authors directly select the particle with the highest reward rather than sampling according to particle weights. Could the authors explain the rationale behind this design choice?

  2. Figure 4 demonstrates the computational efficiency advantage of the adaptive stopping strategy over the full annealing process, but does not compare it with a strategy that stops early at a fixed stage. Could the authors provide additional comparative experiments under the same computational budget to further verify whether adaptive stopping based on answer stability offers additional advantages?

3 comments
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer DrVg (1/3)

We thank the reviewer for the thoughtful review and helpful suggestions. We appreciate the positive assessment of the complete-response reward formulation, the theoretical justification of the corrector, and the computational efficiency of APC. We address each point below and have incorporated the feedback into the revised manuscript, with the corresponding changes highlighted in blue for clarity.


W1: [Recent baselines] (Weaknesses 1) The experimental baselines are primarily drawn from works published in or before 2025, with comparisons against more recent relevant methods, such as Sampling for Quality [1] and Energy-Based Decoding [2], notably absent. These methods also focus on training-free reward-guided decoding. Including comparisons with these recent approaches would help better assess APC's relative advantages over state-of-the-art methods.
[1] Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo
[2] The Missing Piece in Pre-trained Model Evaluation: Reward-Guided Decoding Unlocks Task-Oriented Behavior Without Parameter Updates

We additionally evaluate SMC reward-guided lookahead from Sampling for Quality [1], denoted SMC-RGLA here, and Energy-Based Decoding (EBD) [2], as suggested by the reviewer. We implement both methods in our framework, using the same base and reward models for each task.

SMC-RGLA [1] performs block-wise SMC during response generation, using prefix-level reward potentials and MH rejuvenation with rollout-based lookahead. Unlike our step-level SMC baseline, which uses prefix-only weighting without lookahead or MH rejuvenation, it incorporates these additional mechanisms to refine particles. We evaluate its process-reward-model-based configuration on MATH-500, following the sampling settings in the paper. This configuration requires prefix-level reward evaluations and therefore does not directly match our IFEval setup, which uses complete-response rewards.

EBD [2] refines complete responses through a single MH chain at a fixed reward strength. Like APC, it uses complete-response reward evaluations. We retain the initialization and reward-standardization procedure of EBD and increase the number of refinement steps to obtain compute budgets comparable to APC on MATH-500 and IFEval.

[Table R1: Additional baseline comparison on MATH-500.]

MethodAccuracy (%)TFLOPs
SMC-RGLA86.5 ± 0.21,618
EBD86.7 ± 0.4635
APC (ours)88.8 ± 0.7682

[Table R2: Additional baseline comparison on IFEval.]

MethodAccuracy (%)TFLOPs
Best-of-N76.0 ± 0.4252
QAlign76.4 ± 0.3240
EBD77.2 ± 0.9254
APC (ours)77.9 ± 0.1253

Under these settings, APC achieves higher accuracy than EBD at comparable compute budgets on both benchmarks. On MATH-500, APC also achieves higher accuracy than SMC-RGLA while requiring less computation. These results further support the effectiveness of APC relative to recent reward-guided sampling methods.

We incorporate both works into the discussion of reward-guided sampling methods in the Related Work section of the revised manuscript. We will extend these comparisons to AMC23, add accuracy-compute trade-off curves, and incorporate the updated results into the revised manuscript. We appreciate the reviewer’s suggestion, as both works are relevant to reward-guided LLM sampling. We would also appreciate consideration that, to our knowledge, both works are available only as arXiv preprints, without peer-reviewed publication or publicly released code.

AuthorsOpenReview ↗

Official Comment by Authors for Reviewer DrVg (2/3)

W2: [More empirical evidence] (Weaknesses 2) The experiments cover only two mathematical reasoning datasets and one instruction-following dataset, providing insufficient evidence of APC's effectiveness across a broader range of tasks. Furthermore, on MATH-500, APC achieves an accuracy improvement of only 0.8 percentage points over Self-Consistency, despite incurring higher computational costs.

[Additional experiments on code generation]

To broaden the task coverage, we additionally evaluate APC on HumanEval+ and MBPP+ from EvalPlus (Liu et al., NeurIPS 2023). We use Llama-3.1-8B-Instruct (Grattafiori et al., 2024) as the base model and AceCodeRM-7B as the reward model, following the AceCoder setting (Zeng et al., ACL 2025). Final correctness is evaluated using the EvalPlus test suites, which are not used during sampling. We compare against Best-of-N with sample counts chosen to match the compute budget of APC. APC achieves higher pass@1 on both benchmarks at matched compute, providing additional evidence of effectiveness across diverse tasks. The corresponding analysis in included in Section 5.2 of the revised manuscript.

[Table 3 in the revised manuscript: Results on code generation.]

MethodHumanEval+ Pass@1 (%)HumanEval+ TFLOPsMBPP+ Pass@1 (%)MBPP+ TFLOPs
Single Sample52.7 ± 0.1654.6 ± 0.05
Greedy61.0 ± 0.0659.3 ± 0.04
Best-of-N70.9 ± 0.758367.2 ± 0.3351
APC (ours)73.2 ± 1.558368.7 ± 0.6351

(Liu et al., NeurIPS 2023) Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation.
(Grattafiori et al., 2024) The Llama 3 Herd of Models.
(Zeng et al., ACL 2025) ACECODER: Acing Coder RL via Automated Test-Case Synthesis.

[Comparison with self-consistency]

Table 1 in the main manuscript reports the best accuracy achieved by self-consistency across the evaluated compute budgets. As shown in Figure 3 of the main manuscript, its best accuracy remains 88.0% even when the sampling budget is increased beyond 643 TFLOPs, whereas APC achieves 88.8% at 682 TFLOPs. Therefore, increasing the number of independent samples does not recover the higher accuracy of APC within the evaluated budget range.

APC also reaches 88.0% accuracy at a lower compute budget. The λ=1,κ=3\lambda=1, \kappa=3 configuration, corresponding to the third APC point from the left in Figure 3, requires 335 TFLOPs, compared with 643 TFLOPs for self-consistency. These results show that APC can both reach comparable accuracy with less computation and achieve higher accuracy within the evaluated budgets.

The additional wall-clock time study also demonstrates a benefit under batched execution. On a single NVIDIA A100 GPU processing 100 problems concurrently, this APC configuration takes 6.9 s per problem, compared with 16.5 s for self-consistency at the same 88% accuracy. Detailed single-request and batched comparisons are provided in updated Table 5 and Section 5.3 of the revised manuscript, as well as our response to Reviewer AvcD W2 & Q2.

Additionally, as discussed in Section 5.2, answer-based self-consistency is difficult to apply directly to free-form instruction following, where responses need not share a canonical extracted answer. APC requires only complete-response reward evaluations and can therefore be applied to these tasks.

AuthorsOpenReview ↗

Official Comment by Authors for Reviewer DrVg (3/3)

W3: [Variability across random seeds] (Weaknesses 3) Although Table 1 states that "All results are averaged over three random seeds," the paper does not provide standard deviations or confidence intervals.

We update Table 1 to report the mean and standard error across three random seeds for both accuracy and inference compute. The revised Figure 3 includes error bars across the evaluated compute budgets. We also report means and standard errors for the other experiments repeated across multiple random seeds, including those in Tables 2-4 of the revised manuscript. Self-consistency remains competitive at lower compute budgets, while APC achieves higher mean accuracy over a range of moderate compute budgets.

[Table 1 in the revised manuscript: Accuracy and inference compute on MATH 500 and AMC23, reported as mean ± standard error over three random seeds.]

MethodMATH-500 Accuracy ↑\uparrowMATH-500 TFLOPs ↓\downarrowAMC23 Accuracy ↑\uparrowAMC23 TFLOPs ↓\downarrow
Single Sample80.2 ± 0.710 ± 054.2 ± 5.115 ± 0
Greedy82.6 ± 0.010 ± 062.5 ± 015 ± 0
Selection over N i.i.d samples
Best-of-N87.9 ± 0.41,347 ± 164.2 ± 3.61,956 ± 9
SNIS87.8 ± 0.51,347 ± 163.3 ± 3.11956 ± 9
Self-Consistency88.0 ± 0.1643 ± 173.3 ± 2.9941 ± 5
Weighted Majority88.3 ± 0.61,347 ± 164.2 ± 3.61,956 ± 9
Search over partial responses
Beam Search85.0 ± 0.41,642 ± 4262.5 ± 2.92,984 ± 4
DVTS88.1 ± 0.21,834 ± 2462.5 ± 2.93,201 ± 124
Reward-guided samplers
QAlign87.8 ± 0.2956 ± 267.5 ± 2.51,410 ± 4
Step-level SMC86.7 ± 0.1911 ± 362.5 ± 3.81,626 ± 175
Particle Filtering88.6 ± 0.11,804 ± 2470.8 ± 3.33,703 ± 235
APC (ours)88.8 ± 0.7682 ± 1172.5 ± 2.9253 ± 3

Q1: [Final answer selection] (Questions 1) The paper defines APC as a method for sampling from a reward-tilted distribution. However, in the IFEval experiments, the authors directly select the particle with the highest reward rather than sampling according to particle weights. Could the authors explain the rationale behind this design choice?

APC constructs a weighted population approximating a reward-tilted distribution, while returning a single response requires a final decision rule. Our evaluation goal is to return a high-reward response, so we select the candidate with the highest reward according to the available complete-response reward model. This selection is applied after population refinement and does not change the population updates of APC.

We additionally compare two decision rules on the same terminal populations from the experiments reported in Table 1 (MATH-500) and Table 2 (IFEval), changing only the final answer selection rule. The two rules yield similar accuracies on MATH-500, whereas highest-reward selection achieves higher accuracy on IFEval. These results indicate that the final decision rule has a greater effect on IFEval under these settings. We clarify this distinction between population sampling and final answer selection in Appendix D of the revised version.

[Table 10 in the revised manuscript: Accuracy (%) under different final answer selection rules.]

Final answer selectionMATH-500IFEval
Highest reward88.777.9
Weighted sampling88.876.1

Q2: [Adaptive stopping vs. fixed-stage stopping] (Questions 2) Figure 4 demonstrates the computational efficiency advantage of the adaptive stopping strategy over the full annealing process, but does not compare it with a strategy that stops early at a fixed stage. Could the authors provide additional comparative experiments under the same computational budget to further verify whether adaptive stopping based on answer stability offers additional advantages?

We evaluate fixed-stage stopping on the same full trajectories, stopping every problem after a fixed number mm of annealing stages while keeping all other settings unchanged. We revised Figure 4 include this fixed-stage curve. Adaptive stopping achieves higher accuracy at comparable compute budgets. It also exceeds the best observed fixed-stage accuracy while requiring less computation. These results support using answer stability to adapt the amount of refinement to each problem.

[Selected results from Figure 4 in the revised manuscript: Adaptive stopping vs. fixed-stage stopping on MATH-500.]

Stopping strategyAccuracy (%)TFLOPs
Fixed-stage stopping (m=4m=4)87.6554
Fixed-stage stopping (m=7m=7)88.5814
Adaptive stopping (κ=4\kappa=4)88.7589
Reviewer_M1X8OpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

APC improves LLM responses at inference time by gradually increasing the influence of a reward model. It maintains complete responses, reweights and resamples them, and regenerates suffixes to explore alternatives. The suffix update preserves the target distribution at each stage. For math tasks, sampling stops when answers stabilize. Experiments compare APC with independent sampling, search, and reward-guided samplers on MATH-500, AMC23, and IFEval.

Critical strengths
  1. APC offers a useful accuracy–compute trade-off. On MATH-500, it reaches 88.8% at 682 TFLOPs, compared with 88.6% at 1,804 TFLOPs for particle filtering. It also outperforms best-of-N on IFEval at similar compute. The accounting includes both generation and reward evaluation.

  2. The annealing ablation supports the central design. In Table 3, applying ten corrector updates along the annealing path improves accuracy from 82.2% to 88.8% at nearly identical compute. This shows that the benefit comes from when refinement occurs, rather than simply adding more updates.

Critical weaknesses

The IFEval comparison leaves the sampler's contribution unclear. APC and QAlign differ in reward strength (20 versus 2) and output selection (highest reward versus ROUGE-1 MBR). The 77.9% versus 73.8% result compares these complete pipelines, but does not separate the effects of the sampler and these other choices. Controlled variants would help explain the gain while preserving the comparison between the original methods.

Questions for authors
  1. Could you supplement the IFEval results with controlled variants that separate the effects of the sampler, reward strength, and output selection? Please retain each method's tuned native protocol, alongside shared settings that keep both methods well-defined, and compare at similar compute budgets.

  2. Table 1 averages three seeds. Could you report their variability at a few matched-budget points in Figure 3, especially where APC and self-consistency are close?

1 comment
AuthorsOpenReview ↗

Official Comment by Authors for Reviewer M1X8

We thank the reviewer for the careful review and constructive feedback. We appreciate the positive assessment of the accuracy-compute trade-off and annealing design of APC. We address each point below and have incorporated the feedback into the revised manuscript, with the corresponding changes highlighted in blue for clarity.


W1 & Q1: [IFEval comparison] (Weaknesses 1) The IFEval comparison leaves the sampler's contribution unclear. APC and QAlign differ in reward strength (20 versus 2) and output selection (highest reward versus ROUGE-1 MBR). The 77.9% versus 73.8% result compares these complete pipelines, but does not separate the effects of the sampler and these other choices. Controlled variants would help explain the gain while preserving the comparison between the original methods.

(Questions 1) Could you supplement the IFEval results with controlled variants that separate the effects of the sampler, reward strength, and output selection? Please retain each method's tuned native protocol, alongside shared settings that keep both methods well-defined, and compare at similar compute budgets.

We additionally conduct controlled comparisons on IFEval to examine the effects of the sampler, reward strength, and final output selection. We use the same models and decoding settings at comparable compute budgets. For QAlign, we use seven transitions, the largest evaluated transition count whose computational cost does not exceed that of APC.

First, we compare APC and QAlign using the same reward strength β=20\beta=20 and highest-reward final answer selection for both methods. Under these shared settings, APC achieves higher accuracy than QAlign at comparable compute.

[Table 2 in the revised manuscript: Results from the controlled IFEval comparison.]

MethodAccuracy (%)TFLOPs
Best-of-N (N=28)76.0 ± 0.4252
QAlign76.4 ± 0.3240
APC (ours)77.9 ± 0.1253

We further vary reward strengths and final answer selection, keeping the remaining sampling settings fixed within each method. Highest-reward selection improves accuracy over ROUGE-1 MBR for both samplers, indicating that the final output selection rule affects instruction-following performance. With highest-reward selection, increasing β\beta from 2 to 20 improves the accuracy of APC, while QAlign remains similar. This may partly reflect annealing helping APC explore more concentrated reward-tilted distributions. Under each shared reward strength and selection rule, APC achieves higher accuracy than QAlign at comparable compute budgets.

[Table 13 in the revised manuscript: Reward strength and answer selection on IFEval.]

Samplerβ\betaHighest Reward Accuracy (%)ROUGE-1 MBR Accuracy (%)TFLOPs
Best-of-N (N=28)-76.0 ± 0.4-252
QAlign276.5 ± 0.273.3 ± 0.3240
QAlign2076.4 ± 0.374.4 ± 0.3240
APC (ours)277.0 ± 0.974.5 ± 0.9250
APC (ours)2077.9 ± 0.176.5 ± 0.4253

Q2: [Variability across random seeds] (Questions 2) Table 1 averages three seeds. Could you report their variability at a few matched-budget points in Figure 3, especially where APC and self-consistency are close?

We update Table 1 to report the mean and standard error across three random seeds for both accuracy and inference compute. The revised Figure 3 includes error bars across the evaluated compute budgets. We also report means and standard errors for the other experiments repeated across multiple random seeds, including those in Tables 2-4 of the revised manuscript. Self-consistency remains competitive at lower compute budgets, while APC achieves higher mean accuracy over a range of moderate compute budgets.

[Table 1 in the revised manuscript: Accuracy and inference compute on MATH 500 and AMC23, reported as mean ± standard error over three random seeds.]

MethodMATH-500 Accuracy ↑\uparrowMATH-500 TFLOPs ↓\downarrowAMC23 Accuracy ↑\uparrowAMC23 TFLOPs ↓\downarrow
Single Sample80.2 ± 0.710 ± 054.2 ± 5.115 ± 0
Greedy82.6 ± 0.010 ± 062.5 ± 015 ± 0
Selection over N i.i.d samples
Best-of-N87.9 ± 0.41,347 ± 164.2 ± 3.61,956 ± 9
SNIS87.8 ± 0.51,347 ± 163.3 ± 3.11956 ± 9
Self-Consistency88.0 ± 0.1643 ± 173.3 ± 2.9941 ± 5
Weighted Majority88.3 ± 0.61,347 ± 164.2 ± 3.61,956 ± 9
Search over partial responses
Beam Search85.0 ± 0.41,642 ± 4262.5 ± 2.92,984 ± 4
DVTS88.1 ± 0.21,834 ± 2462.5 ± 2.93,201 ± 124
Reward-guided samplers
QAlign87.8 ± 0.2956 ± 267.5 ± 2.51,410 ± 4
Step-level SMC86.7 ± 0.1911 ± 362.5 ± 3.81,626 ± 175
Particle Filtering88.6 ± 0.11,804 ± 2470.8 ± 3.33,703 ± 235
APC (ours)88.8 ± 0.7682 ± 1172.5 ± 2.9253 ± 3

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response by Authors

We thank all reviewers for their thoughtful comments and constructive suggestions. We appreciate the positive assessments of the accuracy-compute trade-off of APC (M1X8, DrVg, AvcD), its theoretical foundation (DrVg, AvcD), and its annealing design (M1X8, DrVg). Below, we summarize the main concerns and the additional evidence provided during the discussion. Changes in the revised manuscript are highlighted in blue.


  • Role of annealing, correction, and adaptive stopping (AvcD W1, Q1, Q3, Q4; DrVg Q2)

We clarify the contribution as an annealed framework for complete-response refinement with a target-preserving suffix corrector and answer-stability-based adaptive stopping. Theorem 1 establishes that the proposed corrector preserves each intermediate reward-tilted distribution.

The expanded three-seed ablations in Table 4 show that distributing the same corrector updates along the annealing path achieves 88.4% accuracy, compared with 84.4% when applying them directly at the final reward strength, at the same compute. Similar mean accuracies with the alternative corrector support flexibility in corrector choice. Also, Table R3 of our additional response to AvcD reports a statistically significant paired improvement over the no-corrector variant at comparable compute. The revised Figure 4 further compares adaptive stopping with fixed-stage stopping on the same trajectories. Together, these analyses support refining complete responses along the annealing path and adapting the refinement budget to each problem.


  • Variability and statistical analyses (M1X8 Q2; DrVg W3; AvcD Q3 and additional comment)

We report standard errors across three random seeds in Tables 1-4 and add error bars to Figure 3. In Table R3 of our additional response to AvcD, we additionally report paired bootstrap comparisons over evaluation problems after averaging per-problem differences across the repeated runs.


  • Wall-clock time analysis and practical trade-offs (AvcD W2 & Q2; DrVg W2)

The dedicated wall-clock study in Table 5 evaluates single-request and batched execution. At comparable accuracy, APC achieves the lowest TFLOPs and batched processing time among the compared methods, while sequential refinement increases single-request latency relative to independent sampling. We consider batched execution important in practice, particularly for throughput-oriented serving and large-scale evaluation, where multiple requests can be processed concurrently. Tables 11 and 12 further analyze how execution time varies with particle counts, annealing stages, and response lengths.


  • Broader task coverage and recent baselines (DrVg W1 & W2)

We add code generation experiments on HumanEval+ and MBPP+ in Table 3. At matched compute, APC improves pass@1 over Best-of-N from 70.9% to 73.2% on HumanEval+ and from 67.2% to 68.7% on MBPP+.

We also evaluate recent reward-guided sampling baselines using the same base and reward models within each task. As reported in Tables R1 and R2 of our response to DrVg W1, APC achieves higher accuracy at comparable or lower compute budgets.


  • Controlled IFEval comparisons and final answer selection (M1X8 W1 & Q1; DrVg Q1)

We examine the effects of reward strength and final answer selection through controlled comparisons in Tables 2 and 13. With shared reward strength and highest-reward selection, APC achieves 77.9%, compared with 76.4% for QAlign at comparable compute. Table 10 separately compares final answer selection rules on the same terminal populations. These analyses clarify the distinction between population refinement and downstream answer selection.


[Changes in the revised manuscript]

  • Section 2: Clarified the relationship to existing methods and included recent reward-guided sampling work.
  • Table 1 and Figure 3: Added standard errors across three seeds and error bars.
  • Table 2: Updated the IFEval comparison using shared reward strength and final answer selection.
  • Table 3: Added code generation results on HumanEval+ and MBPP+.
  • Table 4: Expanded the ablations to three seeds and added comparisons with an alternative corrector and a larger no-corrector population.
  • Table 5: Added single-request latency and batched processing time.
  • Figure 4: Added fixed-stage stopping comparisons.
  • Table 10: Added comparisons of final answer selection rules.
  • Tables 11 and 12: Added wall-clock time analysis across particle counts, annealing stages, and response-length groups.
  • Table 13: Added IFEval comparisons varying reward strength and final answer selection.

We sincerely appreciate the constructive feedback throughout the discussion, which has helped us clarify the contribution, strengthen the empirical evidence, and explain the practical trade-offs. We would be grateful for any further comments or suggestions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.