acceptodds
Under review as a conference paper at ICLR 2027

Adaptive Power Sampling for LLM Reasoning

Abstract

Sequence-level power sampling has recently emerged as a training-free approach to reasoning by sampling from a sharpened output distribution of a base large language model (LLM). Nevertheless, existing methods typically sharpen the base model distribution uniformly across queries, overlooking variations in query difficulty and in how well the base model already handles each query. The goal of this work is to equip power sampling with query adaptivity. Theoretically, we show that the benefits of further sharpening are determined by the self-reward gap between correct and incorrect responses. Based on this insight, we propose Adaptive Power Sampling (APS), which adjusts the sharpening exponent on a per-query basis at test time using the relationship between answer agreement and the model's self-reward. Experiments across diverse reasoning tasks, including MATH500, HumanEval, and GPQA, show that APS consistently outperforms power sampling with a fixed sharpening exponent, without additional training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model reasoning, Adaptive power sampling, Self-reward alignment

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_ca5QOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The model assigns probabilities to many possible answers. Some answers have high probabilities, while others have low probabilities. Power sampling “sharpens” this distribution. It makes high-probability answers stand out more and makes low-probability answers less likely to be chosen. This can improve reasoning performance, because the model is more likely to choose answers that it considers reliable.

This paper finds that different problems need different levels of sharpening. For problems where the model is confident and the correct answers have clearly higher quality than the wrong ones, stronger sharpening can help the model choose good answers more firmly. But for problems where the model is uncertain, or where correct and wrong answers are hard to separate, too much sharpening may push the model toward a wrong answer.

To solve this problem, the authors propose Adaptive Power Sampling, or APS. APS adjusts the sharpening strength for each problem during test time. It uses two signals: how much the sampled answers agree with each other, and how much reward the model gives to these answers itself. The results show that APS works better than ordinary power sampling with a fixed sharpening strength.

Critical strengths

The work uses experiments to clearly show why different problems need different degrees of sharpening.

The paper identifies an effective proxy metric for estimating how solvable each problem is for the model.

Critical weaknesses

α\alpha undergoes multiple rounds of iterative updates, and a new sampling process is required after each update, which seems likely to increase the time overhead of the algorithm.

There are some writing issues and unclear descriptions in the paper. For example, in Table 1, some non-SOTA values are bolded. In addition, the introduction of sequence-level MCMC sampling is overly brief. Since this module plays an important role in the algorithm, introducing it only through citations make it difficult for readers to understand.

The paper conducts experiments only on models based on the Qwen2.5-7B architecture, lacking evidence for the generalizability of the proposed method.

The experimental analysis of the method is still limited to overall accuracy results, lacking ablation studies that specifically evaluate the effectiveness of individual APS components.

Questions for authors

How much inference time and sampling cost does APS require compared with other methods?

Does APS still work well on models with different architectures and sizes?

Why are the Power-SMC results reported in this paper different from those in the original paper [1]?

How would the results change if either vv or zz is removed from APS?

How would the results change if α\alpha is not adapted? For example, multiple α\alpha values could be used for sampling at the same time, and the final answer could be selected by voting.

[1] Azizi, Seyedarmin, et al. "Power-smc: Low-latency sequence-level power sampling for training-free llm reasoning." arXiv preprint arXiv:2602.10273 (2026).

4 comments
AuthorsOpenReview ↗

Response to Reviewer ca5Q

Thank you for your careful review. We believe the questions and suggestions you have raised are valuable and will help us improve this paper. Below are the authors’ clarifications regarding the issues you raised; we look forward to receiving your feedback.

1.Inference time and sampling cost (W1,Q1) We report mean end-to-end relative latency on Qwen2.5-Math-7B (normalized to single-response PS) after warm-up on a single GPU with batched parallel trajectories within each sampling round. We add PS-voting that generates 12 parallel responses at α=4\alpha=4 and selects the final answer by voting.

MethodMATH500LatencyHumanEvalLatencyGPQALatency
Base.4960.05×0.05\times.3290.06×0.06\times.2780.03×0.03\times
Power sampling (PS).7481.00×1.00\times.5731.00×1.00\times.3891.00×1.00\times
PS-voting, 12 responses.7721.38×1.38\times.5851.32×1.32\times.3991.42×1.42\times
APS, 12 responses in total.8142.48×2.48\times.6282.32×2.32\times.4292.65×2.65\times

APS adapts α\alpha through sequential K=4K=4 batches, which costs 2.48/2.32/2.65 times PS, and 1.80/1.76/1.87 times PS-voting. However, APS gains 6.6/5.5/4.0 percentage points over PS and 4.2/4.3/3.0 percentage points over PS-voting, which means APS's gain over PS-Voting makes up 64%/78%/75% of its total gain over PS. APS trades extra latency for substantial accuracy gains via test-time scaling.

2.Writing clarification (W2) Thank you for pointing these out. In the final version, we will underline only the best result per model/benchmark in Table 1 and add the MH acceptance rule and blockwise MCMC procedure to the appendix. These are the same in (https://arxiv.org/pdf/2510.14901).

3.Generalizability of APS (W3,Q2) Tables 1–2 show gains of APS on Qwen2.5-7B, Qwen2.5-Math-7B, DeepSeek-Math-7B and its RL variant. The Qwen models (7.6B parameters, 28 layers, a hidden dimension of 3584, and grouped-query attention with 28 query heads and 4 key/value heads) and DeepSeek-Math (6.9B parameters, 30 layers, a hidden dimension of 4096, and multi-head attention with 32 query heads and 32 key/value heads) differ in architecture and size. We further adopt Phi-3.5-mini-instruct (3.8B) and Qwen2.5-32B to show generalization across model sizes. Following the setup in the paper, the results are:

ModelMethodMATH500HumanEvalGPQA
32BBase.576.392.376
32BPS.738.567.383
32BAPS.776.811.455
3.8BBase.400.213.273
3.8BPS.508.732.364
3.8BAPS.538.738.379

APS outperforms PS on all three benchmarks at both 3.8B and 32B, showing that its gains extend across model sizes.

4.Clarification of Power-SMC results (Q3) Power-SMC was first introduced by [1], while it uses block N=64N=64 in the experiments. In our paper, Power-SMC (https://arxiv.org/abs/2607.09693) along with other baselines are tested with a candidate budget of N=16N=16 for fair comparison. This explains the difference from the Power-SMC results in [1].

5.Roles of v and z in APS (Q4) It is shown that the effect of sharpening depends on the covariance between correctness and model self-reward. In the proposed APS, viv_i uses answer agreement as a correctness surrogate, while ziz_i represents standardized differences in sequence log-likelihood. Here, standardization reduces the effect of score-scale differences on the update. Thus g^t=1Nt∑i=1Nt(vi−vˉt)zi,\widehat g_t=\frac{1}{N_t}\sum_{i=1}^{N_t}(v_i-\bar v_t)z_i, updates α\alpha through its sign. Removing either vv or zz therefore prevents APS from updating α\alpha.

6.Ablation of APS's adaptation (Q5) We compare APS with two controls to evaluate the contribution of α\alpha adaptation with Qwen2.5-Math-7B on MATH500, using the same experiment setup in the paper. They are: PS-voting: Generate 12 parallel responses at α=4\alpha=4 and select the final answer by voting; Static multiple exponents-voting: Generate 4 response at each α∈{2,4,8}\alpha\in\{2,4,8\} and vote ove all 12 responses.

Methodα\alpha strategyResponse budgetPass@1
APSAgreement-guided updates12.814
PS-votingFix α=4\alpha=412.772
Static multiple exponents-votingPreset 2,4,8\\{2,4,8\\}12.782

APS improves Pass@1 by 4.2 and 3.2 percentage points over PS-voting and Static multiple exponents-voting, respectively. PS-voting commits all 12 responses to one α\alpha, while static multiple exponents-voting allocates four responses to each preset α\alpha regardless of the query. APS uses early samples to adapt α\alpha to each query and then allocates the remaining budget at the selected exponent. Only responses at the selected exponent enter the final vote. By adapting to each query, APS outperforms fixed-exponent voting and static mixing within the same response budget.

AuthorsOpenReview ↗

A Brief Follow-up on Our Response

Dear Reviewer ca5Q,

Thank you again for your careful review. Our response provides additional end-to-end inference latency comparisons and experimental results across different model architectures and sizes. We also clarify the writing issues you raised and explain the differences in the Power-SMC experimental setup. Furthermore, we discuss the role of answer agreement and present additional ablation studies with non-adaptive sampling strategies.

We would be grateful if, when convenient, you could let us know whether our response has addressed your main concerns and whether any further clarification would be helpful.

Best regards,

The Authors

Thank you for your response. However, I still feel that the algorithm's runtime efficiency remains a limitation. I will therefore keep my original score unchanged.

AuthorsOpenReview ↗

Further Response on Runtime Efficiency and Matched-Latency Results

Thank you for following up. Test-time scaling typically intends to improve reasoning accuracy by allocating additional inference time and computational resources. Thus, the key consideration is how effectively these resources are utilized to achieve meaningful accuracy gains. For example, self-consistency shows that sampling more reasoning paths can improve the accuracy of the voted answer (https://arxiv.org/pdf/2203.11171). The gains also depend on how the compute resources are allocated for each query (https://arxiv.org/pdf/2408.03314). In our work, APS uses this additional time to adapt the sharpening exponent to increase the reasoning performance. Specifically, it uses answer agreement and sequence log-likelihood from earlier responses to decide how strongly to sharpen the distribution for later samples, then votes among responses drawn at the selected exponent. We generate the samples within each round in parallel. Each subsequent round should wait for the preceding responses to update α\alpha, introducing latency relative to fixed-exponent voting.

Accuracy gains beyond majority voting. As reported, APS achieves substantial improvements of 6.6, 5.5, and 4.0 percentage points over power sampling on MATH500, HumanEval, and GPQA, respectively. Even when fixed-exponent power sampling uses 12 responses, APS still gains 4.2, 4.3, and 3.0 points under the same response budget, accounting for 64%, 78%, and 75% of its total improvement over single-response PS. These gains are particularly meaningful given the strong voting baseline. To put these improvements into perspective, prior work on self-verification reports improvements of 1.2 and 3.0 percentage points over majority voting on MATH500 and AIME24, respectively, using a 16-sample budget (https://arxiv.org/abs/2506.01369), which provides useful context for the magnitude of our improvements. Overall, our results demonstrate that query-wise exponent adaptation provides meaningful accuracy gains beyond majority voting under the same response budget.

Matched-latency comparison. We further conduct a matched-latency comparison on MATH500 using Qwen2.5-Math-7B. Specifically, for each query, we first measure the end-to-end inference latency of APS with 12 responses. Then, we allocate the same time budget to fixed-exponent PS-voting, allowing it to generate the maximum number of responses within this budget. We report the mean of the per-query latency ratios, with APS as the 1.00×1.00\times reference. Meanwhile, we select the count using timing alone and use the same GPU, batched MH sampler, and KV cache policy for the methods.

MethodMean responses per queryRelative latencyPass@1
Fixed-exponent PS-voting (12-response reference)120.91×0.91\times0.772
Fixed-exponent PS-voting (matched per query)17.91.01×1.01\times0.776
APS121.00×1.00\times0.814

From the results, even when PS-voting can spend the full APS time budget on each query and generate 17.9 responses on average, APS retains a 3.8 percentage point Pass@1 advantage. Under the same per-query time budget, using early responses to adapt the exponent yields significantly higher accuracy than drawing a larger pool at a fixed exponent. We will include this comparison in the revision, and we would appreciate your reconsideration in light of these results.

Reviewer_pQYvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Power Sampling (APS), a training-free inference method that adjusts the sequence-level sharpening exponent separately for each query. Motivated by the relationship between correctness and sequence log-likelihood, APS uses the covariance between answer agreement and model log-likelihood to update the exponent and aggregates responses through majority voting. Experiments on mathematical reasoning, code generation, and scientific question answering report improvements over fixed-exponent power sampling.

Critical strengths
  1. The evaluation covers three reasoning tasks, multiple models, and an RL-post-trained model, and includes sensitivity analyses for the number of MCMC steps and the sampling budget.
  2. The reported improvements over standard Power Sampling on Qwen2.5-Math-7B are substantial enough to motivate further investigation of query-adaptive sharpening.
Critical weaknesses
  1. Theorem 1 reproduces the covariance derivative identity already established in Proposition 4 of Tomihari and Sato (2026), so the paper does not adequately distinguish its theoretical contribution from this cited prior work [1].
  2. APS combines adaptive exponent selection with multiple complete-response samples and majority voting, but the absence of a fixed-exponent baseline with the same sampling budget and aggregation rule prevents attribution of the gains to adaptivity.
  3. The theoretical analysis concerns true correctness, whereas the algorithm uses answer agreement without establishing or empirically validating that the resulting update direction aligns with improvements in correctness.
  4. The fixed-exponent baselines use α=4\alpha=4 without comparison against a validation-tuned exponent, despite the paper’s motivating experiment showing a smaller average oracle gap at α=2\alpha=2.
  5. Table 1 does not support an unqualified claim of superiority, since APS is not the strongest method on any Qwen2.5-7B task and only ties APPS on Qwen2.5-Math-7B GPQA, while its uniformly boldfaced row obscures the actual column maxima.
  6. The comparison with GRPO is not the same category (training v.s. inference time strategy).
  7. Important evaluation details remain unclear, particularly how answer agreement is defined for HumanEval programs and whether each APS sample in the Pass@k evaluation represents an individual trajectory or an entire adaptive sampling-and-voting run.

[1] Tomihari A, Sato I. Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation[J]. arXiv preprint arXiv:2605.04542, 2026.

Questions for authors
  1. Can the authors compare APS against fixed-exponent Power Sampling using the same 12-response budget, voting rule, and tie-breaking procedure, and separately report performance without final voting?
  2. How frequently does the agreement-based update direction match the direction estimated using ground-truth correctness, particularly when an incorrect answer dominates the sampled responses?
  3. Can the authors provide the actual configuration and sequence of exponent updates that produced α⋆=2.497\alpha^\star=2.497 in Table 6?
  4. How are HumanEval programs normalized or grouped for agreement and voting, and what precisely constitutes one sample in the reported Pass@1 and Pass@k results?
  5. Can the authors provide ablations that separately evaluate adaptive exponent updates and majority voting, including fixed-exponent sampling with identical voting and an increase-only exponent update strategy?
  6. Can the method extend to open-ended generation tasks, what the performance compare to different baseline.
4 comments
AuthorsOpenReview ↗

Response to Reviewer pQYv

Thank you for your careful review. We believe the questions will help us improve this paper. Below are our clarifications, and we look forward to your further feedback.

1.Theoretical distinction (W1) Thanks for the insightful question. Unlike [1], Theorem 1 specializes the covariance identity to the query-wise binary correctness probability, showing that the self-reward gap between correct and incorrect responses determines whether increasing α\alpha improves expected performance. Building on this, we propose APS, which adjusts α\alpha for each query using statistics computed from sampled responses. We will reclarify our contribution in the paper.

2.Ablation of adaptation (W2, Q1, Q5) We compare APS with two controls to evaluate the contribution of α\alpha adaptation with Qwen2.5-Math-7B on MATH500, using the same experiment setup in the paper. PS-voting: Generate 12 parallel responses at α=4\alpha=4 and select the final answer by voting; Static multiple exponents-voting: Generates 4 response at each α∈{2,4,8}\alpha\in\{2,4,8\} and votes ove all 12 responses.

Methodα\alpha strategyResponse budgetPass@1
APSAgreement-guided updates12.814
PS-votingFix α=4\alpha=412.772
Static multiple exponents-votingPreset 2,4,8\\{2,4,8\\}12.782

APS gains 4.2/3.2 percentage points over the benchmarks, respectively, supporting the benefit of query-wise α\alpha adaptation.

3.Ablation of answer agreement (W3, Q2, Q5) The ideal update direction depends on Cov(r⋆(Y),rself(Y))\mathrm{Cov}(r^{\star}(Y), r_{\mathrm{self}}(Y)). APS uses answer agreement vv, computed from the sampled responses, as a surrogate for correctness since golden answers are unknown at test time. We evaluate Qwen2.5-Math-7B on MATH500, following the experiment setup in the paper. To test the contribution of the update direction, we add Random-direction APS. At each update in (19), it computes ∣g^t∣|\widehat g_t| from its current response batch and randomly chooses the sign with probability 1/21/2: $ \alpha_{t+1} =\mathrm{clip}(\alpha_t\pm\eta|\widehat g_t|,\alpha_{\min},\alpha_{\max})$. We also add increase-only APS, which instead replaces g^t\widehat g_t with max⁡(g^t,0)\max(\widehat g_t,0). This variant could only increase or retain α\alpha.

Methodα\alpha strategyResponse budgetPass@1
APSAdaptive α\alpha12.814
Random-direction APSRandomize α\alpha direction12.766
Increase-only APSIncrease or retain α\alpha12.776

APS performs best under the same budget, supporting that the adaptation guided by answer agreement as a proxy is beneficial.

4.Baseline choice (W4, W6) Fig.1 is a motivating example instead of a α\alpha selection experiment, showing that preferred exponents varies across queries. In Fig.1, α=2\alpha=2 has a lower average gap than α=4\alpha=4 only indicates that α=2\alpha=2 performs better on this sampled subset and does not justify its use as a universal fixed exponent. Thus, following existing literature (https://arxiv.org/pdf/2510.14901; https://arxiv.org/pdf/2602.10273), we adopt α=4\alpha=4 for fixed-exponent baselines, and the reported GRPO in Table 1 is a post-training baseline, which will be clarified in the final version.

5.Writing clarification (W5) We will underline only the best result per model/benchmark in Table 1. APS exceeds the sampling baselines in most comparisons and GRPO in six of nine settings.

6.Clarification for HumanEval (W7, Q3, Q4) For HumanEval, we group completed function bodies by Python abstract syntax tree (AST) after removing leading docstrings and source locations; formatting and comments do not affect groups.

Following (https://arxiv.org/pdf/2510.14901), one evaluation sample for APS is the final program returned by one complete APS run. Thus, Pass@1 evaluates one complete APS output. Meanwhile, for Pass@kk, we run APS kk times and mark a problem solved if any output passes the test cases of the problem.

For the change of α\alpha, we will provide the specific configuration used for Tables 5-7 in the final version, and the sequence of exponent update in Table 6 is

Iterationαt\alpha_t
14.000
23.200
32.497

7.Extension to open-ended generation tasks (Q6) APS can extend to open-ended generation by adjusting the answer agreement in (19) with a semantic agreement signal over sampled responses, as studied in (https://arxiv.org/pdf/2601.02535). Let y1,…,yKy_1,\ldots,y_K be the KK sampled responses to one query. We compute their pairwise semantic similarities $S_{ij}=\mathrm{sim}(y_i,y_j) $, and cluster the responses using SS. If c(i)c(i) is the cluster of response yiy_i, define its agreement score as $ v_i=\frac{1}{K}\sum_{j=1}^{K}\mathbb{1}[c(i)=c(j)]. $ Thus, viv_i is the fraction of sampled responses that are semantically consistent with yiy_i. We would leave the evaluation of this extension to future work.

AuthorsOpenReview ↗

A Brief Follow-up on Our Response

Dear Reviewer pQYv,

Thank you again for your valuable review. Our response clarifies the theoretical distinction from prior work and the reasons behind the baseline choices. We also present additional ablation studies comparing exponent adaptation with fixed-exponent sampling strategies and validating the role of answer agreement in guiding adaptive sampling. Furthermore, we address the unclear writing issues you raised.

We would be grateful if, when convenient, you could let us know whether our response has addressed your main concerns and whether any further clarification would be helpful.

Best regards,

The Authors

Sorry for my late response and thank you for your follow-up experiments, which partly answer my questions. The voting and directional ablations meaningfully strengthen the evidence for APS. However, my concern in W4 remains unresolved: the fixed-exponent comparison uses only alpha=4, while voting across {2,4,8} is not equivalent to evaluating a validation-tuned fixed exponent. Since the central contribution is query-wise adaptation, it remains unclear that reported gains require such adaptation or could be achieved with a better fixed exponent under the same sampling budget and aggregation rule. I suggest the authors could further ablations on that part and adequately discuss APS's theoretical contribution from the prior work in method section. I keep my score.

AuthorsOpenReview ↗

APS Outperforms the Best Fixed-Exponent Baseline

Thank you for following up. Given the limited time remaining in the rebuttal period, we prioritized an additional experiment to test whether a better fixed exponent can recover APS’s gains.

Comparison with the best fixed exponent. Using Qwen2.5-Math-7B, we evaluate fixed-exponent PS-voting at α∈2,2.5,3,4,6,8\alpha\in\\{2,2.5,3,4,6,8\\} on the full MATH500 dataset. We select the single exponent with the highest mean accuracy across two random seeds and use it for all queries. This deliberately favors the fixed baseline by allowing exponent selection using the test labels: on the evaluated runs, a validation-selected exponent from the same candidate set cannot outperform this best observed choice.

All methods use the same sampling settings and a total budget of 12 responses per query. Fixed-exponent PS votes over all 12 responses; APS votes among responses at its selected exponent.

MethodExponent selectionMATH500 Pass@1
PS-votingFixed at α=4\alpha=477.2%
Best fixed-exponent PS-votingSelected on full MATH500: α⋆=3\alpha^\star=378.4%
APSQuery-wise adaptation81.4%

APS still outperforms the best fixed exponent by 3.0 percentage points. Thus, even with test-set-informed selection, a better global exponent does not recover APS’s gains among the evaluated candidates.

The offline search across six candidate exponents also incurs additional sampling cost, which is excluded from this comparison. Although this cost can be amortized by reusing the selected exponent, selecting the best exponent for new queries in the same way would require their ground-truth answers. APS instead adjusts the exponent using the current query’s sampled responses, without ground-truth answers or a separate parameter sweep, within the same budget.

Theoretical contributions. Our analysis characterizes when sharpening helps or hurts a query and how this changes with the exponent. Corollary 1 links the local accuracy change to the correct–incorrect mean log-likelihood gap; Lemma 1 characterizes how this gap evolves, explaining why a beneficial update direction can reverse. These results motivate query-wise updates that can either strengthen or relax sharpening. We will clarify our contributions and distinctions to the paper (https://arxiv.org/pdf/2605.04542).

We will include these results in the revision and would appreciate your reconsideration in light of this additional evidence.

Reviewer_4bNsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Adaptive Power Sampling (APS), which adjusts the sequence-level sharpening exponent for each query using answer agreement and sequence log-likelihood. It provides a theoretical analysis of how sharpening affects expected correctness and evaluates APS on mathematical reasoning, coding, and scientific question answering. MCMC is not time consuming. I concern with the latency.

Critical strengths
  1. The theoretical characterization clearly explains when stronger sharpening helps or hurts expected correctness, providing a reasonable motivation for query-dependent adaptation.
  2. The reported improvements over fixed-exponent power sampling span several models and benchmarks.
Critical weaknesses
  1. APS requires repeated proposal generation and likelihood evaluation within MCMC, along with multiple responses to adapt α\alpha. The paper does not report end-to-end latency per problem or establish an accuracy–cost advantage over standard sampling and fixed-exponent power sampling with majority voting. It therefore remains unclear whether the accuracy gains justify the additional computation.
  2. The paper lacks an ablation that disables adaptation while keeping the sampling budget and aggregation procedure unchanged. It also does not directly validate whether answer agreement reliably identifies beneficial update directions, although the theory relies on true correctness.
  3. The experiments cover only one model and evaluation benchmark, with vanilla GRPO as the sole baseline. APS requires 1.41× the training time for the same number of updates. Comparisons with fixed-exponent sharpened rollouts and dynamic sampling are needed to establish whether adaptive sharpening offers a competitive accuracy–cost trade-off.
Questions for authors
  1. How much does APS increase end-to-end latency per problem relative to standard decoding and fixed-exponent power sampling? What changes to vLLM are required, and how are batching and KV-cache reuse handled?
  2. Section 5.1 motivates answer agreement as a proxy for correctness through self-consistency. Is there evidence that this proxy also reliably indicates whether increasing or decreasing the sharpening exponent will improve correctness?
  3. Algorithm 2 uses sharpened rollouts followed by a standard GRPO update. Are the likelihood ratios computed using the original policy probabilities, or do they account for the sharpened sampling distribution?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 4bNs

Thank you for your careful review. We believe the questions you have raised are valuable and will help us improve this paper. Below are our clarifications; we look forward to your further feedback.

1.Inference time and sampling cost (W1,Q1) The inference uses Transformers 4.51.1, and vLLM is for RL post-training in Appendix C. KV caching is within each generation call; MH retains prefix tokens/log-probabilities but re-prefills the prefix for each new proposal. The setup below batches trajectories per round.

We report mean end-to-end relative latency on Qwen2.5-Math-7B (normalized to single-response PS) after warm-up on a single GPU with batched parallel trajectories within each sampling round. We add PS-voting that generates 12 parallel responses at α=4\alpha=4 and selects the final answer by voting.

MethodMATH500LatencyHumanEvalLatencyGPQALatency
Base.4960.05×0.05\times.3290.06×0.06\times.2780.03×0.03\times
Power sampling (PS).7481.00×1.00\times.5731.00×1.00\times.3891.00×1.00\times
PS-voting, 12 responses.7721.38×1.38\times.5851.32×1.32\times.3991.42×1.42\times
APS, 12 responses in total.8142.48×2.48\times.6282.32×2.32\times.4292.65×2.65\times

APS adapts α\alpha through sequential K=4K=4 batches, which costs 2.48/2.32/2.65 times PS, and 1.80/1.76/1.87 times PS-voting. However, APS gains 6.6/5.5/4.0 percentage points over PS and 4.2/4.3/3.0 percentage points over PS-voting, which means APS's gain over PS-Voting makes up 64%/78%/75% of its total gain over PS.

2.Ablation and validation of answer agreement (W2,Q2) APS uses answer agreement as an observable surrogate for correctness. Studies have proved that agreement across sampled responses is informative for distinguishing correct from incorrect answers (https://arxiv.org/pdf/2203.11171; https://arxiv.org/pdf/2305.11860; https://arxiv.org/pdf/2403.09849).

We evaluate Qwen2.5-Math-7B on MATH500, following the experiment setup in the paper. The static multiple exponent-voting generates 4 responses at each α∈{2,4,8}\alpha\in\{2,4,8\} and votes over all 12 responses. To test the contribution of the update direction, we add Random-direction APS. At each update in (19), this control computes ∣g^t∣|\widehat g_t| from its current response batch and randomly chooses the sign with probability 1/21/2: αt+1=clip⁡(αt±η∣g^t∣,αmin⁡,αmax⁡).\alpha_{t+1} =\operatorname{clip}(\alpha_t\pm\eta|\widehat g_t|,\alpha_{\min},\alpha_{\max}). This comparison tests if agreement provides more useful update directions than random choices.

Methodα\alpha strategyResponse budgetPass@1
APSAgreement-guided updates12.814
PS-votingFix α=4\alpha=412.772
Static multiple exponents-votingPreset 2,4,8\\{2,4,8\\}12.782
Random-direction APSRandomize the update direction12.766

Under the same response budget, APS achieves the best performance among all compared methods. Notably, random-direction APS degrades performance below even PS-voting. This supports that the adaptation guided by answer agreement as a proxy is beneficial.

3.Likelihood ratios and training efficiency in APS-assisted GRPO (W3,Q3) In APS-assisted GRPO, likelihood ratios are computed by the original policy probabilities: ρi,t(θ)=pθ(yi,t∣x,yi,<t)pθold(yi,t∣x,yi,<t).\rho_{i,t}(\theta)=\frac{p_\theta(y_{i,t}\mid x,y_{i,<t})}{p_{\theta_{\mathrm{old}}}(y_{i,t}\mid x,y_{i,<t})}. As shown in Appendix C, APS selects α\alpha to favor mixed groups with nonzero advantages. This likelihood ratio principle is similar to DAPO (https://arxiv.org/pdf/2503.14476). AGPO (https://arxiv.org/pdf/2605.20722) also adapts rollout temperature while computing the clipped ratio from original model probabilities.

We add two training controls on Qwen2.5-7B using the same setup in the paper. Fixed-exponent PS-GRPO: Fix α=4\alpha=4 and omit pilot-based exponent selection; GRPO-Dynamic Sampling: Sample from the original policy, discard all-correct and all-incorrect groups, and continue until mixed groups fill the training batch. This control adds DAPO's group filtering and batch refilling to GRPO. Training cost and MATH500 Pass@1 are:

MethodUpdatesRelative timePass@1
Base, no GRPO--49.0%
Vanilla GRPO801.00×1.00\times52.0%
Fixed-exponent PS-GRPO670.98×0.98\times53.0%
Fixed-exponent PS-GRPO801.31×1.31\times54.6%
GRPO-Dynamic Sampling540.96×0.96\times53.4%
GRPO-Dynamic Sampling801.54×1.54\times55.4%
APS-assisted GRPO620.96×0.96\times54.2%
APS-assisted GRPO801.41×1.41\times57.6%

Relative time is normalized to Vanilla GRPO at 80 updates. APS achieves higher accuracy than both controls at comparable training time. At 80 updates, it is more accurate than Dynamic Sampling and requires less training time.

AuthorsOpenReview ↗

A Brief Follow-up on Our Response

Dear Reviewer 4bNs,

Thank you again for your detailed review. Our response provides additional end-to-end latency comparisons with standard power sampling and fixed-exponent power sampling with majority voting. We also present ablation studies to validate the effectiveness of answer agreement in guiding adaptive sampling. Furthermore, we clarify the likelihood ratio computation and provide additional training efficiency comparisons with fixed-exponent and dynamic sampling methods.

We would be grateful if, when convenient, you could let us know whether our response has addressed your main concerns and whether any further clarification would be helpful.

Best regards,

The Authors

Thank you for the additional experiments. The ablations address several of my concerns, and APS outperforms fixed-exponent voting at comparable latency on MATH500. My main concern is still response time. On the 7B model, APS takes roughly 50 times as long as standard decoding on MATH500, and the wait could be longer with larger models. For interactive use, I’m not convinced the gains justify that delay. I’m keeping my rating at 2 (weak reject).

AuthorsOpenReview ↗

APS’s Accuracy–Latency Trade-off and Matched-Latency Advantage

Thank you for acknowledging that the ablations address several of your concerns and that APS outperforms fixed-exponent voting at comparable latency.

Accuracy–latency trade-off. Similar latency multipliers have been reported for other reasoning methods: approximately 47× over greedy CoT for Tree-of-Thoughts with LLaMA2-7B across seven tasks (https://arxiv.org/pdf/2406.09136), and 53× over single-path CoT for Semantic Exploration with Llama3-8B-Instruct on ARC (https://arxiv.org/pdf/2501.05752). APS likewise offers configurable response budgets and MH steps, as shown in Appendix D.3, allowing users to spend more time when answer quality matters more than an immediate response.

Cost relative to power sampling. The approximately 50× figure is APS’s total latency relative to Base decoding. In the same MATH500 timing comparison, fixed-exponent PS already takes approximately 20× as long as Base. APS takes 2.48× as long as PS while improving accuracy from 74.8% to 81.4%.

Matched-latency advantage. Giving fixed-exponent sampling the same time does not recover APS’s accuracy. With matched per-query time budgets, PS-voting generates 17.9 responses on average and achieves 77.6%, while APS achieves 81.4% with 12 responses, retaining a 3.8 percentage-point advantage. This directly demonstrates that adapting the exponent uses the available inference time more effectively than drawing additional samples at a fixed exponent.

We would appreciate your reconsideration of APS’s contribution in light of the rationality of additional latency and the demonstrated accuracy advantage under the same time budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.