acceptodds
Withdrawn from ICLR 2027

Rethinking Power Sampling for LLM Reasoning: From Sharpening–Loss Theory to Marginal-Aware Rescaling

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

During decoding of large language models, temperature governs the trade-off between exploration and precision, yet locally optimal preferences do not necessarily yield globally better reasoning trajectories. Power Sampling addresses this by sharpening global sequence likelihood—favoring tokens that lead to confident continuations rather than merely locally confident ones—and nearly matches or even outperforms RL post-training on several reasoning benchmarks. However, its MCMC inference is costly, its joint-likelihood objective favors shorter outputs, and the preferred length-normalized mean likelihood cannot be sharpened directly. Best-of-N (BoN) is a natural alternative but has struggled to match this performance. We develop a unified sharpening-loss theory that explains this gap: Power Sampling and BoN share the same sequence-level sharpening mechanism, governed by sharpening direction and sharpening strength. A wrong direction concentrates probability on inferior candidates, causing preference loss, while insufficient strength leaves probability dispersed among distractors, causing dispersion loss. With length-normalized mean log-likelihood scored at an elevated temperature, BoN achieves a better sharpening direction, but its core bottleneck is insufficient strength: the probability of selecting high-scoring trajectories grows too slowly with the sampling budget. Building on this theory, we propose Marginal-Aware Rescaling (MAR), a training-free method that computes expected logits to identify the boundary of tokens worth exploring and thereby adaptively controls both direction and strength. MAR lowers the expected sampling budget needed to find the optimal candidate, and across diverse reasoning tasks and model families it matches or surpasses Power Sampling and RL post-training at substantially lower sampling cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLMs, reasoning, sampling, inference-time compute, power-sampling

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xqxkOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper proposes Marginal-Aware Rescaling (MAR), a training-free decoding method that suppresses deep-tail candidates while preserving competition near the peak. It combines this proposal with Best-of-N sampling and length-normalized, elevated-temperature scoring. A sharpening-loss framework motivates separating candidate selection from proposal coverage. MAR-BoN improves MATH500 accuracy over MCMC Power Sampling on all four evaluated models, and the reported cost comparison favors MAR. Results on other tasks are more mixed. The main concerns are the justification for the quadratic transform, errors and insufficient assumptions in the theoretical analysis, and gaps in the baseline comparisons and external-verifier claim.

Critical strengths
  1. The method addresses a useful problem: improving reasoning through inference-time sampling without additional training. MAR is straightforward to implement, adds O(V)O(V) work per decoding step, and supports batched autoregressive generation.

  2. The MATH500 results are encouraging across all four evaluated models. Table 4 also reports substantially lower token and wall-clock costs than MCMC under its stated accounting. The proposal and scoring ablations in Table 3 help distinguish the contributions of the two components.

  3. The distinction between score ranking and proposal coverage is a useful way to understand Best-of-N. The budget-scaling experiments support the argument that improving the candidate distribution can matter even when the selection rule is held fixed.

Critical weaknesses
  1. Appendix B.4 motivates the quadratic transform, but its design requirements do not uniquely determine it. For max-shifted logits z≤0z\leq0 and a=∣Et(Tb)∣>0a=|E_t(T_b)|>0, alternatives such as z~=−∣z∣p/ap−1\widetilde z=-|z|^p/a^{p-1} with p>1p>1 also preserve ordering and compress peak gaps Δ=−z\Delta=-z when 0<Δ<a0<\Delta<a, while expanding them when Δ>a\Delta>a. The claim that the theory fixes the functional form is therefore too strong. Comparing a few alternative exponents would help justify the quadratic choice.

  2. Several theoretical statements need correction. In Eq. (50), the derivative is dET[z]/dT=−Var⁡pT(z)/T2≤0d\mathbb E_T[z]/dT=-\operatorname{Var}_{p_T}(z)/T^2\leq0. Increasing TbT_b therefore moves the expected shifted logit away from the peak and widens the compression region, contrary to the discussion. For fixed non-degenerate logits, as Tb→0+T_b\to0^+, peak logits remain zero while non-peak transformed logits tend to −∞-\infty. Eq. (28) also drops prefix-dependent normalizers: pT(X)=p(X)1/T/∏tZT(ht)p_T(X)=p(X)^{1/T}/\prod_t Z_T(h_t), where ZT(ht)=∑vp(v∣ht)1/TZ_T(h_t)=\sum_v p(v\mid h_t)^{1/T}. Although the authors discuss local versus global normalization, the displayed sequence-level proportionality generally does not hold. Finally, the first-crossing assumption in Proposition B.11 does not exclude later crossings and is insufficient for the claimed unique maximum. The conditions supporting that conclusion should be explicit.

  3. The paper includes Standard BoN and a temperature-proposal baseline with the repaired score, but does not compare MAR with representative truncation or adaptive sampling methods discussed in Related Work. Comparisons using the same candidate budget and scoring rule would better establish the benefit of this rescaling. The accuracy tables also lack uncertainty estimates, which matters for small differences such as the 1.2-percentage-point HumanEval deficits relative to MCMC on three models.

  4. The introduction states that gains persist with external verifiers, but Appendix D evaluates consistency voting and joint log-likelihood. These experiments support robustness to alternative selection signals, not independent external verification. The claim should be narrowed unless additional verifier experiments are provided.

Questions for authors
  1. Could the authors correct Eqs. (28) and (50), state sufficient assumptions for Proposition B.11, and clarify how these changes affect the theoretical conclusions about MAR and elevated-temperature scoring?

  2. Could the external-verifier claim be supported with an independent verifier experiment, or revised to refer specifically to the alternative selection signals evaluated in Appendix D?

Reviewer_EerSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper treats power sampling and Best-of-N (BoN) as two ways of sharpening the model's distribution. It argues that BoN with a length-normalized likelihood score at a higher temperature is already a better target than power sampling. The problem is that good trajectories are rarely sampled. To fix this, the authors propose MAR, which rescales each logit by the expected logit at a boundary temperature. This pushes down the tail and gives tokens with similar scores a fairer chance. MAR-BoN with N=8 is compared with base sampling, low temperature, standard BoN, MCMC power sampling and GRPO on MATH500, HumanEval and GPQA with four models. The paper reports that it matches or beats these baselines at a much lower cost than MCMC.

Critical strengths

The method is simple and cheap. It only changes the logits at each step, adds very little overhead, and the N candidates can be generated in one batch, so it is much cheaper than MCMC power sampling. I liked the idea of separating the score used for selection from the proposal used for sampling, and the ablation shows that both parts help on Qwen2.5-Math-7B.

Critical weaknesses

In practice, MAR cuts the tail of the next-token distribution and makes the tokens near the top more even. The truncation samplers, such as top-p, min-p, p-less and min-k, also cut the tail, and with a warmer temperature on the kept tokens they also flatten the top. Still, the only proposal compared inside BoN is plain temperature sampling. So I can't tell what the quadratic rescaling adds over simple truncation.

I found the theory part long and hard to follow, and I am not sure how much of it the method actually needs.

Appendix D shows that plain majority voting over the same 8 MAR samples gets 0.824 on MATH500, which is higher than MAR-BoN with the paper's own repaired score (0.806). Majority voting over normal low temperature samples already gets 0.782, above power sampling (0.748). So self-consistency should be a baseline in Table 2.

I could not find error bars or multiple seeds, and Tg is set per benchmark to whichever temperature baseline did better. The paper does not mention a validation set, so this choice seems to be based on the test results. With 164 HumanEval questions and 198 GPQA Diamond questions, many of the 1 to 2 point differences are within noise.

The main claim of the paper is that MAR lets BoN catch up with power sampling, but a simpler method in the paper's own appendix does better, and the main table is missing it. The abstract also says MAR matches or beats RL post-training, but on Qwen3-4B MATH500 it gets 0.538 against 0.774 for GRPO. For this reason I don't think the results support the claim in the current version.

Questions for authors

Could you add majority voting with N=8 (from both the temperature proposal and the MAR proposal) to Table 2 for all models?

How was Tg chosen, and was a separate validation set used?

Could you also run BoN with top-p, min-p or p-less as the proposal, with the temperature tuned after truncation, using the same N and the same selection score?

Reviewer_pnmqOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces marginal-aware rescaling (MAR), a training-free decoding method which improves LLM reasoning by addressing the lock-fork trade-off in token sampling. In the paper, the authors develop a unified sharpening-loss framework in which they analyze temperature, Power Sampling, and Best-of-N (BoN) through preference loss and dispersion loss. Built upon this theory, MAR computes expected logits to identify the boundary of tokens worth exploring by applying nonlinear logit transformation to suppress low-probability distractors and presere exploration among competitive tokens. Experiments on MATH500, HumanEval, and GPQA Diamond demonstrate competitive performance of the proposed method.

Critical strengths
  1. The idea of unifying Power Sampling and Best-of-N under a sharpening-loss framework is a useful theoretical perspective for inference-time sampling.

  2. The proposed MAR is a simple yet effective training-free decoding method.

  3. The experimental results demonstrate promising improvements across reasoning benchmark.

Critical weaknesses
  1. The theoretical analysis does not fully support the main claim. The paper argues that the performance gap between Power Sampling and Best-of-N mainly comes from insufficient sharpening strength rather than direction. However, while the analysis explains the sampling limitations of Best-of-N, it does not clearly show that these limitations are the main reason for the performance gap between the two methods.

  2. Some theoretical derivations contain mathematical inconsistencies. In Appendix B.3, the identity relating the uniform average log-likelihood to entropy (following Eq. 40) does not hold in general. Since this is used to characterize the optimal sharpening strength, the validity of the subsequent classification of mild and severe forks requires further clarification.

  3. The evaluation lacks sufficient comparisons and ablations. The paper does not provide direct performance comparison between MAR and related adaptive decoding methods, such as Min-k Sampling[1]. Furthermore, the necessity of the proposed quadratic logit transformation over simpler alternatives remains insufficiently justified.

[1]Min-k Sampling: Decoupling Truncation from Temperature Scaling via Relative Logit Dynamics(ACL2026)

Questions for authors

See weakness.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.