acceptodds
Withdrawn from ICLR 2027

A Critical look at Power Sampling when Test-Time Sharpening LLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Reinforcement learning with verifiable rewards (RLVR) has substantially improved LLM reasoning, yet growing evidence suggests it mainly performs distribution sharpening: concentrating probability mass on reasoning paths the base model already produces. Recent work therefore seeks RLVR's gains without needing reward signals, by sampling from the power distribution via Markov chain Monte Carlo or sequential Monte Carlo. We show that these samplers fail to converge to under realistic compute budgets, and that their gains are largely recreated via simple Best-of- with appropriate grading . This suggests they succeed by selecting high-probability sequences, not by approximating , raising a natural question: is the power distribution itself a worthwhile target? We test this hypothesis directly by fine-tuning the base model to match , sidestepping sampler convergence issues entirely. Despite approximating the target far more faithfully than existing samplers, this strategy yields gains that are noisy and specific to certain models. Together, our results suggest that the value of power sampling lies not in faithfully targeting , but in simply extracting sequences the base model already deems likely.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
LLM fine-tuning, Variational Inference, Markov Chain Monte Carlo

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8mpAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper argues that the previous power sampling papers’ explanation is wrong: their samplers don’t actually sample from the power distribution πα\pi_\alpha, and their gains can be matched by simple inference-time alternatives such as Best-of-N by likelihood.

They propose a fine-tuning method (Power-RLOO) that trains the model so ordinary sampling directly follows πα\pi_\alpha. They use it to test whether πα\pi_\alpha itself improves reasoning, and find the gains are inconsistent.

Critical strengths
  1. The paper offers a timely critique of existing power-sampling methods that we do not need fancy algorithms for approximating πα\pi_\alpha. It also shows that simple inference time strategies is enough for performance improvement, saving actual training cost comparing to previous methods;
  2. The finetuning part of the paper provides insight of whether optimizing for πα\pi_\alpha is beneficial for optimizing reasoning performance, which is important for this line of research.
Critical weaknesses
  1. The experiment part of the paper has incomplete results. Table 1 reports N/A for Power-RLOO on Qwen2.5-7B-Base; the authors should still report its accuracy and output lengths, and the table omits key baselines such as MCMC [1].
  2. The experiment setup is only restricted to older model families, and Qwen2.5 has been shown to suffer from benchmark contamination [2]. The author should justify the motivation of studying on these models and provide at least one results from recent models such as Olmo3 [3] or Qwen3 [4] families, or report results on a contamination-free benchmark.
  3. It is unclear how πα\pi_\alpha is measured after training. The paper proposed a method that is optimized towards πα\pi_\alpha, but only second-hand evidence such as likelihood histograms and training loss drop. The reverse-KL objective is mode-seeking, so Power-RLOO may itself under-cover πα\pi_\alpha . A controlled experiment where πα\pi_\alpha is tractable (e.g., a small model with short sequences) would substantially strengthen both claims.
  4. The paper lacks statistical reliability. The standard deviation of MATH500 and GPQA is about 2 absolute points, and the differences of different methods in the main table is within such range. The author should justify how the number in the table is obtained clearly.

[1] Karan, Aayush, and Yilun Du. "Reasoning with sampling: Your base model is smarter than you think." International Conference on Learning Representations. Vol. 2026. 2026.

[2] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).

[3] Olmo, Team, et al. "Olmo 3." arXiv preprint arXiv:2512.13961 (2025).

[4] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).

Questions for authors

See weakness above.

Reviewer_KyM1OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper examines whether the reasoning gains of sequence-level power sampling depend on accurately approximating the power distribution. The authors study mixing and particle degeneracy in existing MCMC and SMC approaches and introduce a simple Best-of-N baseline with periodic selection of high-likelihood prefixes. They also propose Power-RLOO, which fine-tunes a model through a reverse-KL objective using base-model log-probabilities, low-temperature parameterization, and a leave-one-out policy-gradient estimator. Experiments cover three models on MATH500 and GPQA Diamond. Simple selection methods approach Power-SMC in several settings, while Power-RLOO produces model-dependent gains and short-output collapse on Qwen2.5-7B-Base. The paper argues that much of the observed benefit comes from likelihood-guided selection rather than faithful approximation of the power distribution.

Critical strengths
  1. The paper asks a meaningful question about the gap between a sampler's theoretical target and its behavior under practical budgets. Distinguishing these two is useful for understanding the mechanisms behind test-time scaling.

  2. The simple prefix-selection baseline is informative. It matches Power-SMC at 76.9 on Qwen2.5-7B-Instruct on MATH500 and comes close on the other two models. This motivates stronger likelihood-guided selection baselines when evaluating more complex samplers.

  3. Power-RLOO has a clear mathematical motivation. The reverse-KL objective and low-temperature parameterization provide a useful way to study sequence-level sharpening. The appendices explain the gradient estimator and provide an optimization comparison supporting the warm start.

Critical weaknesses
  1. The quality of the approximation to the target distribution is insufficiently validated. The central argument relies on Power-RLOO approximating the power distribution more faithfully, but decreasing training loss and changes in sample likelihoods do not establish this premise. In particular, the paper does not explain how the unknown normalizing constant is handled in its KL estimates. The conclusion that gains arise from likelihood-based selection rather than accurate sampling therefore needs more direct validation of distributional error.

  2. The pass@k comparison uses different sampling protocols. SMC is evaluated using particles that share ancestry within a single run, whereas the comparison methods use independently generated responses. Correlation among particles limits within-population coverage, so this result illustrates particle degeneracy but does not directly establish lower overall diversity for SMC. A comparison using independent SMC runs, even on a smaller subset of questions, would be more convincing.

Questions for authors

Please see weaknesses.

Reviewer_kG3fOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper examines multiple power sampling methods that sample from the power distribution πα(x∣z)\pi_\alpha(x|z) via Metropolis-Hastings MCMC (Karan & Du, 2025), Monte-Carlo rollout estimation (Ji et al., 2026) and SMC (Azizi et al., 2026).

Contributions:

  1. Diagnostics arguing that existing samplers do not converge to πα\pi_\alpha at practical budgets: Metropolis-Hastings acceptance collapses to greedy hill-climbing as alpha grows, and Power-SMC suffers effective sample size collapse and ancestral degeneracy, so both act as alpha-insensitive likelihood-guided search.
  2. A Best-of-N-with-resampling baseline claimed to largely recreate Power-SMC's gains, supporting the hypothesis that the samplers succeed by selecting high-likelihood sequences rather than by approximating πα\pi_\alpha.
  3. Power-RLOO: a RL fine-tuning method with self-reward r(x)=β(α−1)log π0(x∣z)r(x) = \beta(\alpha-1) log\ \pi_0(x|z) with a trackable training-loss proxy for fidelity to πα\pi_\alpha.
  4. The central experimental finding: despite higher fidelity, faithfully targeting πα\pi_\alpha yields only modest, model and task-dependent gains over low-temperature decoding and Best-of-N selection. The authors conclude that the value of power sampling lies in extracting high-probability sequences, not in approximating πα\pi_\alpha, and recommend the field reconsider power sampling as a general path to reasoning improvement.
Critical strengths
  1. Rather than critiquing power samplers only through convergence diagnostics, the authors sidestep sampler convergence by fine-tuning the model toward πα\pi_\alpha thereby separating "is πα\pi_\alpha a good target?" from "can we sample πα\pi_\alpha?". This is a correct experiment for the ongoing debate over whether RLVR gains are only distribution sharpening, and the resulting findings could be informative observations that no prior work has surfaced.

  2. The failure-mode analyses of the Metropolis-Hastings and Power-SMC samplers are clear and convincing. The collapse and ancestral-degeneracy analysis of Power-SMC is a good illustration of the path-degeneracy problem of SMC, and the Metropolis-Hastings acceptance analysis plausibly demonstrates collapse to greedy hill-climbing as alpha grows.

Critical weaknesses
  1. The reported evidence cannot support the paper's headline conclusions, and parts of the narrative directly contradict the paper's own Table 1. All experiments use a single seed with no uncertainty quantification, essentially, every method ordering the narrative lies within noise. Worse, Section 5.2 states that Best-of-N-mean outperforms Power-RLOO on GPQA "for every model" while Table 1 shows Power-RLOO winning on both models with reported numbers. The abstract's claim that power distribution gains are "largely recreated" by simple Best-of-N is contradicted by the GPQA column. Contribution 4's claim that gains beyond low-temperature decoding arise "only when the base model is well-aligned" is contradicted by Phi-3.5 on MATH500 (43.6 vs 41.7).

  2. The central premise that Power-RLOO approximates πα\pi_\alpha substantially more faithfully than the samplers is never truly validated. Since πα\pi_\alpha is intractable, fidelity is only inferred from: (i) estimate of D_KL(pi_theta || pi_alpha) in Figure 5 (ii) a log-likelihood histogram from a single prompt. The optimization minimizes the KL(pi_theta || pi_alpha) over a restricted family (frozen base + LoRA) whose optimum is generally not πα\pi_\alpha. If Power-RLOO's fidelity is not established, the core syllogism (samplers fail -> we approximate faithfully -> faithful approximation doesn't help) collapses, and "πα\pi_\alpha is not a worthwhile target" is unsupported (a conclusion that the table 1's single largest effect in fact contradicts).

Questions for authors
  1. What exact quantity is plotted in Figure 5? Can you validate Power-RLOO's fidelity and the samplers' non-convergence on a tractable testbed where ZαZ_\alpha is exactly computable?

  2. What prompt dataset is Power-RLOO trained on, and is the GPQA evaluation zero-shot transfer from a math-trained adapter? To obtain Figure 4, did you retrained 4 checkpoints of Power-RLOO?

  3. You do not really show that estimating the twist with MC-rollout (Ji et al., 2026) is not faithful to πα\pi_\alpha at practical budgets. What evidences do you have to support this claim?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.