- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper argues that the previous power sampling papers’ explanation is wrong: their samplers don’t actually sample from the power distribution , and their gains can be matched by simple inference-time alternatives such as Best-of-N by likelihood.
They propose a fine-tuning method (Power-RLOO) that trains the model so ordinary sampling directly follows . They use it to test whether itself improves reasoning, and find the gains are inconsistent.
Critical strengths
- The paper offers a timely critique of existing power-sampling methods that we do not need fancy algorithms for approximating . It also shows that simple inference time strategies is enough for performance improvement, saving actual training cost comparing to previous methods;
- The finetuning part of the paper provides insight of whether optimizing for is beneficial for optimizing reasoning performance, which is important for this line of research.
Critical weaknesses
- The experiment part of the paper has incomplete results. Table 1 reports N/A for Power-RLOO on Qwen2.5-7B-Base; the authors should still report its accuracy and output lengths, and the table omits key baselines such as MCMC [1].
- The experiment setup is only restricted to older model families, and Qwen2.5 has been shown to suffer from benchmark contamination [2]. The author should justify the motivation of studying on these models and provide at least one results from recent models such as Olmo3 [3] or Qwen3 [4] families, or report results on a contamination-free benchmark.
- It is unclear how is measured after training. The paper proposed a method that is optimized towards , but only second-hand evidence such as likelihood histograms and training loss drop. The reverse-KL objective is mode-seeking, so Power-RLOO may itself under-cover . A controlled experiment where is tractable (e.g., a small model with short sequences) would substantially strengthen both claims.
- The paper lacks statistical reliability. The standard deviation of MATH500 and GPQA is about 2 absolute points, and the differences of different methods in the main table is within such range. The author should justify how the number in the table is obtained clearly.
[1] Karan, Aayush, and Yilun Du. "Reasoning with sampling: Your base model is smarter than you think." International Conference on Learning Representations. Vol. 2026. 2026.
[2] Shao, Rulin, et al. "Spurious rewards: Rethinking training signals in rlvr." arXiv preprint arXiv:2506.10947 (2025).
[3] Olmo, Team, et al. "Olmo 3." arXiv preprint arXiv:2512.13961 (2025).
[4] Yang, An, et al. "Qwen3 technical report." arXiv preprint arXiv:2505.09388 (2025).
Questions for authors
See weakness above.