acceptodds
Under review as a conference paper at ICLR 2027

Sharpen Without Search: On-Policy Distillation of Sequence-Level Power Distribution

Abstract

A language model can give a correct answer more probability than any single incorrect answer and still usually sample an incorrect one, because the incorrect answers together hold more probability. The _power distribution_ assigns each complete answer a probability proportional to the model's probability of that answer raised to a power greater than one, which moves probability toward the answers the model considers most likely. We call this sharpening. Sampling from the power distribution improves reasoning accuracy without changing the model's parameters, but it requires generating and scoring many candidate answers per query. We show that a model can instead be trained once to produce such answers in one ordinary generation. Our method, on-policy power distillation (OPPD), runs a sequential Monte Carlo sampler in which the model being trained generates candidate answers and a frozen teacher's power distribution weights them. The same teacher probabilities give each complete answer a training weight for weighted maximum likelihood. Training raises single-generation accuracy by up to points on MATH500 and on GSM8K over the untrained model at the same temperature, and one generation of the trained model scores and points above published power sampling with candidates, obtaining 94% of the gain that candidates give the untrained model. To place the size of this gain in context, we also train GRPO, a reinforcement-learning method that rewards answers verified as correct, from the same checkpoint with the same budget: oppd scores , and points higher on MATH500, GSM8K and AIME while using no reference answers. The two methods use different information and are complementary rather than alternatives, and applied after GRPO OPPD adds up to points more. Although it trains only on mathematics questions, OPPD raises HumanEval code accuracy by up to points. We measure the sharpening a model absorbs as an exponent fitted between its token probabilities and those of the untrained model. One loss coefficient moves this exponent between and , against for ordinary on-policy distillation, and it rises mostly on the answers the model writes itself. The gains hold across model families and sizes, including a model already trained with verified rewards, on which lowering the sampling temperature gives nothing while OPPD adds points on MATH500.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Power Distribution, LLMs, On-policy Distillation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_A2mDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes OPPD, which trains a student to approximate a teacher's sequence-level power distribution without correctness labels. It combines on-policy sequential Monte Carlo sampling with importance-weighted training derived from a Rényi-divergence objective and an optional token-level anchor. The experiments report substantial improvements on mathematical reasoning and reduced inference-time sampling requirements.

Critical strengths

S1: The reported improvements are substantial and practically relevant. Table 1 shows gains over both the original model and lower-temperature decoding, supporting the claim that training can absorb benefits that token-level temperature changes alone do not provide.

S2: The paper distinguishes sequence-level sharpening from local temperature scaling and develops an on-policy sampling and training procedure around that distinction. Since amortizing power sampling through distillation already appears in the prior work discussed in Section 6, the specific combination of sampling and training weights is the most consequential methodological contribution.

Critical weaknesses

W1: The connection between the stated sequence-level objective and the implemented blockwise loss is unclear. Section 3.4 derives importance weights for complete sequences, but then states that the training-weight accumulator restarts at each block boundary. It subsequently describes the resulting gradient estimate as consistent, with bias of order 1/N1/N.

If earlier token losses retain only their block-local weights, later suffix likelihoods do not directly reweight those decisions. This can differ from the gradient of the stated sequence-level objective, including when no resampling occurs. The main text does not explain how these descriptions are reconciled. This concerns the central justification of the method and is the main reason for my current recommendation.

Questions for authors

Q1: Please state the exact loss used across multiple blocks. Are weights for earlier token losses updated using later blocks, or do they retain only their block-local weights?

Q2: In the no-resampling case, why does resetting the accumulator preserve the claimed sequence-level gradient? A small, exactly enumerable example comparing the implemented estimator with the full objective would resolve this concern. Establishing this correspondence would materially improve my assessment.

6 comments

(1/2)

Thank you for your review and for the clear questions. You are right that the update we submitted did not follow from the objective (W1). The revised PDF fixes the update and reruns the experiments with it.

Q1 (the exact loss). In the submitted code, a token in block k was weighted only by the product of the weights over block k, and later blocks never changed that weight. Equations 7 and 8 in Section 3.4 define a weight over the complete answer. However, the sentence after them and Appendix Q describe the block-local rule, and that is the rule we ran. The revised update follows the equations, and every token of an answer now gets that answer's final sampler weight. For β≠1\beta \neq 1, this weight is also multiplied by (pS/pTα)1−β(p_S/p_T^{\alpha})^{1-\beta} over the whole answer (Appendix Q, Eq. 14). As a result, later blocks now affect earlier tokens through the final weight. Before the last resampling step, they also affect which ancestors survive.

Q2 (resetting the accumulator). Resetting does not preserve the gradient, and our claim that the bias vanishes as 1/N was wrong for the submitted update. Proposition 2 in Appendix Q gives the limit of that update. Without resampling, the fixed point of a tabular student is the block power distribution

∏kpT(yk∣y<k)α∑upT(u∣y<k)α\prod_k \frac{p_T(y_k \mid y_{\lt k})^{\alpha}}{\sum_u p_T(u \mid y_{\lt k})^{\alpha}}

This distribution is the teacher at temperature 1/alpha when the block length is 1, and in general it equals pi_alpha only when a single block covers the whole answer. Proposition 1 proves that the revised update converges to the gradient of Theorem 2, with bias and variance of order 1/N, including when the ESS triggers resampling. The proof is for multinomial resampling, and our checks at finite N use the systematic resampling of our implementation.

The enumerable example you asked for. The checks at finite N run the same rollout and loss code that we use in training, and we also compute the limit of many candidates in closed form. In every check below, the submitted update misses the exact value, and the revised update reaches it, or approaches it as N grows.

CheckExactSubmitted updateRevised update
Appendix A example (decisions in separate blocks, alpha=2, beta=1): mass on the first token78.68%69.23% at every N78.68% in the limit. Gap at N=16, 64, 256, 1024: -0.50, -0.11, -0.00, -0.02 points
Eight random models (6 positions, alpha=4, beta=1), no resampling: cosine with the exact gradient, blocks of length 11-0.41 to 0.961.000
Same, blocks of length 210.12 to 0.961.000
Same, with resampling at ESS < N/2, blocks of length 1 and 210.43 to 0.96 and 0.54 to 0.981.000
Real rollouts, blocks of length 2: relative error at N=16, 64, 256, 102400.577, 0.612, 0.641, 0.709 (limit 0.780)0.138, 0.034, 0.010, 0.008
Fixed point on six models: total variation distance to pi_alpha, blocks of length 1, 2, 3 (temperature 1/alpha gives 0.350)00.497, 0.410, 0.255 (beta=0.7, as submitted)0.0002 at every block length

With the submitted update, the error does not shrink as N grows. It rises toward its limit, which is the problem you pointed out.

(2/2)

Effect on the experiments. We reran all pass@1 and best-of-4 tables with the revised update on all problems. The Qwen3-8B-Base divergence measurement uses the same first 250 MATH500 problems as in the submission, so that its untrained rows carry over.

The revision also sets beta=1, the KL case of the objective, where the submission used beta=0.7. To separate the two changes on Qwen2.5-Math-7B (lambda=0, seed 0), we also trained the revised update at beta=0.7. The table gives pass@1 minus that of the submitted model, with 95% paired intervals (Appendix Q).

Benchmark, temperatureRevised update, beta=0.7Revised update, beta=1 (used throughout)
MATH500, 1/alpha-0.95 [-2.45, +0.50]+0.60 [-0.90, +2.10]
MATH500, 1+0.05 [-2.00, +2.05]+0.60 [-1.40, +2.55]
GSM8K, 1/alpha-0.27 [-0.95, +0.42]-0.08 [-0.89, +0.74]
GSM8K, 1-0.38 [-1.38, +0.61]+0.34 [-0.72, +1.40]

Every interval includes zero. The fix changes what the method provably computes, but on this model we measure no loss in accuracy from it. Between the two revised runs, beta=1 scores 1.55 [0.05, 3.10] points above beta=0.7 on MATH500 at temperature 1/alpha.

With the revised update, the gains over the untrained model hold across three training seeds:

BenchmarkTemperatureUntrainedRevised, mean of 3 seeds (SD)Difference [95% paired interval]
MATH500, 500 problems148.6570.17 (0.55)+21.52 [+18.98, +24.05]
MATH500, 500 problems1/alpha69.1579.40 (0.23)+10.25 [+8.03, +12.52]
GSM8K, 1,319 problems159.8086.58 (0.29)+26.78 [+25.32, +28.26]
GSM8K, 1,319 problems1/alpha76.2991.00 (0.18)+14.71 [+13.25, +16.20]

The revised update also improves the other models we tested. The GRPO baseline on Qwen3-8B-Base starts from the same checkpoint and uses reference answers and the same sampling budget. The public Qwen2.5-Math-7B checkpoint was trained with GRPO and verified rewards. deepseek-math-7b-rl comes from a third model family and was also trained with verified rewards. All of these results use all problems, and the values are pass@1 points with 95% paired intervals.

Model and settingCompared withMATH500GSM8K
Llama-3.1-8B-Instruct, lambda=1untrained, default sampling+4.30 [+2.25, +6.35]+4.30 [+3.11, +5.50]
Llama-3.1-8B-Instruct, lambda=1ordinary on-policy distillation+2.55 [+0.50, +4.55]+2.63 [+1.48, +3.81]
Qwen3-8B-Base, lambda=0, T=1untrained+17.00 [+14.60, +19.40]+13.61 [+12.42, +14.82]
Qwen3-8B-Base, lambda=0, T=1GRPO, seed 0+3.50 [+1.65, +5.35]+3.62 [+2.73, +4.51]
Qwen3-8B-Base, lambda=0, T=1, mean of three OPPD seedsGRPO, seed 0+3.33 [+1.72, +4.95]+3.77 [+2.95, +4.62]
Public GRPO checkpoint of Qwen2.5-Math-7B, T=1that checkpoint+10.65 [+8.15, +13.15]+12.17 [+10.71, +13.63]
Public GRPO checkpoint of Qwen2.5-Math-7B, T=1/alphathat checkpoint+4.10 [+1.95, +6.25]+7.43 [+6.18, +8.68]
deepseek-math-7b-rl, lambda=1untrained+3.20 [+1.20, +5.20]+2.77 [+1.86, +3.68]
Qwen3-14B-Base, lambda=0untrained+17.05 [+14.80, +19.35]+8.04 [+7.05, +9.02]

Thank you again for your review. We are happy to answer any further questions. If our responses address your concerns, we would be grateful if you would consider updating your score.

AuthorsOpenReview ↗

Gentle reminder: summary of how we addressed your concerns

Dear Reviewer A2mD,

As the discussion closes on Nov 18 (AoE), we would like to provide a gentle reminder on the discussion participation. Following is the summary of our rebuttal addressing your concerns.

  • W1 was correct, and it is fixed. Every token now gets the final weight of its complete answer (Eqs. 7–8). Proposition 1 proves that the update converges to the gradient of Theorem 2, also with resampling.
  • The example. The exact target is 78.68%. The submitted update gives 69.23% at every N, and the revised update gives 78.68% in the limit. On eight random models, its cosine with the exact gradient is 1.000, against −0.41 to 0.96 for the submitted update.
  • Results hold. All tables were rerun. Accuracy shows no significant change, and the gains hold over three seeds.

We understand you wrote that establishing this correspondence "would materially improve my assessment". The enumerable example you asked for is now provided in Part 1 of our reply (Appendix Q).
Notably, Reviewer nqfT has confirmed that our responses resolved their major concerns. We are glad to receive support from them. We also hope to have your support for the paper, reflected in an updated score.

Reviewer_nqfTOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

Given a problem, suppose a language model puts probability 0.3 on the correct completion sequence and 0.1 each on 7 in-correct completions. Among all completions, the correct completion receives the highest probability, but the overall mass is highest on the incorrect completions. Based on this intuition, prior work proposed (approximately) sampling from a power distribution (e.g. sample each completion with probability proportional to the square of its original probability), in the hope of sharpening the original distribution towards high probability completions, resulting in higher accuracy. However, such sampling procedures require sampling and scoring multiple candidate completions per prompt, which increases the inference cost. The main idea of this paper is to transfer this cost to training. The proposed method uses such a sampler to generate samples from the power distribution, and trains the model on them. The trained model, when used with regular inexpensive sampling during inference, is shown to outperform baselines such as low-temperature decoding on benchmarks such as GSM8K and MATH500.

Critical strengths

Given power-distribution sampling outperforms regular sampling, the main idea of the paper on shifting its cost to training is very natural. Since it doesn't need any correctness labels, it gives a way to self-improve the model.

I was also surprised to see that training on GSM8K-like problems give significant gains on the harder MATH500 problems, suggesting the proposed method exhibits non-trivial transfer.

Critical weaknesses

Missing comparison with prior work: Huang et al. [1] propose an arguably simpler method of sharpening a model's distribution by sampling several completions from the model, and training it on the high likelihood ones. It is possible that the proposed power-distribution sampling leads to better performance, but an empirical comparison of the two methods (ideally at comparable compute) would be needed to establish that. This work is cited but a direct comparison is missing. (Tomihari & Sato [2] is another missing comparison, but since it will appear in Neurips 2026, it can be treated as concurrent work under ICLR policy.)

Unclear effectiveness over longer horizons: The experiments in this paper use base or instruct models with token-budget at most 4k. Modern reasoning models (e.g. deepseek R1 and similar models) use significantly longer reasoning chains (typically 10-30k) to solve hard problems. The effectiveness of the proposed method (even just power sampling as an inference time intervention) in this regime seems unclear. If the effectiveness of this method can be shown in this long-horizon generation regime with models similar to R1, that would significantly strengthen the paper. If it is not effective, some discussion of the bottlenecks would be helpful.

[1] Huang, Audrey, et al. "Self-improvement in language models: The sharpening mechanism." ICLR 2025.

[2] Tomihari, Akiyoshi, and Issei Sato. "Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillation." NeurIPS 2026.

Questions for authors

Addressing the points raised in the weaknesses section would be helpful.

Some more questions:

  • MATH500 evaluations seem to use a subset of 200 problems. Why not use all 500?

  • One can also imagine using the frozen teacher itself to get the proposal samples. Could you explain the benefit of using the student to propose samples? If possible, an experimental comparison here would also be helpful.

7 comments

(1/4)

Thank you for the review and the helpful suggestions. We answer each point below, and the revised PDF reports the new experiments.

Huang et al. The revised PDF adds a direct comparison at an equal sampling budget (Section 5.1, Appendix B.3). SFT-Sharpening draws N answers per prompt from the frozen model at temperature 1, keeps the one with the highest log p(y|x) (their Eq. 4), and fine-tunes on it. We run it on the same 600 training prompts with N=16, so it draws the same number of answers as OPPD (9,600). We follow the recipe in their Section 5.4 and Table 2 (GSM8k row): LoRA rank 16, learning rate 3e-4 decayed linearly to zero, gradient clipping 0.1, weight decay 0.5, batch size 4, and 30 epochs over the selected answers. Our setup differs from theirs in three ways:

  • Models: Qwen2.5-Math-7B and Qwen3-8B-Base instead of Phi3.5-Mini and Mistral-7B-Instruct.
  • Data: 600 prompts instead of the full training set, so that the budget matches.
  • Checkpoint: the final checkpoint instead of the best one during training.

Below are the results on Qwen2.5-Math-7B, on all 500 MATH500 and all 1,319 GSM8K problems, with four answers each. The Untrained, SFT-Sharpening and OPPD columns are pass@1, and the differences come with paired 95% bootstrap intervals over problems.

Task, temperatureUntrainedSFT-SharpeningOPPDOPPD minus SFT-Sharpening, pass@1Same, best-of-4
MATH500, 148.6568.3570.35+2.00 [-0.50, +4.50]+2.0 [-0.6, +4.6]
MATH500, 1/alpha69.1571.6079.35+7.75 [+5.30, +10.30]+5.4 [+2.8, +8.2]
GSM8K, 159.8077.7186.30+8.59 [+6.92, +10.27]+6.97 [+5.38, +8.64]
GSM8K, 1/alpha76.2978.6690.94+12.28 [+10.48, +14.12]+11.52 [+9.63, +13.42]

On MATH500 at temperature 1 the interval includes zero, and in the other three settings OPPD scores higher. Most of the gain from SFT-Sharpening is what a lower temperature already gives, and it lowers GSM8K best-of-4. The gains over the untrained model at the same temperature are:

Gain over the untrained modelSFT-SharpeningOPPD
pass@1, MATH500, T=1/alpha+2.45 [-0.05, +4.95]+10.20 [+7.90, +12.55]
pass@1, GSM8K, T=1/alpha+2.37 [+0.10, +4.63]+14.65 [+13.14, +16.21]
best-of-4, GSM8K, T=1-3.41 [-5.31, -1.52]+3.56 [+2.20, +4.93]
best-of-4, GSM8K, T=1/alpha-9.10 [-11.15, -7.05]+2.43 [+1.14, +3.79]

SFT-Sharpening fits one answer per prompt, whereas OPPD fits the weighted set of sixteen.

The summed log-probability favors short answers. In this run, 48 of the 600 selected answers are empty. The trained model gives an empty answer for 9.4% of GSM8K samples at temperature 1, compared with 1.35% for the untrained model. Appendix P corrects the same effect in the OPPD target. Their method has no rule against empty answers. To make sure the comparison does not depend on them, we also ran the recipe with answers shorter than two tokens excluded from selection whenever a longer answer was drawn. This version gives an empty answer for 0.08% of GSM8K samples.

Task, temperatureSFT-Sharpening, empty answers excludedOPPDOPPD minus SFT-Sharpening, pass@1Same, best-of-4
MATH500, 171.9070.35-1.55 [-3.90, +0.80]+3.60 [+1.20, +6.00]
MATH500, 1/alpha73.8079.35+5.55 [+3.25, +7.90]+6.20 [+3.80, +8.80]
GSM8K, 185.7586.30+0.55 [-0.70, +1.82]+1.67 [+0.53, +2.88]
GSM8K, 1/alpha86.6090.94+4.34 [+2.98, +5.72]+3.49 [+2.12, +4.93]

At temperature 1 the two methods have about the same pass@1. At temperature 1/alpha, the better temperature for both, OPPD scores higher by 5.55 and 4.34 points. OPPD solves more problems within four attempts in all four settings. On MATH500 at temperature 1/alpha, the best-of-4 of SFT-Sharpening is 3.00 [0.20, 6.00] points below that of the untrained model, and that of OPPD is 3.20 [1.20, 5.40] points above it.

(2/4)

Huang et al. (continued). Huang et al. also select by the mean token log-probability, which does not favor short answers. This is the strongest version of SFT-Sharpening we ran with 16 answers per prompt. None of its answers is empty, and Table 1 of the revised PDF uses it.

Task, temperatureSFT-Sharpening, mean log-probabilityOPPDOPPD minus SFT-Sharpening, pass@1Same, best-of-4
MATH500, 174.1070.35-3.75 [-5.95, -1.55]+2.20 [-0.20, +4.60]
MATH500, 1/alpha76.0079.35+3.35 [+1.10, +5.65]+4.20 [+2.00, +6.60]
GSM8K, 187.4586.30-1.16 [-2.27, -0.02]+1.44 [+0.38, +2.50]
GSM8K, 1/alpha88.0890.94+2.86 [+1.61, +4.15]+2.35 [+0.99, +3.71]

Compared with this version, OPPD:

  • scores lower at temperature 1, by 3.75 points on MATH500 and 1.16 on GSM8K;
  • scores higher at temperature 1/alpha, the better temperature for both methods, by 3.35 and 2.86 points, so with each method at its better temperature OPPD has the higher pass@1 on both benchmarks;
  • solves more problems within four attempts in all four settings, with intervals above zero in three of them;
  • gains 9.00 and 4.64 points from a lower temperature, against 1.90 and 0.63 for SFT-Sharpening.

We attribute the difference in temperature gains to the targets the two methods fit. SFT-Sharpening fits the answer with the highest mean token log-probability among N, and this target concentrates further as N grows. As a result, at temperature 1 it already behaves much as it does at a low temperature. OPPD fits pi_alpha and keeps more of the answers that a lower temperature or a second attempt can use.

All the comparisons above give both methods the same budget of 16 answers per prompt, following your request for comparable compute. We also ran the mean-score version with N=50, which generates 3.1 times as many answers as OPPD:

  • At temperature 1 it scores 77.20 on MATH500 and 89.27 on GSM8K, which is 6.85 [4.75, 8.95] and 2.98 [1.88, 4.08] points above OPPD.
  • At temperature 1/alpha, the better temperature for both methods, OPPD still scores higher with a third of the answers, by +1.65 [-0.25, +3.60] on MATH500 and +1.33 [+0.19, +2.46] on GSM8K. Its best-of-4 is also higher, by +1.40 [-0.40, +3.40] and +1.44 [+0.30, +2.65] points.
  • Lowering the temperature adds 0.50 and 0.34 points to this model, compared with 9.00 and 4.64 for OPPD. This fits the view that a larger N concentrates SFT-Sharpening further (Appendix B.3 of the revised PDF).

We repeated the comparison on Qwen3-8B-Base with the same recipe, the same mean log-probability selection and the same draw of 600 prompts. OPPD scores higher in all four settings, in both pass@1 and best-of-4:

Task, temperatureSFT-Sharpening, mean log-probabilityOPPDOPPD minus SFT-Sharpening, pass@1Same, best-of-4
MATH500, 175.5577.90+2.35 [+0.35, +4.40]+3.20 [+0.80, +5.80]
MATH500, 1/alpha75.9079.80+3.90 [+1.30, +6.50]+5.00 [+2.40, +7.80]
GSM8K, 193.0193.14+0.13 [-0.70, +0.97]+0.38 [-0.53, +1.29]
GSM8K, 1/alpha92.9793.71+0.74 [-0.21, +1.71]+0.83 [-0.15, +1.82]

On this model OPPD is also ahead at temperature 1. At temperature 1/alpha, SFT-Sharpening scores below the untrained model on MATH500 (75.90 against 78.45), while OPPD scores above it (79.80).

Before adopting their recipe, we ran their selection rule inside our own training loop (one pass, full fine-tuning, the OPPD learning rate). That version is weaker. At temperature 1 it scores 53.25 on MATH500 and 56.65 on GSM8K, with 25.9% empty answers on GSM8K (Appendix B.3).

In whole-answer KL to pi_alpha (see our reply to AJP5, Q3), OPPD is closer than that version by 1.96 [1.37, 2.52] nats under one reference and 4.95 [4.27, 5.65] under the other. The versions trained with the published recipe move away from pi_alpha. Under a second pair of reference sets, each of them is farther from pi_alpha than the untrained model, and OPPD is closer than each by 9.79 to 13.23 nats under both references:

Version of SFT-SharpeningFirst referenceSecond reference
Published recipe10.45 [8.32, 12.91]13.23 [10.76, 16.02]
Empty answers excluded9.79 [7.85, 11.86]9.85 [8.07, 11.83]
Mean log-probability10.70 [8.73, 12.84]13.16 [11.18, 15.21]

Fitting one selected answer per prompt lowers the probability of the answers that pi_alpha draws, which is consistent with the lower best-of-4 of SFT-Sharpening.

(3/4)

Tomihari and Sato. We agree that this work is concurrent, and Section 6 discusses it. Their power self-distillation trains a model on samples that a power sampler produced beforehand. In OPPD the student itself is the sampler's proposal, so it trains on its own answers. One teacher pass per block serves both the sampler and the update, and the objective is the Renyi family rather than supervised learning on stored samples.

Long horizons. We tested this regime on DeepSeek-R1-Distill-Qwen-7B, with up to 16,384 generated tokens on AIME 2024 and 2025 (60 problems, four answers each). On these problems its answers have a median length of 11,982 tokens (Appendix B.18). At its recommended temperature of 0.6, the untrained model scores 45.00 pass@1 and 68.33 best-of-4. At 0.25 it scores 42.08, so a lower temperature does not help this model.

You asked whether power sampling itself works in this regime as an inference-time method. At the paper's alpha=4 with 16 candidates, it scores 46.67, the highest of the decoding methods we measured on this model. This is +4.58 [-5.00, +14.58] over plain sampling at its proposal temperature of 0.25, and +1.67 [-7.50, +10.83] over plain sampling at 0.6. With 60 problems, neither interval excludes zero.

The same run also shows what limits power sampling at this length. Each resampling step replaces low-weight candidates with copies of high-weight ones. Over 16k tokens the sampler resamples 102.6 times per problem, so the 16 final answers share a single ancestor for most of their length:

Position in the answer641,0242,0484,0968,192
Problems whose answers reach this position6060594633
Distinct prefixes among the 16 final answers1.001.001.251.131.21
Share of these problems where all 16 share one prefix100%100%94.9%95.7%87.9%

Appendix B.17 shows that most of the sampler's gain comes from discarding candidates while their continuations still differ. At this horizon, the sampler works on one lineage for most of each answer, which is known as path degeneracy in sequential Monte Carlo with many resampling steps. Path degeneracy limits inference-time power sampling at long horizons, and it is one reason to amortize the sampler. OPPD runs the sampler only during training, over 4,096 tokens, and the trained model generates without it.

We then trained OPPD on this model at alpha=4 and lambda=1. The rule in Appendix B.10 picks lambda=1 when a lower temperature does not help. Training uses GSM8K questions with at most 4,096 tokens per answer. The training answers average 736 tokens, and the sampler resamples 7.62 times per prompt during training. Plain sampling and OPPD are sampled at temperature 0.6, with eight answers per problem from two evaluation seeds.

AIME 2024 and 2025Candidates per answerpass@1Answers at the 16,384-token limitMedian length
Plain sampling, untrained145.2134.6%11,598
Power sampling, alpha=4, proposal at temperature 0.251646.6739.7% of candidates11,250 over candidates
OPPD, trained147.0829.6%10,436

OPPD minus plain sampling is +1.88 [-2.50, +6.25], and it is positive under each evaluation seed (+1.25 and +2.50). OPPD minus power sampling is +0.42 [-9.17, +9.79]. This means the trained model is on par with inference-time power sampling while generating one answer instead of sixteen candidates, and its answers are shorter.

200 of 500. Our only reason for using a fixed subset of the first 200 problems was to save evaluation time. All pass@1 and best-of-4 tables now use all 500 MATH500 problems and all 1,319 GSM8K problems, with paired intervals. On Qwen2.5-Math-7B, the revised model at temperature 1/alpha scores 80.25 on the first 200 problems and 79.35 on all 500. On Qwen3-8B-Base, the untrained model at temperature 1/alpha scores 79.62 on the first 200 and 78.45 on all 500. For both models, the first 200 problems are about one point easier than the full set.

Reviewer_AJP5OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes On-Policy Power Distillation (OPPD), which aims to transfer the benefits of sequence-level power sampling into a language model that generates a single answer. It combines a student proposal, sequential Monte Carlo sampling using frozen-teacher probabilities, importance-weighted likelihood training motivated by a Rényi-divergence gradient, and an optional token-level KL anchor. Training uses problem statements without reference answers or rewards for correctness. Experiments evaluate mathematical reasoning across several model families, repeat the main setting across three training seeds, and test transfer to HumanEval. In a matched 250-problem MATH500 comparison, the trained model's single-generation accuracy approaches that of the untrained model's 16-candidate sampler.

Critical strengths
  1. Meaningful empirical gains beyond changing decoding temperature. At the same decoding temperature of 0.25, Qwen2.5-Math-7B with OPPD (lambda=0) improves from 70.13 to 78.50 on MATH500 and from 76.13 to 91.13 on GSM8K, with substantial gains retained across three training seeds (Tables 1 and 3). In the matched 250-problem comparison, one trained-model generation scores 78.4 versus 79.2 for the untrained model's 16-candidate sampler (Table 9). This supports the practical value of moving some inference-time benefit into training without reference answers.

  2. Validation extends beyond the main model and training task. The experiments cover multiple model families and include math-to-code transfer. In particular, DeepSeekMath-RL improves on MATH500 even though reducing its decoding temperature gives no improvement (Table 6). HumanEval improvements have positive paired 95% intervals in four of five OPPD checkpoint comparisons (Table 2). This makes the practical result more persuasive than a single favorable model/benchmark example.

Critical weaknesses

My main concern is whether the described training rule matches the objective used to justify it. Theorem 2 and Appendix H derive a gradient using weights computed over complete answers. Section 3.4 instead resets the training-weight sum at every block boundary, and Appendix Q retains the weights recorded for earlier blocks. When no resampling occurs, later parts of an answer do not affect those earlier weights.

An adaptation of Appendix A’s example shows why this matters. Set teacher equal to student, α=2 and β=1, and place the two decisions in separate blocks using deterministic padding. The complete-answer target assigns 78.7% probability to answers beginning with token a. In the large-particle limit, the stored first-block weights instead place 69.2% of the training weight on that branch. Every candidate has the same first-block weight, and the continuation probabilities never enter those weights. ESS remains above the resampling threshold, with ESS/N approaching 0.96. The gap persists as the number of candidates grows, contradicting the claim in Section 3.4 that the gradient estimator’s bias vanishes as 1/N.

Appendix P introduces a further change by adjusting the weights of answers that have already ended. This changes their weight relative to unfinished answers. The paper does not establish that this adjustment preserves the target in Equation 1 or specify the resulting distribution.

The accuracy gains are useful evidence for the method. My concern is the claimed connection to the sequence-level power distribution. The authors need to resolve the discrepancy between the derived gradient and the described update, and clarify the target after the stopping adjustment.

Questions for authors
  1. How do later parts of an answer affect the training weights on earlier blocks when resampling does not occur? Please resolve the counterexample above by comparing the limiting gradient of the described blockwise update with the exact Rényi gradient. Do the checks in Appendix O test this full procedure across block boundaries, or the complete-answer gradient identity separately?

  2. What distribution does the sampler target after the adjustment in Appendix P, and which reported experiments use it? Can you establish whether the adjustment preserves Equation 1 on a small variable-length model with EOS, or specify the modified target?

  3. How is the ESS measurement in Section 5.3 computed and aggregated across blocks? Equation 4 concerns the Rényi divergence of order 2 between the complete-answer target and the proposal. Appendix M reports a per-block quantity, and the proposal is the student at temperature 1/α. What justifies interpreting this measurement as whole-answer closeness of the untempered student to the target?

  4. Can you provide paired uncertainty for the Qwen3-8B best-of-4 differences in Table 14 using the existing per-problem outcomes? Separately, what supports the diversity interpretation in Appendices B.6 and E, given that reweighting answers the model already produces can improve both pass@1 and best-of-4? Please distinguish success within four attempts from claims about answer diversity or broader reasoning coverage.

7 comments

(1/3)

Thank you for the detailed review and for the counterexample, which is correct. When we run it through our training code, we get the numbers you describe: target 78.68%, submitted update 69.23%, ESS/N 1.00 and 0.96.

Q1. In the submitted update, when no resampling happened, later parts of an answer did not change the weights of earlier parts. The revised update uses one weight for the complete answer. Proposition 1 in Appendix Q proves that its limit is the gradient of Theorem 2 for every beta > 0. We compared the limits with the exact Renyi gradient:

  • Revised update, beta in {0.25, 0.5, 0.7, 1, 2, 5}: largest relative error 1.1e-6.
  • Revised update, beta=1, every block length, with no resampling, with resampling when ESS < N/2, and with resampling after every block: relative error at most 6.0e-7.
  • Submitted update, no resampling: cosine with the exact gradient of -0.41 to 0.96 at block length 1, and 0.12 to 0.96 at block length 2.

The checks in Appendix O of the first version tested the complete-answer identity and the loss separately. They never ran the full procedure across block boundaries, so they could not catch this error. We rewrote Appendix O. The new checks run the real rollout and loss across block boundaries, including resampling, and compare the result with the exact gradient.

Q2. The adjustment does not preserve Eq. 1. Its target is Eq. 13 in Appendix P, which is pi_alpha multiplied by a function of answer length only. Within each length class the relative probabilities are the same as in Eq. 1, but the total mass of each length class changes. The length function is built from the mean log incremental weight of the unfinished candidates, so it depends on the proposal. With the student as proposal it changes during training, but the relative probabilities within a length class stay the same. All reported experiments use this adjustment, and Section 3.5 now says so. At beta=1, which all revised runs use, Proposition 1 holds with this target in place of pi_alpha.

We checked the closed form on enumerable models with EOS:

  • On eight models, the limit matches it to a TV distance below 4e-7.
  • Real rollouts approach it, with TV 0.0325, 0.0132, 0.0062 and 0.0025 at N=16, 64, 256 and 4096.
  • The mean number of tokens before EOS is 4.67 under the adjusted target, 2.37 under pi_alpha and 4.56 under the proposal.

(2/3)

Q3. We computed the monitored ESS in each block from the cumulative log weights since the last resampling, and then averaged it over blocks and problems. The proposal in this measurement was the student at temperature 1/alpha. Reading this number as the whole-answer closeness of the untempered student was not justified, and we withdraw that reading. Section 2 now states the scope of Eq. 4, and Section 5.3 replaces the ESS with a direct measurement over complete answers, without tempering.

The measurement uses an identity that holds for any two models a and b:

KL(πα∥pa)−KL(πα∥pb)=Eπα[log⁡pb(y)−log⁡pa(y)]\mathrm{KL}(\pi_\alpha \Vert p_a) - \mathrm{KL}(\pi_\alpha \Vert p_b) = \mathbb E_{\pi_\alpha}[\log p_b(y) - \log p_a(y)]

The gain in Eπα[log⁡p(y)]\mathbb E_{\pi_\alpha}[\log p(y)] from training is thus equal to the drop in the whole-answer KL to pi_alpha. We estimate it in three steps:

  1. On 100 held-out MATH500 problems, we draw samples from the training target (pi_alpha with the length adjustment from Q2) with the SMC sampler at N=64. This target has the same relative probabilities as pi_alpha within each answer length, and below we write pi_alpha for it.
  2. With finite N the sampler leans toward its proposal, and the length adjustment also depends on the proposal. We therefore draw two reference sets, one with the untrained model as proposal and one with the trained model, and report both.
  3. We score each complete answer under every model.

Each value is the drop in KL(πα∥p)\mathrm{KL}(\pi_\alpha \Vert p) relative to the untrained model at temperature 1, in nats per answer, with 95% intervals over problems.

ModelReference drawn with untrained proposalReference drawn with trained proposal
Revised update, seeds 0, 1, 23.66 [3.03, 4.30], 3.36 [2.70, 4.00], 3.75 [3.15, 4.37]6.72 [6.01, 7.45], 6.33 [5.66, 7.02], 6.18 [5.51, 6.88]
Submitted update4.36 [3.75, 4.99]6.21 [5.54, 6.93]
Uniform weights, no resampling1.40 [0.04, 2.67]4.86 [3.42, 6.18]
SFT-Sharpening (Huang et al.), our training loop1.70 [1.54, 1.87]1.77 [1.62, 1.94]
Ordinary on-policy distillationnot measured4.25 [3.77, 4.76]
Untrained, temperature 1/alpha6.26 [5.46, 7.09]4.90 [3.61, 6.09]

Three results hold under both references:

  • Training lowers the whole-answer KL to pi_alpha for all three seeds.
  • The trained model is closer to pi_alpha than SFT-Sharpening, by 1.96 [1.37, 2.52] and 4.95 [4.27, 5.65] nats.
  • It is also closer than the uniform-weights run, by 2.26 [1.21, 3.45] and 1.86 [0.77, 3.14] nats. That run trains on answers from the same proposal but drops the sampler's weights and resampling, so this gain comes from the sampler. It is also less accurate on all problems: 5.65 [3.15, 8.20] points below the trained model on MATH500 and 24.01 [21.89, 26.14] on GSM8K at temperature 1, and 7.95 [5.60, 10.40] and 26.33 [23.96, 28.70] at 1/alpha.

Under a second pair of reference sets, drawn in the same way, the three versions of SFT-Sharpening trained with its published recipe (see our reply to nqfT) are each farther from pi_alpha than the untrained model. The trained model is closer to pi_alpha than each of them by 9.79 to 13.23 nats under both references, and every interval is above 7.8.

To check the bias toward the proposal, we raised N on the first 50 problems, to 256 for the first reference and to 128 for the second:

  • In the second reference, no entry moves by more than 0.12 nats.
  • In the first reference, every entry except SFT-Sharpening falls by 0.82 to 1.07 nats, so the differences among these models change by at most 0.25.
  • Under all four references, the trained model stays closer to pi_alpha than both the untrained model and SFT-Sharpening. Its margin over SFT-Sharpening is at least 1.57 [0.26, 2.71] nats.

The comparison of the trained model with the untrained model at temperature 1/alpha depends on the reference. The difference is -2.22 [-3.72, -0.98] under the first reference at N=256 and +0.48 [-0.76, +1.87] under the second at N=128. More candidates do not resolve this difference, and answer length does not explain it, so we make no claim about which of the two models is closer to pi_alpha. The same holds for ordinary on-policy distillation and for the submitted update. Appendix B.4 gives the full tables for both reference sets.

(3/3)

Q4. Table 14 of the submission is Table 19 in the revised PDF. It now uses all problems and the revised update. From the per-problem outcomes (seed 0), the paired 95% intervals for the best-of-4 differences are:

OPPD minusMATH500, T=1GSM8K, T=1MATH500, T=1/alphaGSM8K, T=1/alpha
Untrained+4.40 [+2.00, +7.00]-0.15 [-1.06, +0.76]-0.60 [-2.80, +1.60]-1.21 [-2.12, -0.30]
Ordinary on-policy distillation+1.00 [-1.00, +3.20]-0.15 [-0.99, +0.68]+0.00 [-1.80, +1.80]+0.53 [-0.53, +1.59]

At temperature 1, OPPD solves more MATH500 problems within four attempts than the untrained model. On GSM8K the untrained model already solves 96.6% of problems within four attempts. At temperature 1/alpha, every trained model in Table 19 has a lower GSM8K best-of-4 than the untrained model, and OPPD is not below ordinary on-policy distillation.

Against GRPO (Table 18), OPPD has the higher pass@1. The paired intervals at temperature 1 are:

MATH500GSM8KAIME
OPPD seed 0 minus GRPO+3.50 [+1.65, +5.35]+3.62 [+2.73, +4.51]+2.08 [-1.67, +6.67]
OPPD seed 1 minus GRPO+2.95 [+1.00, +4.90]+3.64 [+2.69, +4.61]not evaluated
OPPD seed 2 minus GRPO+3.55 [+1.65, +5.45]+4.06 [+3.17, +4.97]not evaluated
OPPD seed 0 minus untrained+17.00 [+14.60, +19.40]+13.61 [+12.42, +14.82]+3.33 [-1.67, +8.75]

The MATH500 and GSM8K margins exclude zero for every seed. AIME has only 60 problems, and its intervals include zero.

On diversity, you are right that reweighting the answers that the model already produces can raise both pass@1 and best-of-4. We withdraw the sentence that said selection cannot give such a gain. The revised paper reports success within K attempts (K=4 and 16) separately from the number of distinct final answers per problem, each with a paired interval (Appendix B.5). It claims no broader reasoning coverage than these measurements show. On Qwen2.5-Math-7B (all problems, mean of three seeds), the trained model at temperature 1 compares with the untrained model at temperature 1/alpha as follows:

pass@1best-of-4Distinct answers per problem
MATH500+1.02 [-1.12, +3.15]+3.53 [+1.40, +5.73]+0.097 [+0.034, +0.161]
GSM8K+10.29 [+8.82, +11.78]+3.69 [+2.50, +4.93]-0.017 [-0.051, +0.016]

Training keeps more distinct answers per problem than lowering the temperature on MATH500, and about as many on GSM8K, while the trained model solves more problems within four attempts on both. The extra distinct answers on MATH500 are wrong ones (distinct wrong answers +0.125 [+0.063, +0.187]). Against the untrained model at the same temperature, the trained model gives fewer distinct answers (-0.439 [-0.520, -0.357] at temperature 1 on MATH500), as expected when the distribution concentrates.

With sixteen answers per problem on MATH500, success within sixteen attempts stays about the same after training:

  • Qwen2.5-Math-7B (seed 0): +0.40 [-1.40, +2.20] at temperature 1 and +1.00 [-1.00, +3.00] at 1/alpha.
  • Qwen3-8B-Base: +1.00 [-0.60, +2.60] and +0.20 [-1.80, +2.20]. In the same setting, pass@1 rises by 16.52 [15.03, 18.02] and 1.60 [0.15, 3.06] points.

Thank you again for your review. We are happy to answer any further questions. If our responses address your concerns, we would be grateful if you would consider updating your score.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of our revisions and answers to common questions

We thank all three reviewers very much for their careful reviews and constructive suggestions.

We have answered every question and concern raised in the reviews, clarified each point the reviewers found ambiguous, and revised the paper wherever the reviewers asked. Below is a short overview. Our reply to each reviewer gives the details.

Contributions of our paper.

OPPD trains a model once so that a single ordinary generation captures the benefit of sampling from the sequence-level power distribution, without reference answers or a reward model. The reviewers found the idea natural and a way for a model to improve itself without correctness labels (nqfT), the gains substantial and practically relevant beyond lowering the temperature (A2mD, AJP5), the validation broad across model families and from math to code (AJP5), and the transfer from GSM8K training to MATH500 notable (nqfT). A2mD identified the combination of sampling and training weights as the most consequential methodological contribution.

Overview of revisions and new experiments

  • Fixed the training update to use complete-answer weights, as the objective requires, and proved its limit for every beta, with and without resampling (Section 3.4, Appendix Q).
  • Rewrote the exact correctness checks to run the real rollout and loss across block boundaries, and reproduced and resolved AJP5's counterexample (Appendix O).
  • Stated the target after the stopping adjustment in closed form and checked it on enumerable models (Section 3.5, Appendix P).
  • Replaced the ESS-based reading with a direct measurement of whole-answer KL under two reference sets (Section 5.3, Appendix B.4).
  • Reran all pass@1 and best-of-4 tables on all 500 MATH500 and all 1,319 GSM8K problems, with paired 95% intervals; the main results also hold over three training seeds (Appendices B.2, B.5 and L).
  • Added a direct comparison with SFT-Sharpening (Huang et al.) at an equal sampling budget, in three versions and on two models (Section 5.1, Appendix B.3).
  • Added the long-horizon regime: DeepSeek-R1-Distill-Qwen-7B at 16,384 tokens, for both inference-time power sampling and OPPD (Appendix B.18).
  • Added the frozen teacher as proposal, with a direct comparison to the student proposal (Section 3.2, Appendix B.16).
  • Discussed Tomihari and Sato as concurrent work (Section 6).

Common question 1: Does the training update follow the sequence-level objective? (A2mD, AJP5)

We thank both reviewers for finding this.

  • The problem: the submitted update restarted the training weight at each block boundary, so each token was weighted by its own block only, not by the complete answer. This does not follow the objective.
  • The fix: the revised update gives every token the final weight of its complete answer, as Equations 7 and 8 define.
  • Proof and checks: Proposition 1 proves that the revised update converges to the gradient of the objective. On small models where every answer can be listed, it matches the exact gradient (cosine 1.000), and the submitted update does not.
  • Accuracy: on Qwen2.5-Math-7B, the revised model is on par with the submitted one (no pass@1 difference is statistically significant), and its gains over the untrained model hold across three training seeds (for example, +21.52 points on MATH500 at temperature 1).

Common question 2: How reliable is the evaluation? (AJP5, nqfT)

All pass@1 and best-of-4 tables now use the full benchmarks, and every difference comes with a paired 95% interval over problems. As AJP5 asked, we report success within K attempts separately from the number of distinct answers, and we replaced the ESS reading with a direct measurement over complete answers. Training lowers the whole-answer KL to the target under both reference sets, for all three seeds.

Other questions, answered in the individual replies (nqfT)

  • SFT-Sharpening: at an equal sampling budget, with each method at its better temperature, OPPD has the higher pass@1 on both benchmarks, and it solves more problems within four attempts in every setting.
  • Long horizons: at 16,384 tokens, the trained model is on par with inference-time power sampling (47.08 against 46.67 on AIME) while generating one answer instead of sixteen candidates, and its answers are shorter.
  • Frozen teacher as proposal: the student proposal scores higher in all four settings, by 2.08 to 6.35 points on Qwen2.5-Math-7B.

We thank the reviewers again for helping us improve the paper. We are happy to answer any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.