- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes OPPD, which trains a student to approximate a teacher's sequence-level power distribution without correctness labels. It combines on-policy sequential Monte Carlo sampling with importance-weighted training derived from a Rényi-divergence objective and an optional token-level anchor. The experiments report substantial improvements on mathematical reasoning and reduced inference-time sampling requirements.
Critical strengths
S1: The reported improvements are substantial and practically relevant. Table 1 shows gains over both the original model and lower-temperature decoding, supporting the claim that training can absorb benefits that token-level temperature changes alone do not provide.
S2: The paper distinguishes sequence-level sharpening from local temperature scaling and develops an on-policy sampling and training procedure around that distinction. Since amortizing power sampling through distillation already appears in the prior work discussed in Section 6, the specific combination of sampling and training weights is the most consequential methodological contribution.
Critical weaknesses
W1: The connection between the stated sequence-level objective and the implemented blockwise loss is unclear. Section 3.4 derives importance weights for complete sequences, but then states that the training-weight accumulator restarts at each block boundary. It subsequently describes the resulting gradient estimate as consistent, with bias of order .
If earlier token losses retain only their block-local weights, later suffix likelihoods do not directly reweight those decisions. This can differ from the gradient of the stated sequence-level objective, including when no resampling occurs. The main text does not explain how these descriptions are reconciled. This concerns the central justification of the method and is the main reason for my current recommendation.
Questions for authors
Q1: Please state the exact loss used across multiple blocks. Are weights for earlier token losses updated using later blocks, or do they retain only their block-local weights?
Q2: In the no-resampling case, why does resetting the accumulator preserve the claimed sequence-level gradient? A small, exactly enumerable example comparing the implemented estimator with the full objective would resolve this concern. Establishing this correspondence would materially improve my assessment.
6 comments
(2/2)
Effect on the experiments. We reran all pass@1 and best-of-4 tables with the revised update on all problems. The Qwen3-8B-Base divergence measurement uses the same first 250 MATH500 problems as in the submission, so that its untrained rows carry over.
The revision also sets beta=1, the KL case of the objective, where the submission used beta=0.7. To separate the two changes on Qwen2.5-Math-7B (lambda=0, seed 0), we also trained the revised update at beta=0.7. The table gives pass@1 minus that of the submitted model, with 95% paired intervals (Appendix Q).
| Benchmark, temperature | Revised update, beta=0.7 | Revised update, beta=1 (used throughout) |
|---|---|---|
| MATH500, 1/alpha | -0.95 [-2.45, +0.50] | +0.60 [-0.90, +2.10] |
| MATH500, 1 | +0.05 [-2.00, +2.05] | +0.60 [-1.40, +2.55] |
| GSM8K, 1/alpha | -0.27 [-0.95, +0.42] | -0.08 [-0.89, +0.74] |
| GSM8K, 1 | -0.38 [-1.38, +0.61] | +0.34 [-0.72, +1.40] |
Every interval includes zero. The fix changes what the method provably computes, but on this model we measure no loss in accuracy from it. Between the two revised runs, beta=1 scores 1.55 [0.05, 3.10] points above beta=0.7 on MATH500 at temperature 1/alpha.
With the revised update, the gains over the untrained model hold across three training seeds:
| Benchmark | Temperature | Untrained | Revised, mean of 3 seeds (SD) | Difference [95% paired interval] |
|---|---|---|---|---|
| MATH500, 500 problems | 1 | 48.65 | 70.17 (0.55) | +21.52 [+18.98, +24.05] |
| MATH500, 500 problems | 1/alpha | 69.15 | 79.40 (0.23) | +10.25 [+8.03, +12.52] |
| GSM8K, 1,319 problems | 1 | 59.80 | 86.58 (0.29) | +26.78 [+25.32, +28.26] |
| GSM8K, 1,319 problems | 1/alpha | 76.29 | 91.00 (0.18) | +14.71 [+13.25, +16.20] |
The revised update also improves the other models we tested. The GRPO baseline on Qwen3-8B-Base starts from the same checkpoint and uses reference answers and the same sampling budget. The public Qwen2.5-Math-7B checkpoint was trained with GRPO and verified rewards. deepseek-math-7b-rl comes from a third model family and was also trained with verified rewards. All of these results use all problems, and the values are pass@1 points with 95% paired intervals.
| Model and setting | Compared with | MATH500 | GSM8K |
|---|---|---|---|
| Llama-3.1-8B-Instruct, lambda=1 | untrained, default sampling | +4.30 [+2.25, +6.35] | +4.30 [+3.11, +5.50] |
| Llama-3.1-8B-Instruct, lambda=1 | ordinary on-policy distillation | +2.55 [+0.50, +4.55] | +2.63 [+1.48, +3.81] |
| Qwen3-8B-Base, lambda=0, T=1 | untrained | +17.00 [+14.60, +19.40] | +13.61 [+12.42, +14.82] |
| Qwen3-8B-Base, lambda=0, T=1 | GRPO, seed 0 | +3.50 [+1.65, +5.35] | +3.62 [+2.73, +4.51] |
| Qwen3-8B-Base, lambda=0, T=1, mean of three OPPD seeds | GRPO, seed 0 | +3.33 [+1.72, +4.95] | +3.77 [+2.95, +4.62] |
| Public GRPO checkpoint of Qwen2.5-Math-7B, T=1 | that checkpoint | +10.65 [+8.15, +13.15] | +12.17 [+10.71, +13.63] |
| Public GRPO checkpoint of Qwen2.5-Math-7B, T=1/alpha | that checkpoint | +4.10 [+1.95, +6.25] | +7.43 [+6.18, +8.68] |
| deepseek-math-7b-rl, lambda=1 | untrained | +3.20 [+1.20, +5.20] | +2.77 [+1.86, +3.68] |
| Qwen3-14B-Base, lambda=0 | untrained | +17.05 [+14.80, +19.35] | +8.04 [+7.05, +9.02] |
Thank you again for your review. We are happy to answer any further questions. If our responses address your concerns, we would be grateful if you would consider updating your score.
Gentle reminder: summary of how we addressed your concerns
Dear Reviewer A2mD,
As the discussion closes on Nov 18 (AoE), we would like to provide a gentle reminder on the discussion participation. Following is the summary of our rebuttal addressing your concerns.
- W1 was correct, and it is fixed. Every token now gets the final weight of its complete answer (Eqs. 7–8). Proposition 1 proves that the update converges to the gradient of Theorem 2, also with resampling.
- The example. The exact target is 78.68%. The submitted update gives 69.23% at every N, and the revised update gives 78.68% in the limit. On eight random models, its cosine with the exact gradient is 1.000, against −0.41 to 0.96 for the submitted update.
- Results hold. All tables were rerun. Accuracy shows no significant change, and the gains hold over three seeds.
We understand you wrote that establishing this correspondence "would materially improve my assessment". The enumerable example you asked for is now provided in Part 1 of our reply (Appendix Q).
Notably, Reviewer nqfT has confirmed that our responses resolved their major concerns. We are glad to receive support from them. We also hope to have your support for the paper, reflected in an updated score.
(1/2)
Thank you for your review and for the clear questions. You are right that the update we submitted did not follow from the objective (W1). The revised PDF fixes the update and reruns the experiments with it.
Q1 (the exact loss). In the submitted code, a token in block k was weighted only by the product of the weights over block k, and later blocks never changed that weight. Equations 7 and 8 in Section 3.4 define a weight over the complete answer. However, the sentence after them and Appendix Q describe the block-local rule, and that is the rule we ran. The revised update follows the equations, and every token of an answer now gets that answer's final sampler weight. For β=1, this weight is also multiplied by (pS/pTα)1−β over the whole answer (Appendix Q, Eq. 14). As a result, later blocks now affect earlier tokens through the final weight. Before the last resampling step, they also affect which ancestors survive.
Q2 (resetting the accumulator). Resetting does not preserve the gradient, and our claim that the bias vanishes as 1/N was wrong for the submitted update. Proposition 2 in Appendix Q gives the limit of that update. Without resampling, the fixed point of a tabular student is the block power distribution
∏k∑upT(u∣y<k)αpT(yk∣y<k)α
This distribution is the teacher at temperature 1/alpha when the block length is 1, and in general it equals pi_alpha only when a single block covers the whole answer. Proposition 1 proves that the revised update converges to the gradient of Theorem 2, with bias and variance of order 1/N, including when the ESS triggers resampling. The proof is for multinomial resampling, and our checks at finite N use the systematic resampling of our implementation.
The enumerable example you asked for. The checks at finite N run the same rollout and loss code that we use in training, and we also compute the limit of many candidates in closed form. In every check below, the submitted update misses the exact value, and the revised update reaches it, or approaches it as N grows.
With the submitted update, the error does not shrink as N grows. It rises toward its limit, which is the problem you pointed out.