- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces DPR, an LLM RL algorithm that updates the top-entropy tokens toward a self-consistency-based reward while anchoring the remaining tokens toward the reference model. Its contribution is a controlled evaluation on implementation choices of related algorithms. Experiments show the effect of token selection choice or ratio is nuanced across different setups.
Critical strengths
A strength is transparent and multi-seed evaluation. The paper runs 13 seed per setting across two models and two datasets, and reports both positive and negative results.
Critical weaknesses
-
Lack of motivation for studying token selection method. It is not standard practice in LLM RL to limit effective updates to only 5% of tokens and this brings no training compute benefit. The paper's own results in Appendix I and Q show that standard 100% token updates and even token selection with 10% updates achieve higher accuracy than the studied 5% update. Comparing which 5% of tokens to update is therefore poorly motivated.
-
No actionable insight from the evaluation. The paper provide little information for the community even if studying 5% token selection is justified. It shows that entropy selection beats random selection on Qwen2.5-Math-7B but loses on DeepSeekMath-7B-Base, and explicitly admits (Section 5.5) that it cannot explain why this mismatch happens.
Questions for authors
- If updating with more tokens achieves better accuracy with same FLOPs, why would we select 5% tokens to update on?
- Is there any explanation for why high entropy selection may outperform or underperform random selection depending on concrete setups?
4 comments
Official Response to Reviewer iC3Q (Part 2/2)
About the Model-Dependent Results
Is there any explanation for why high entropy selection may outperform or underperform random selection depending on concrete setups?
The evidence separates pre-RL conditions, trained endpoints and output/scorer interactions. All three update-81 settings retain their full 13 paired seeds under gold-v2:
| Setting | Decision | Random | Difference; paired 95% CI (pp) |
|---|---|---|---|
| Qwen anchor | 70.95% | 68.18% | +2.77 [1.83, 3.71] |
| Qwen training-data shift | 72.08% | 68.08% | +4.00 [3.14, 4.86] |
| DeepSeek model shift | 0.00% | 2.38% | -2.38 [-2.65, -2.12] |
All data-shift contrasts are positive and all DeepSeek contrasts negative. The Qwen base scores 68.00% before RL, whereas DeepSeek scores 2.80% under this prompt and decoding. The initial measurements establish the pre-RL operating conditions, and the model-dependent endpoint ordering persists under gold-v2 scoring. The Qwen historical-control fresh-Random contrast is also positive (+2.03 pp).
The same 13 Qwen anchor pairs also favor Decision on five additional contest sets: 24.25% versus 16.56% across 170 questions, a +7.69 pp difference [6.93, 8.45]. The full results appear in tVBL Part 4.
In the seed-2000 DeepSeek Decision checkpoint, 499/500 outputs contain an explicit formatting example ending in boxed 123. Synthetic controls show that appending this example can override an earlier correct prediction during extraction. The 499 detected outputs contain no other non-123 balanced box. These measurements identify an output/extractor interaction alongside the observed training behavior. The reference-entropy ranking measures uncertainty; the consensus reward and the fixed extraction rules determine how generated answers enter the learning and evaluation pipeline.
About the Practical Findings
No actionable insight from the evaluation.
-
Match the PG denominator. For a fixed batch selecting of tokens, Eq. (6), , gives the pure-PG denominator effect. The selected-to-total factor is . This clarifies the direction stated in Section 5.2, consistent with Eq. (6) and the implemented loss; KL and AdamW transformations are treated separately.
-
Use paired training runs. Appendix P.4's reproduced historical example gives pooled-problem p=0.043 but paired-run p=0.211. All completed 13-seed comparisons use paired-run inference and report run-level uncertainty, with questions and checkpoints aggregated within each training run.
-
Test semantic grading and irrelevant formatting. Bare gold
70 \sqrt{2}could become70, and a boxed formatting example can override an earlier answer. The all-endpoint audit localizes all 3,524 upward corrections to the 108 preidentified gold-check failures, while retaining 96 downward corrections. The gold checks and synthetic formatting controls quantify distinct parts of the scoring pipeline. -
Check the selector contrast against fresh masks. All 13 fresh-Random runs are complete: retained Decision is +2.03 pp higher, and every leave-one-seed-out mean stays positive (+1.83 to +2.22 pp). The corresponding legacy-routing contrast is +2.77 pp. tVBL Part 1 specifies the historical-control design and reports the separate current-code Decision bridge, preserving the checkpoint identities for each comparison.
-
Control perturbation strength and privileged tokens. The submitted v3 intervention records 22.99 nats of forced-replacement surprisal at low-entropy positions versus 0.38 at high-entropy positions. The separate v4 artifacts show 423/4,926 high-entropy rows in common support. Comparing injected surprisal and common support makes the intervention strength explicit when interpreting token location.
-
Measure the actual pre-RL operating regime. The new 68.00% versus 2.80% initial measurements make the model-specific starting conditions explicit. The training tables separately report the frequency of nonconstant rewards and the legacy-scored correctness of consensus labels, while endpoint tables provide the gold-v2 policy evaluations.
Official Comment by Authors
Dear Reviewer iC3Q:
Thank you for reviewing our paper and for recognizing its transparent multi-seed evaluation. Your questions about the motivation for token selection and the practical lessons from the results have helped us explain the central contribution more directly.
Our responses clarify the fixed-fraction question and give complete model/data-shift comparisons, an initial-policy reference, the fresh-random sensitivity, and a completed full-token pilot addressing your coverage question. We connect these results to concrete normalization, grading and control choices. We hope this clarifies the practical lessons and contribution of the study.
We would appreciate your consideration of the responses and supporting evidence. If they address your concerns about motivation and informativeness, we respectfully ask you to reconsider your assessment. We welcome any remaining questions.
Best regards,
The Authors
I thank the authors for the responses. However, my concerns are still not properly addressed as the responses are also not easy to parse. As discussion period is about to end, I ask several quick questions:
-
Why study token selection at all? Is there intuitive or theoretical reason why leaving the conventional full-token policy gradient can be beneficial and worth the complexity? To be clear, I am not asking why pick 5% instead of 10% or other ratio. Without a justification, studying where to allocate loss has zero impact.
-
What are true contribution of this paper? Comparing against random selection is not informative, especially when the results are not clean, e.g. high entropy selection can't even reliably beat random baseline.
I'll maintain my rating unless above key concerns are properly addressed.
Official Response to Reviewer iC3Q (Part 1/2)
Thank you for recognizing the transparent multi-seed evaluation. DPR supplies a configurable allocation procedure within consensus-reward learning. The Qwen anchor shows gains over the shared initial policy and both random-routing comparators at fixed coverage.
About the Motivation for Studying 5% Allocation
Five percent fixes a nontrivial selector comparison. At 100%, entropy and random selectors both select every response position and have an empty KL complement: both reduce to Global PG, so their selector contrast disappears. At 5%, their count rule and PG denominator remain matched while PG and complementary KL locations differ. The shared policy parameters remain trainable. This tests the joint allocation procedure; the choice of coverage is assessed separately.
The submitted experiment addresses fixed coverage: Table 1 (p. 7) reports an anchor contrast of +2.12 pp [1.22, 3.02]. Gold-v2 rescoring of those checkpoints gives +2.77 pp [1.83, 3.71]. The historical-control fresh-Random sensitivity gives +2.03 pp [1.28, 2.79]; tVBL Part 1 describes the cohorts and the separate bridge repeat.
Under gold-v2 evaluation, the initial Qwen policy scores 340/500=68.00%. Decision's trained mean is +2.95 pp higher; legacy Random is +0.18 pp and fresh Random +0.92 pp. All 13 anchor Decision checkpoints exceed the shared initial measurement (+0.20 to +5.00 pp), as do all 13 data-shift Decision checkpoints (+2.60 to +5.60 pp). These are descriptive pre/post comparisons against the same initial checkpoint.
The motivation is to concentrate a sequence-level reward signal at positions of high reference entropy while anchoring the complement. The Qwen selector contrasts measure accuracy at fixed coverage and a matched nominal update/rollout budget. Coverage choice is assessed separately, with each arm retaining its own rollouts and consensus labels.
About the Compute Cost and Coverage Comparisons
DPR changes where the objective applies PG and reference anchoring. This implementation performs dense policy/reference computation; the comparison measures learning outcomes at a matched nominal budget. Appendix I (p. 17) and Appendix Q (p. 23) report:
The 10% row is itself entropy-guided DPR; 5% is the main fixed-coverage condition. For Table 16's two paired 10% runs, Decision scores 59.50% versus Global 58.60% (p=0.323); Global 59.10% above is the four-seed reference. These legacy results use selected-token normalization. The new seed-1000 pilot uses total-token PG normalization and gold-v2 evaluation: Global 72.00%, Global+KL 73.20%, Decision 69.40% and fresh Random 69.60%. Gne1 Part 3 provides the matched core settings and the single-seed, non-concurrent design. The fraction sweep and full-token pilot describe coverage, while the 13-pair experiment measures allocation at fixed coverage.
The original Introduction/Related Work (pp. 2–3) distinguishes entropy localization from measured learning outcomes. The contribution is this controlled evaluation of allocation.
A concrete normalization diagnostic: recomputing Table 12's 480 updates per arm gives mean pre-clipping norms 0.406 (Global), 5.013 (Decision), 2.176 (Random), and 0.431 (Global+KL). At threshold 1, derived exceedance rates are 0%, 100%, 100%, and 0%. These historical pre-clipping measurements describe the gradient scales entering the optimizer under the recorded normalization settings.
For a fixed batch selecting about 5% of tokens, selected-token versus total-token normalization alone changes pure-PG magnitude by about 20 times. Holding that denominator fixed makes the selector question interpretable and supplies a practical control when comparing other fractions. The positive Qwen contrasts also hold on the 392 old-gold-check-pass questions (+2.61/+3.94 pp; tVBL Part 6). Together, the selector, normalization and coverage controls provide concrete guidance for evaluating selective credit assignment.