acceptodds
Under review as a conference paper at ICLR 2027

DPR: From High-Entropy Tokens to Learning Claims — Matched Evaluation for Token-Selective LLM Reinforcement Learning

Abstract

Decision-point reinforcement (DPR) is a verifier-free algorithmic framework for routing sequence-level reinforcement signals to informative response tokens. For each prompt, the policy generates on-policy rollouts and constructs a consensus pseudo-label from semantically equivalent parsed answers, yielding group-level reward signals. A frozen reference model ranks response positions by next-token entropy. DPR applies a clipped policy-gradient objective to the highest-entropy positions and anchors complementary tokens to the reference policy through a low-variance KL penalty. This dual-mask objective integrates entropy-guided credit assignment, reference-policy regularization, and total-token normalization within a unified training procedure. A count-matched random-routing counterpart and shared optimization contract enable precise evaluation of token-location quality. Controlled experiments show that DPR outperforms matched random routing in the anchor and data-shift evaluations, with favorable effects across learning checkpoints. Overall, DPR provides an effective and auditable approach to selective credit assignment for language-model reinforcement learning.

Primary area:
reinforcement learning
Keywords:
decision-point reinforcement, verifier-free reinforcement learning, token-selective reinforcement learning, high-entropy tokens, consensus pseudo-labeling, credit assignment, KL regularization, matched-control evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iC3QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces DPR, an LLM RL algorithm that updates the top-entropy tokens toward a self-consistency-based reward while anchoring the remaining tokens toward the reference model. Its contribution is a controlled evaluation on implementation choices of related algorithms. Experiments show the effect of token selection choice or ratio is nuanced across different setups.

Critical strengths

A strength is transparent and multi-seed evaluation. The paper runs 13 seed per setting across two models and two datasets, and reports both positive and negative results.

Critical weaknesses
  1. Lack of motivation for studying token selection method. It is not standard practice in LLM RL to limit effective updates to only 5% of tokens and this brings no training compute benefit. The paper's own results in Appendix I and Q show that standard 100% token updates and even token selection with 10% updates achieve higher accuracy than the studied 5% update. Comparing which 5% of tokens to update is therefore poorly motivated.

  2. No actionable insight from the evaluation. The paper provide little information for the community even if studying 5% token selection is justified. It shows that entropy selection beats random selection on Qwen2.5-Math-7B but loses on DeepSeekMath-7B-Base, and explicitly admits (Section 5.5) that it cannot explain why this mismatch happens.

Questions for authors
  1. If updating with more tokens achieves better accuracy with same FLOPs, why would we select 5% tokens to update on?
  2. Is there any explanation for why high entropy selection may outperform or underperform random selection depending on concrete setups?
4 comments
AuthorsOpenReview ↗

Official Response to Reviewer iC3Q (Part 1/2)

Thank you for recognizing the transparent multi-seed evaluation. DPR supplies a configurable allocation procedure within consensus-reward learning. The Qwen anchor shows gains over the shared initial policy and both random-routing comparators at fixed coverage.

About the Motivation for Studying 5% Allocation

If updating with more tokens achieves better accuracy with same FLOPs, why would we select 5% tokens to update on?

Five percent fixes a nontrivial selector comparison. At 100%, entropy and random selectors both select every response position and have an empty KL complement: both reduce to Global PG, so their selector contrast disappears. At 5%, their count rule and PG denominator remain matched while PG and complementary KL locations differ. The shared policy parameters remain trainable. This tests the joint allocation procedure; the choice of coverage is assessed separately.

The submitted experiment addresses fixed coverage: Table 1 (p. 7) reports an anchor contrast of +2.12 pp [1.22, 3.02]. Gold-v2 rescoring of those checkpoints gives +2.77 pp [1.83, 3.71]. The historical-control fresh-Random sensitivity gives +2.03 pp [1.28, 2.79]; tVBL Part 1 describes the cohorts and the separate bridge repeat.

Under gold-v2 evaluation, the initial Qwen policy scores 340/500=68.00%. Decision's trained mean is +2.95 pp higher; legacy Random is +0.18 pp and fresh Random +0.92 pp. All 13 anchor Decision checkpoints exceed the shared initial measurement (+0.20 to +5.00 pp), as do all 13 data-shift Decision checkpoints (+2.60 to +5.60 pp). These are descriptive pre/post comparisons against the same initial checkpoint.

The motivation is to concentrate a sequence-level reward signal at positions of high reference entropy while anchoring the complement. The Qwen selector contrasts measure accuracy at fixed coverage and a matched nominal update/rollout budget. Coverage choice is assessed separately, with each arm retaining its own rollouts and consensus labels.

About the Compute Cost and Coverage Comparisons

DPR changes where the objective applies PG and reference anchoring. This implementation performs dense policy/reference computation; the comparison measures learning outcomes at a matched nominal budget. Appendix I (p. 17) and Appendix Q (p. 23) report:

ConfigurationRecorded update-81 accuracySeeds
Global, full-token PG59.10%4
Decision, 1%55.40%2
Decision, 5% historical reference57.45%4
Decision, 10%59.50%2
Global+KL58.00%4

The 10% row is itself entropy-guided DPR; 5% is the main fixed-coverage condition. For Table 16's two paired 10% runs, Decision scores 59.50% versus Global 58.60% (p=0.323); Global 59.10% above is the four-seed reference. These legacy results use selected-token normalization. The new seed-1000 pilot uses total-token PG normalization and gold-v2 evaluation: Global 72.00%, Global+KL 73.20%, Decision 69.40% and fresh Random 69.60%. Gne1 Part 3 provides the matched core settings and the single-seed, non-concurrent design. The fraction sweep and full-token pilot describe coverage, while the 13-pair experiment measures allocation at fixed coverage.

The original Introduction/Related Work (pp. 2–3) distinguishes entropy localization from measured learning outcomes. The contribution is this controlled evaluation of allocation.

A concrete normalization diagnostic: recomputing Table 12's 480 updates per arm gives mean pre-clipping norms 0.406 (Global), 5.013 (Decision), 2.176 (Random), and 0.431 (Global+KL). At threshold 1, derived exceedance rates are 0%, 100%, 100%, and 0%. These historical pre-clipping measurements describe the gradient scales entering the optimizer under the recorded normalization settings.

For a fixed batch selecting about 5% of tokens, selected-token versus total-token normalization alone changes pure-PG magnitude by about 20 times. Holding that denominator fixed makes the selector question interpretable and supplies a practical control when comparing other fractions. The positive Qwen contrasts also hold on the 392 old-gold-check-pass questions (+2.61/+3.94 pp; tVBL Part 6). Together, the selector, normalization and coverage controls provide concrete guidance for evaluating selective credit assignment.

AuthorsOpenReview ↗

Official Response to Reviewer iC3Q (Part 2/2)

About the Model-Dependent Results

Is there any explanation for why high entropy selection may outperform or underperform random selection depending on concrete setups?

The evidence separates pre-RL conditions, trained endpoints and output/scorer interactions. All three update-81 settings retain their full 13 paired seeds under gold-v2:

SettingDecisionRandomDifference; paired 95% CI (pp)
Qwen anchor70.95%68.18%+2.77 [1.83, 3.71]
Qwen training-data shift72.08%68.08%+4.00 [3.14, 4.86]
DeepSeek model shift0.00%2.38%-2.38 [-2.65, -2.12]

All data-shift contrasts are positive and all DeepSeek contrasts negative. The Qwen base scores 68.00% before RL, whereas DeepSeek scores 2.80% under this prompt and decoding. The initial measurements establish the pre-RL operating conditions, and the model-dependent endpoint ordering persists under gold-v2 scoring. The Qwen historical-control fresh-Random contrast is also positive (+2.03 pp).

The same 13 Qwen anchor pairs also favor Decision on five additional contest sets: 24.25% versus 16.56% across 170 questions, a +7.69 pp difference [6.93, 8.45]. The full results appear in tVBL Part 4.

In the seed-2000 DeepSeek Decision checkpoint, 499/500 outputs contain an explicit formatting example ending in boxed 123. Synthetic controls show that appending this example can override an earlier correct prediction during extraction. The 499 detected outputs contain no other non-123 balanced box. These measurements identify an output/extractor interaction alongside the observed training behavior. The reference-entropy ranking measures uncertainty; the consensus reward and the fixed extraction rules determine how generated answers enter the learning and evaluation pipeline.

About the Practical Findings

No actionable insight from the evaluation.

  1. Match the PG denominator. For a fixed batch selecting S>0S>0 of TT tokens, Eq. (6), gSPG=(T/S)gTPGg_S^{\mathrm{PG}}=(T/S)g_T^{\mathrm{PG}}, gives the pure-PG denominator effect. The selected-to-total factor is S/TS/T. This clarifies the direction stated in Section 5.2, consistent with Eq. (6) and the implemented loss; KL and AdamW transformations are treated separately.

  2. Use paired training runs. Appendix P.4's reproduced historical example gives pooled-problem p=0.043 but paired-run p=0.211. All completed 13-seed comparisons use paired-run inference and report run-level uncertainty, with questions and checkpoints aggregated within each training run.

  3. Test semantic grading and irrelevant formatting. Bare gold 70 \sqrt{2} could become 70, and a boxed formatting example can override an earlier answer. The all-endpoint audit localizes all 3,524 upward corrections to the 108 preidentified gold-check failures, while retaining 96 downward corrections. The gold checks and synthetic formatting controls quantify distinct parts of the scoring pipeline.

  4. Check the selector contrast against fresh masks. All 13 fresh-Random runs are complete: retained Decision is +2.03 pp higher, and every leave-one-seed-out mean stays positive (+1.83 to +2.22 pp). The corresponding legacy-routing contrast is +2.77 pp. tVBL Part 1 specifies the historical-control design and reports the separate current-code Decision bridge, preserving the checkpoint identities for each comparison.

  5. Control perturbation strength and privileged tokens. The submitted v3 intervention records 22.99 nats of forced-replacement surprisal at low-entropy positions versus 0.38 at high-entropy positions. The separate v4 artifacts show 423/4,926 high-entropy rows in common support. Comparing injected surprisal and common support makes the intervention strength explicit when interpreting token location.

  6. Measure the actual pre-RL operating regime. The new 68.00% versus 2.80% initial measurements make the model-specific starting conditions explicit. The training tables separately report the frequency of nonconstant rewards and the legacy-scored correctness of consensus labels, while endpoint tables provide the gold-v2 policy evaluations.

AuthorsOpenReview ↗

Official Comment by Authors

Dear Reviewer iC3Q:

Thank you for reviewing our paper and for recognizing its transparent multi-seed evaluation. Your questions about the motivation for token selection and the practical lessons from the results have helped us explain the central contribution more directly.

Our responses clarify the fixed-fraction question and give complete model/data-shift comparisons, an initial-policy reference, the fresh-random sensitivity, and a completed full-token pilot addressing your coverage question. We connect these results to concrete normalization, grading and control choices. We hope this clarifies the practical lessons and contribution of the study.

We would appreciate your consideration of the responses and supporting evidence. If they address your concerns about motivation and informativeness, we respectfully ask you to reconsider your assessment. We welcome any remaining questions.

Best regards,

The Authors

I thank the authors for the responses. However, my concerns are still not properly addressed as the responses are also not easy to parse. As discussion period is about to end, I ask several quick questions:

  1. Why study token selection at all? Is there intuitive or theoretical reason why leaving the conventional full-token policy gradient can be beneficial and worth the complexity? To be clear, I am not asking why pick 5% instead of 10% or other ratio. Without a justification, studying where to allocate loss has zero impact.

  2. What are true contribution of this paper? Comparing against random selection is not informative, especially when the results are not clean, e.g. high entropy selection can't even reliably beat random baseline.

I'll maintain my rating unless above key concerns are properly addressed.

Reviewer_tVBLOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies whether directing policy-gradient updates to high-entropy tokens improves language-model reinforcement learning relative to count-matched random token selection. DPR constructs sequence-level rewards from consensus among sampled answers, selects the top 5% of response positions by frozen-reference entropy, and applies policy-gradient updates at these positions while regularizing complementary positions toward the reference model.

The main contribution is an evaluation protocol that controls token-count rules, gradient normalization, optimization settings, scoring semantics, and statistical analysis. Experiments use 13 seed-matched training pairs in each of three settings. High-entropy selection improves MATH-500 accuracy over random routing by 2.12 percentage points in the anchor setting and 3.03 points after a training-data shift, but performs 2.18 points worse after a model-family shift. Historical diagnostics illustrate potential confounds involving gradient scaling, counterfactual interventions, parsing, and the statistical unit of analysis. The authors appropriately frame the findings as setting-dependent rather than establishing universal superiority of entropy-based selection.

Critical strengths
  1. Careful separation of the claims and statistical units. The paper distinguishes descriptive entropy localization, interventions using privileged replacement tokens, and actual training improvements. Its prospective comparisons use 13 seed-matched run pairs per setting, paired confidence intervals, sign-flip sensitivity analyses, and explicit secondary-test families. This is consequential because the historical example demonstrates how pooling evaluation problems across seeds can produce an apparently significant result that is not supported by run-level inference.

  2. Transparent reporting of negative results and implementation limitations. The model-family reversal is retained prominently, and the paper explicitly discloses benchmark reuse, incomplete historical provenance, and a deviation from outcome blindness. The discussion of normalization and counterfactual perturbation strength provides useful guidance for evaluating token-selective RL. These disclosures make the conclusions more credible and avoid presenting a favorable result in one configuration as a general property of high-entropy tokens.

Critical weaknesses
  1. The random-routing comparator has an important unresolved dependence structure. Section 3.5 and Appendix A state that its random seed depends on the training seed and within-update prompt slot, but not on update or rollout identity. Equal-length completions can therefore reuse the same positional mask. This is a legitimate comparison against the implemented random procedure, but it does not establish an advantage over freshly sampled count-matched routing, and persistent positional coverage could affect learning. Since the central contribution is a controlled selector comparison, a sensitivity test with independently keyed masks is important. Moreover, changing the selector changes both policy-gradient locations and complementary KL locations, so the estimand concerns their joint allocation rather than policy-gradient placement alone.

  2. The evidence remains narrow, and the most informative negative result is difficult to interpret. Appendix N acknowledges repeated MATH-500 inspection during development and an endpoint choice informed by earlier benchmark behavior. The replications therefore provide additional training-seed evidence on a reused benchmark, rather than independent task-level confirmation. More importantly, Table 10 reports model-shift pseudo-label accuracies of only 0.17% and 1.16%, far outside the informative range used to motivate the anchor configuration. The reversal is valid evidence about that complete training setup, but it remains unclear whether it reflects selector behavior under useful supervision or behavior when consensus rewards are almost entirely incorrect. Finally, the preregistrations, executable validators, and raw artifacts remain internal, limiting independent assessment of the auditability that constitutes the paper’s principal contribution.

Questions for authors
  1. Control-design clarification and sensitivity: Does the implementation reset the random generator for each completion, causing identical masks for equal-length completions in the same prompt slot? Does the Decision–Random ordering persist when random masks are independently keyed by training seed, update, prompt, and rollout? Please also state explicitly that the comparison jointly changes policy-gradient and complementary anchoring locations.

  2. Model-shift clarification: Are the 0.17% and 1.16% pseudo-label accuracies in Table 10 correctly reported as percentages? Please provide initial and checkpoint-level absolute evaluation accuracies for both E10 arms, together with pseudo-label quality over training. Were the base model, prompting format, parser, and gold-diagnostic equivalence checks validated before training? These details would clarify whether the reversal occurs during meaningful learning, reward-induced degradation, or a scoring mismatch.

  3. Independent evaluation evidence: Is there an evaluation set that was not inspected during model, learning-rate, generation-budget, or endpoint selection? Evaluation of the existing checkpoints on such a set would materially strengthen the claim that the reported ordering extends beyond the reused MATH-500 benchmark.

  4. Control-design clarification and sensitivity: Does the implementation reset the random generator for each completion, causing identical masks for equal-length completions in the same prompt slot? Does the Decision–Random ordering persist when random masks are independently keyed by training seed, update, prompt, and rollout? Please also state explicitly that the comparison jointly changes policy-gradient and complementary anchoring locations.

  5. Model-shift clarification: Are the 0.17% and 1.16% pseudo-label accuracies in Table 10 correctly reported as percentages? Please provide initial and checkpoint-level absolute evaluation accuracies for both E10 arms, together with pseudo-label quality over training. Were the base model, prompting format, parser, and gold-diagnostic equivalence checks validated before training? These details would clarify whether the reversal occurs during meaningful learning, reward-induced degradation, or a scoring mismatch.

7 comments
AuthorsOpenReview ↗

Official Response to Reviewer tVBL (Part 1/6)

Thank you for identifying the random-mask dependence. We directly answer Questions 1 and 4 together: the legacy dependence is real, and the planned fresh-mask training sensitivity is now complete.

About the Random-Mask Construction

Does the implementation reset the random generator for each completion, causing identical masks for equal-length completions in the same prompt slot?

Yes. The legacy seed is 1,000,003 × training_seed + within_update_prompt_slot, without update or rollout identity. Equal-length completions in the same slot therefore reuse positional masks, including across updates. This is disclosed in Section 3.5 and Appendix A and reproduced in CPU checks. The original result concerns that implemented comparator.

About the Fresh-Mask Sensitivity

Does the Decision–Random ordering persist when random masks are independently keyed by training seed, update, prompt, and rollout?

The prespecified historical-control comparison remains positive: the 13 retained Decision checkpoints average 70.95%, versus 68.92% for 13 newly trained independently keyed Random runs, a +2.03 pp difference [1.28, 2.79]. The new key includes training seed, completed-update index, within-update prompt slot and original rollout index. All 13 fixed runs reached update 81 and passed artifact and CPU validation. The cohorts were trained at different times.

Gold-v2 comparison on the same 500 questionsDecisionRandomPaired difference; 95% CI (pp)
Original Decision vs legacy Random, 13 pairs70.95%68.18%+2.77 [1.83, 3.71]
Retained Decision vs fresh Random, 13 pairs70.95%68.92%+2.03 [1.28, 2.79]

For the fresh comparison, 12 contrasts are positive and one is -0.20 pp; exact sign-flip p=4/8192 ≈ 4.883e-4. Fresh Random averages 0.74 pp above legacy Random, while the retained Decision cohort has a positive mean contrast against each comparator. These figures describe the two historical-control comparisons; Part 5 provides every seed's counts.

Original evidence and separate bridge. Before the fresh endpoints, the analysis plan fixed all 13 original Decision checkpoints against all 13 fresh-Random runs. We also report the separate seed-1000 Decision repeat. Under the same gold-v2 evaluation, retained Decision scores 354/500 (70.80%), the current-code Decision bridge 347/500 (69.40%), and fresh Random 348/500 (69.60%). The bridge is 1.40 pp below retained Decision and -0.20 pp relative to fresh Random. These are separate trained policies. The 13-pair analysis retains its prespecified historical Decision controls; the four-arm pilot uses the current-code bridge. Reporting the analyses separately preserves the checkpoint identities and training cohorts of each comparison.

About the Joint PG and KL Allocation

Please also state explicitly that the comparison jointly changes policy-gradient and complementary anchoring locations.

Yes: the estimand is the joint allocation strategy. Decision assigns reward-driven PG to frozen-reference high-entropy positions and KL anchoring to the complement; Random assigns the same two signals using count-matched random locations on its own live completion. These two supports cover the response, and the shared policy parameters remain trainable. CPU mask/gradient checks confirm this structure in both the preserved implementation and the diagnostic work copy. Thus the contrast concerns the joint placement of PG and complementary KL.

Count matching means the same rule at a given response length. Each arm has its own rollout text, consensus labels, response lengths and realized gradients. Nominal training/rollout budgets and total-token PG normalization are held fixed. For a fixed batch with S>0S>0 selected tokens among TT response tokens,

gSPG=TSgTPG.g_S^{\mathrm{PG}}=\frac{T}{S}g_T^{\mathrm{PG}}.

Here gSg_S uses the selected-token denominator and gTg_T the total-response-token denominator. This identity concerns pure PG, with KL and the optimizer transformation treated separately. The original paired comparison supplies the primary selector evidence, complemented by the historical-control fresh-mask sensitivity.

AuthorsOpenReview ↗

Official Response to Reviewer tVBL (Part 2/6)

About the Percentage Units

Are the 0.17% and 1.16% pseudo-label accuracies in Table 10 correctly reported as percentages?

Yes. Recalculation gives Decision 0.17% and Random 1.16%, conditional on groups with a known majority label. The calculation averages updates within each run and then the 13 run means. Both numbers are percentages computed with the legacy gold interface, using run-level aggregation.

About the Initial and Checkpoint Accuracies

Please provide initial and checkpoint-level absolute evaluation accuracies for both E10 arms, together with pseudo-label quality over training.

The recorded legacy-scored trajectories are:

Completed updatesDecision, 13-run meanRandom, 13-run mean
12.49%2.46%
410.18%2.28%
810.00%2.18%
1200.00%2.12%

All 13 Decision runs have zero recorded accuracy at updates 81 and 120. Update 1 follows the first optimization step; the initial policy is evaluated separately at update 0 below.

The completed update-0 evaluation uses the same 500 problems, greedy decoding and 1,024-token cap. DeepSeek scores 13/500 (2.60%) under the legacy interface and 14/500 (2.80%) under gold-v2. The three upward and two downward changes use identical text. This shared initial checkpoint establishes the measured starting performance before training.

All 13 retained E10 pairs have now been reevaluated at update 81. Every Decision checkpoint scores 0/500 under gold-v2. Random's gold-v2 counts in the fixed paired-run order are:

12, 13, 10, 15, 16, 13, 13, 12, 9, 8, 12, 11, 11 out of 500.

The full-cohort means are 0.00% versus 2.38%, difference -2.38 pp, paired 95% CI [-2.65, -2.12], exact sign-flip p=2/8192 ≈ 2.441e-4. All 13 differences are negative. The 13,000 regenerated legacy decisions exactly reproduce their archived vectors. This supplies complete endpoint evidence and establishes that the observed reversal survives the gold-interface correction. The table above uses legacy intermediate scores; the update-0 and update-81 measurements also report gold-v2 results.

About the Pseudo-Label Quality During Training

Appendix E's 0.40–0.80 pseudo-label-accuracy band describes the anchor configuration. For E10, Table 10 reports informative-group rates of 74.85%/98.64%, where informative groups are defined by nonconstant rewards. Consensus-answer correctness is a distinct diagnostic, reported below with its scoring interface specified.

The legacy-scored pseudo-label trajectory, using equal-weight run means within each window, is:

UpdatesDecisionRandom
1–400.48%1.35%
41–800.02%0.92%
81–1200.00%1.21%

The training-diagnostic table uses the legacy scoring interface; the endpoint tables separately report the completed gold-v2 reevaluation.

The added evidence identifies concrete transfer checks: initial task performance, gold representation, output format and consensus-answer quality. The initial-policy measurement establishes the pre-RL regime, the gold-v2 endpoints retain the observed reversal, and Part 3 documents the interaction between generated formatting examples and answer extraction. Together these results characterize E10 under the tested training and evaluation recipe.

AuthorsOpenReview ↗

Official Response to Reviewer tVBL (Part 3/6)

About the Scorer Validation and Gold-Interface Correction

Were the base model, prompting format, parser, and gold-diagnostic equivalence checks validated before training?

Before E10 training, the retained checks verified model/tokenizer identities, prompt/context compatibility, and checkpoint/optimizer/RNG recovery. Inputs used the plain problem with a step-by-step, boxed-answer instruction and no chat template. The added initial-policy evaluation, gold-interface checks and formatting controls assess task behavior under that setup.

Completed evaluation-gold repair. In the old interface, bare gold 70 \sqrt{2} could be read as 70; -\frac{\pi}{6} could fail extraction. The versioned interface uses explicit whole-expression parsing for mathematical golds, conservative handling of supported literal text, and an error for unsupported gold. Mathematical prediction parsing and consensus clustering are unchanged; literal golds use the narrow whole/final-box comparison. All 500 evaluation golds pass the representation checks. The fixed prediction extractor is also tested with the formatting controls below.

An optional-diagnostic work copy separates consensus training from gold diagnostics. Across 96 tiny-model CPU steps (16 original, 80 adapted), losses, gradients, parameters and AdamW states match exactly. With diagnostics off, training completes with missing, unsupported or access-forbidden gold, and correctness is never called. Gold-reward training continues to enforce supported-gold inputs. These CPU checks verify the executable information boundary; the historical 7B runs retain their original implementation and training records.

About the Score Changes Across All 78 Endpoints

Before reevaluation, the old-interface gold self-check partitioned the fixed 500 IDs into 392 passing and 108 failing. We have now extended the earlier seed-1000 stratum analysis to all 78 original checkpoints and 39,000 paired scoring decisions:

Fixed gold-check stratumUpward grade changesDownward grade changes
392 old-check pass, all endpoints022
108 old-check fail, all endpoints3,52474

Every upward grade change falls in the preidentified failing stratum. Part 6 reports Decision–Random differences within both fixed strata. Predictions are held fixed between scorers, and all 39,000 regenerated legacy decisions match their archived bits. This post-hoc diagnostic localizes the absolute-score increase to the gold-reading correction; the algorithm comparisons use paired-run inference.

All downward corrections are retained. The 22 changes in the passing stratum affect three distinct IDs. For example, boxed 10 followed by % is returned as 0.1 by the fixed extractor, while the question requests the percentage number and gold is 10. That recorded grade is retained. Scoring replay reproduces the archived legacy correctness vectors; comparisons between the two scorer versions use identical regenerated completion text.

About the Model-Shift Output Format

Output format also matters in E10. Across 1,500 saved E10 completions (500 from update 0 and 500 from each seed-2000 update-81 arm), the fixed extractor returns canonical 123 for initial 145/500, Decision 500/500, and Random 385/500. In 499 Decision outputs, a strict rule detects the same explicit formatting-example block containing boxed 123. Those 499 completions have 46 distinct full-text hashes but one common example-block hash. The remaining output uses a display-format variant outside the fixed detection rule.

We ran 24 synthetic controls: two answers, correct/incorrect candidates, and six formatting conditions. For gold $42$, Calculation complete: \boxed{42}. is accepted. Appending the tested formatting guide ending in \boxed{123} $\boxed{123}$ makes the extractor return 123 and reject the answer; prepending the same guide preserves 42. Both scoring interfaces exhibit this pattern. An ordinary final-answer phrase does not prevent the override; a higher-priority anchor or removing the example's boxes does. All incorrect-answer controls remain incorrect.

These controls reproducibly demonstrate how the tested formatting example affects answer extraction. The 499 detected Decision outputs contain no other non-123 balanced box. The checkpoint comparison retains its measured counts of 0 versus 12/500; the formatting analysis describes the associated output/extractor behavior.

Reviewer_Gne1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Authors propose a reinforcement learning method which does token-level credit assignment without a verifier. The reward is calculated without verifiers by generating multiple rollouts and clustering them. The token-level credit assignment is done by selecting the highest entropy tokens for the PPO-style clipped objective, and while only KL regularizer is applied to the rest of the tokens. Authors validate this method against a baseline which selects tokens randomly. The proposed method outperforms the random selection baseline on DeepScaleR Level 2and NuminaMath-CoT subset with Qwen2.5-Math7B, but the advantage is not shown when DeepSeekMath-7B-Base was used as the base model.

Critical strengths

Authors put a lot of effort on experimental design against the random baseline, and explaining every nuance of the experimental setup. Details usually left out, such as how random seeds were selected or what the rollout filtering criteria were, are carefully described. Such a rigor in experimental design help readers trust results and better interpret implications.

Authors also do a great job on the literature review, discussing what relevant papers are and how they are related to the submitted paper. This helps readers to contextualize the work.

Critical weaknesses

The empirical usefulness of the proposed method is not demonstrated. Authors compare only against a random token selection baseline. This comparison helps us understand that high-entropy token selection has some usefulness, but this comparison alone doesn't really answer how much useful the proposed method is. Rejecting the null hypothesis "the high-entropy testing does not outperform random token selection" itself is not empirically useful, because the empirical decision practitioners need to make is "what algorithm should I choose?" Without comparisons against standard algorithms which can serve as the baseline or "oracle" algorithms which leverage verifiers (like GRPO, which are standard), readers never know when the proposed method should be used.

Also, experimental depth of the study is shallow. I appreciate authors reporting negative results that the proposed method didn't work well with DeepSeek base model, but authors do not offer deeper analyses and insights on why there was a negative result. Hence, an unexplained negative result makes it difficult for readers to judge to how much extent the proposed experimental claims should be interpreted. Also, key parameters such as p_{DP} are set as frozen to a single number, so it is difficult to interpret how the results are sensitive to the parameter. Consequently, practitioners can learn very little from the empirical findings of the paper, particularly given the usefulness of high-entropy tokens was already well known.

Questions for authors

What are the baseline algorithms which use the same amount of privileged information as the proposed method (so, no ground truth annotation)? How does the proposed algorithm compare against them?

How would the proposed algorithm compare against standard reinforcement learning methods which leverage ground truths and verifiers?

Why does the advantage against random baseline not exhibited for DeepSeekMath-7B-Base?

5 comments
AuthorsOpenReview ↗

Official Response to Reviewer Gne1 (Part 1/3)

Thank you for recognizing the controlled design and literature review. DPR learns from rollout-consensus rewards. The Qwen anchor shows gains over the shared initial policy and positive paired contrasts against both random-routing controls.

About the Same-Information Baselines

What are the baseline algorithms which use the same amount of privileged information as the proposed method (so, no ground truth annotation)?

Global and Global+KL use the same rollout-consensus reward without gold correctness in the reward. Global applies policy gradient to all response positions without anchoring; Global+KL applies policy gradient and anchoring to all positions. Random applies the same selected-count rule as Decision while changing selected and complementary anchoring positions.

The submission already contains the following full-token and gold-reward results: Appendix I, Tables 5–7 (p. 17), at update 81. The recorded results are:

RewardGlobalDecisionRandomGlobal+KL
Consensus59.10%57.45%55.85%58.00%
Gold verification58.40%57.50%55.00%57.50%

Four seeds were run per arm, except gold-reward Random with two; all 30 configurations are retained. These tables provide the submission's full-token and gold-reward controls. The historical gated arms use selected-token normalization; Part 3 reports the completed total-token-normalized full-token comparison under the current evaluation protocol.

About the Initial Policy and Fresh-Random Comparison

Qwen anchor policyGold-v2 MATH-500 accuracy
Initial model, one shared checkpoint68.00% (340/500)
Decision at update 81, 13 original seeds70.95%
Legacy Random at update 81, 13 original seeds68.18%
Fresh Random at update 81, 13 newly trained runs68.92%

Model/tokenizer identity, 500 evaluation questions, prompt, greedy decoding and the 1,024-token cap are matched. Decision's mean is +2.95 pp over the measured initial policy. All 13 anchor Decision checkpoints exceed the shared baseline, with individual gains from +0.20 to +5.00 pp. All 13 data-shift Decision checkpoints also exceed it, by +2.60 to +5.60 pp. These pre/post comparisons use one shared initial checkpoint.

Decision exceeds legacy Random by +2.77 pp (paired 95% CI [1.83, 3.71]). Retained Decision exceeds newly trained fresh Random by +2.03 pp ([1.28, 2.79]). This historical-control sensitivity uses independently keyed masks and cohorts trained at different times; tVBL Part 1 gives the separate Decision-bridge diagnostic. The Qwen advantage also appears on the 392 questions whose gold passed the old self-check: E00 +2.61 pp and E01 +3.94 pp, with Holm-adjusted exact p=1.953e-3 and 1.465e-3 across six exploratory strata. tVBL Part 6 reports every stratum.

TTRL uses majority-vote rewards for test-time adaptation. Our same-information comparison evaluates Global and Global+KL within the DPR training pipeline. The completed full-token pilot gives Global/Global+KL 72.00%/73.20%, versus Decision 69.40%. Part 3 provides all four arms and the single-seed design.

About Verifier-Based Reinforcement Learning

How would the proposed algorithm compare against standard reinforcement learning methods which leverage ground truths and verifiers?

Appendix I directly compares consensus and gold-verification rewards within the same training pipeline. The gold-reward row above gives Global 58.40%, Decision 57.50%, Random 55.00% and Global+KL 57.50%. These controls assess the reward-information comparison under the recorded experimental settings.

We also checked the consensus-reward information boundary in 120 reward-boundary CPU cases and 96 tiny-model CPU steps. In the diagnostic-isolation implementation, disabling gold diagnostics preserves losses, gradients, parameters and optimizer states, including cases with missing or unsupported gold. These CPU checks verify that the consensus-reward learning path operates independently of gold diagnostics. The 7B performance results are reported in the tables above.

AuthorsOpenReview ↗

Official Response to Reviewer Gne1 (Part 2/3)

About the Model-Family Reversal

Why does the advantage against random baseline not exhibited for DeepSeekMath-7B-Base?

Table 1 (p. 7) already reports the model-family reversal. The added diagnostics identify a low measured baseline before RL and an output/extractor interaction. The following table is the separate gold-v2 scoring audit of the original checkpoints:

Setting; all 13 paired seedsDecision, gold-v2Random, gold-v2Difference; paired 95% CI (pp)
Qwen anchor70.95%68.18%+2.77 [1.83, 3.71]
Qwen training-data shift72.08%68.08%+4.00 [3.14, 4.86]
DeepSeek model shift0.00%2.38%-2.38 [-2.65, -2.12]

Every data-shift contrast is positive and every DeepSeek contrast negative. All 39,000 legacy decisions from the 78 reevaluated endpoints match the archived vectors. Gold-interface repair therefore retains both positive Qwen contrasts and the model-shift reversal shown above. The positive Qwen ordering is present in both training-data settings before and after the scoring correction.

Initial-policy evaluation supplies a concrete transfer diagnostic. The untrained DeepSeek model scores 14/500 (2.80%) with gold-v2 under this prompt and decoding, versus Qwen's 340/500 (68.00%): the difference in measured starting performance precedes RL. Across the original DeepSeek runs, the legacy-scored Decision mean moves from 2.49% after one update to zero at updates 81 and 120. Table 10's 0.17% and 1.16% are correctly expressed as percentages and use the legacy gold interface. The initial-policy measurements and endpoint rescoring identify the operating conditions of each model under the tested protocol.

In the seed-2000 Decision checkpoint, 499/500 outputs contain the same explicit formatting-example block ending in boxed 123; the fixed detection rule finds such blocks in 341/500 Random outputs. Synthetic controls show that appending this example can override an earlier correct answer during extraction. The 499 detected Decision outputs contain no other non-123 balanced box. This identifies a concrete formatting/extraction interaction alongside the model-dependent training results and supplies a practical diagnostic when transferring the recipe.

About the Selection Fraction

it is difficult to interpret how the results are sensitive to the parameter.

Five percent fixes the main controlled comparison, while DPR supports configurable coverage. Appendix Q, Table 16 (p. 23), reports the following exploratory sweep:

FractionRecorded Decision accuracy at update 81Seeds
1%55.40%2
5% historical reference57.45%4
10%59.50%2

Table 16's 10% row uses two paired runs: Decision 59.50% versus Global 58.60% (paired p=0.323), under legacy selected-token normalization. The 10% point is itself an entropy-guided DPR configuration; 5% defines the main fixed-coverage comparison. The sweep describes accuracy at several coverage levels, with two seeds at 1% and 10% and four at the 5% historical reference. We interpret these recorded points within their stated normalization and scoring setup. The 13-seed experiment holds the total-token PG denominator fixed and specifies complementary KL coverage.

About the Contribution Relative to Prior Work

the usefulness of high-entropy tokens was already well known.

High-Entropy Minority Tokens studies RLVR with batch-level top-20% selection and no KL loss. Its Eq. (6) changes both the token mask and the denominator to the selected-token count relative to full-token DAPO. DPR asks a separate question: does entropy-guided allocation outperform random allocation when the selected-count rule and total-response-token PG denominator are held fixed? It uses frozen-reference ranking, consensus rewards and complementary KL. The estimand remains joint PG/KL allocation on each arm's own rollouts.

DPR makes entropy-guided allocation directly testable under a matched selected-count rule and a common policy-gradient denominator. The Qwen contrasts, completed fresh-random sensitivity and model-family comparison provide evidence for that controlled question. Section 2 (p. 3) distinguishes entropy localization from learning comparisons. The scoring and format analyses add concrete checks for interpreting those comparisons.

AuthorsOpenReview ↗

Official Response to Reviewer Gne1 (Part 3/3)

About the Full-Token Controls

Appendix I already provides the baseline data quoted in Part 1. The additional four-arm pilot uses a different seed cohort, total-token gated PG normalization, independently keyed Random and gold-v2 endpoint scoring. It compares full-token consensus-reward training with the current-code Decision and fresh-Random controls at seed 1000.

ArmPolicy-gradient positionsKL positionsCorrect /500Accuracy
GlobalAllNone36072.00%
Decision (bridge)Frozen-reference top 5%Complement34769.40%
Fresh RandomCount-matched random subsetComplement34869.60%
Global+KLAllAll36673.20%

At seed 1000, Global and Global+KL exceed Decision by 2.60 and 3.80 pp. All four arms exceed the shared initial checkpoint's 340/500 (68.00%), by +4.00, +1.40, +1.60 and +5.20 pp in table order. These are descriptive comparisons against one shared initial measurement.

About the Experimental Setup

All four arms use Qwen2.5-Math-7B, the same training-data records, rollout-consensus rewards, seed 1000, learning rate, total-response-token PG denominator, and update-81 endpoint. The configured budget is eight prompts and eight rollouts per prompt per update, with a 1,024-token generation cap. Evaluation uses the same 500 MATH questions, prompt, greedy decoding, cap and gold-v2 scorer. Each arm generates its own training rollouts; the nominal rollout and update budgets are matched.

Global and Global+KL were newly trained; Decision and Random are the previously completed current-code bridge and independently keyed control. This pilot uses its prespecified bridge Decision checkpoint, while the 13-pair historical-control analysis retains the original Decision checkpoints. Seed, endpoint and both new arms were fixed before their results, and both are reported. All 81 updates, endpoint checkpoints and 500 evaluation outputs passed artifact checks and saved-text scoring replay under the frozen scorer. The table reports a descriptive comparison with one training seed per arm, run at different times within the same consensus-reward pipeline.

About the Practical Implications

The controls provide distinct practical information: the initial-policy comparison measures change from starting accuracy, count-matched Random tests entropy-guided allocation, and Global/Global+KL benchmark full-token alternatives. Reporting these together makes the source and size of each accuracy contrast explicit in the studied training setup. This implements the controlled-evaluation contribution stated in the submission (pp. 2–3), retaining its measured gains and counterexamples.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response to All Reviewers

Thank you for the detailed reviews. We have added the requested controls and analyses, while keeping the original experiments separate from checkpoint rescoring and new training.

DPR allocates reward-driven policy gradient to high-entropy positions and KL anchoring to the complement. The 5% fraction defines the controlled selector comparison; the shared policy parameters remain trainable, and the implementation retains dense computation. The contribution is a controlled evaluation of this joint allocation under consensus rewards.

The submitted 13-pair comparisons report Decision–Random differences of +2.12 pp for the Qwen anchor, +3.03 pp after a training-data shift, and -2.18 pp after a model-family shift. Gold-v2 rescoring of the same checkpoints gives +2.77, +4.00 and -2.38 pp, respectively. The new initial-policy measurement also places all 13 Decision checkpoints in each Qwen setting above the shared starting checkpoint.

The completed fresh-mask sensitivity compares retained Decision with 13 newly trained Random runs: +2.03 pp [1.28, 2.79]. Evaluation of the same 13 original pairs on five additional contest sets gives 24.25% versus 16.56%, a +7.69 pp difference [6.93, 8.45]. The separate single-seed full-token pilot scores 72.00% for Global and 73.20% for Global+KL, alongside 69.40% for Decision and 69.60% for fresh Random.

The reviewer-specific replies provide the designs, paired counts, scoring checks and model-shift analysis. Together, these results distinguish policy improvement, allocation at fixed coverage, and the choice of coverage. We will incorporate the clarified presentation and the additional results into the revised manuscript.

AuthorsOpenReview ↗

Note to the Area Chair

Contribution and Supporting Results

DPR combines rollout-consensus rewards, frozen-reference entropy ranking and complementary KL anchoring. Its 5% rule allocates reward-driven PG; the other positions retain KL support and shared policy parameters remain trainable. The paper evaluates when that allocation improves learned-policy accuracy and which controls are needed to interpret the result.

Results Reported in the Submitted Paper (Table 1, p. 7)

SettingOriginal difference (pp)Paired 95% CI
Qwen anchor+2.12[1.22, 3.02]
Qwen training-data shift+3.03[2.23, 3.83]
DeepSeek model shift-2.18[-2.45, -1.92]

These are original-paper results. Gold-v2 reevaluation of all 78 original update-81 checkpoints gives differences of +2.77 [1.83, 3.71], +4.00 [3.14, 4.86] and -2.38 [-2.65, -2.12] pp, respectively. This applies the corrected scoring interface to the original trained policies. The fresh-random sensitivity uses new training; the additional-task evaluation uses the retained checkpoints.

All 39,000 legacy correctness decisions reproduce the archive. Gold-interface correction preserves the direction of all three comparisons. The exploratory stratum analysis also finds positive Qwen contrasts on the 392 old-gold-check-pass questions (+2.61/+3.94 pp), with all six setting-by-stratum comparisons reported and Holm-corrected. Across the endpoints, all 3,524 upward corrections lie in the 108 questions preidentified by the old gold self-check as failures; all 96 downward corrections are retained.

We also completed the fresh-mask sensitivity requested by tVBL. The 13 retained Decision checkpoints average 70.95%, versus 68.92% for 13 newly trained independently keyed Random runs: +2.03 pp [1.28, 2.79], exact sign-flip p=4.883e-4. The planned contrast retains historical Decision controls; the arms are non-concurrent. tVBL Part 1 reports the separate Decision-bridge diagnostic and the comparison design.

Under gold-v2, the initial Qwen model scores 68.00%, providing a descriptive +2.95 pp reference for the audited anchor Decision mean. Every one of the 13 anchor Decision checkpoints exceeds this shared baseline (+0.20 to +5.00 pp), as does every data-shift Decision checkpoint (+2.60 to +5.60 pp). These are descriptive comparisons with one initial measurement.

Additional tasks: Evaluation of the same 13 E00 pairs on five additional contest sets totaling 170 questions gives Decision 24.25% versus legacy Random 16.56%, +7.69 pp [6.93, 8.45], exact p=2/8192 ≈ 2.441e-4. Part 4 of the tVBL response gives the complete results and aggregate paired counts. This evaluates cross-task transfer of the existing policies, with the paired training seed remaining the statistical unit.

The DeepSeek reversal was already present in the submission. Its newly evaluated initial checkpoint scores 2.80%, and all 13 gold-v2 paired contrasts remain negative. The percentage units are verified; the controlled formatting examples document an output/extractor interaction under the tested model-shift protocol.

The completed seed-1000 pilot gives Global 360/500 (72.00%) and Global+KL 366/500 (73.20%), alongside the current-code Decision bridge at 347/500 (69.40%) and fresh Random at 348/500 (69.60%). All four use consensus rewards and total-token PG normalization with matched core settings. Global and Global+KL exceed Decision by 2.60 and 3.80 pp. This descriptive comparison uses one seed per arm, with the newly trained full-token arms and previously completed bridge controls run at different times.

These results address the scope stated in the submitted Contributions and Positioning sections (pp. 2–3): assessing a token-allocation claim requires matched optimization, a valid comparator, reliable grading and run-level inference. DPR supplies explicit controls and measured contrasts for that assessment. We ask that the contribution be assessed against these controlled questions and the full reported results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.