acceptodds
Under review as a conference paper at ICLR 2027

CreditFilter: Calibrated Token Weights over the Within-Response Entropy Rank for RLVR

Abstract

Critic-free reinforcement learning with verifiable rewards (RLVR) typically assigns a uniform advantage to every token. Token-reweighting heuristics like the 80/20 mask favor high-entropy tokens, but fail to measure a token's actual impact on the outcome. We formalize this impact as the forking value: the variance of the success probability over the policy's next-token choices. Because the expected forking values along a sequence sum exactly to the reward variance, token credit acts as a budget each response spends at its own forks; consequently, weights should redistribute credit strictly within a response. Mathematically, the forking value factorizes into a fork consequence, which requires costly continuations to estimate, and a fork probability, which is intrinsically provided by the softmax. On Qwen3-8B-Base, this fork probability decays exponentially with a token's within-response entropy rank at class-specific rates, revealing that most tokens carry negligible forking value. CreditFilter weights each token using this exponential envelope with no extra forward passes, stabilized by difficulty damping and an entropy-gated fallback. CreditFilter outperforms GRPO and DAPO by 4–5 points on MATH500 and 8–10 on the AIME sets, and its dynamically adaptive variant matches it. The ablations reveal that the majority of this gain stems from ranking tokens within each response rather than globally across the batch, the coordinate on which the 80/20 mask ranks.

Primary area:
reinforcement learning
Keywords:
Reinforcement learning with verifiable rewards, LLM reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_oTEfOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to distribute a response-level reward more appropriately across the tokens generated by a model. Tokens are the basic units used by the model to generate text. Existing methods typically assign the same advantage signal to all tokens in a response, even though different positions may have very different effects on the final answer.

The authors use “forking value” to measure the importance of a position: how much would the probability of a correct final answer differ if different next tokens were chosen at that position? Measuring this effect directly requires many continuations. The authors therefore use the more easily computed “fork probability” as a proxy and examine its relationship with uncertainty rank within a response.

CreditFilter ranks tokens by uncertainty within each response and gradually reduces their training weights as the rank moves down. It also includes a minimum weight for correct responses, reduced reinforcement of successful responses on easy problems, and a fallback to uniform weights when overall uncertainty is too high. The dynamic variant, CreditFilter-D, updates the rate of weight decay during training.

The main experiments use Qwen3-8B-Base and DAPO-17k. In Table 1, CreditFilter achieves late-checkpoint average accuracies of 87.6%, 33.8%, and 27.7% on MATH500, AIME24, and AIME25, respectively, compared with DAPO's 83.0%, 24.4%, and 17.7%. The paper also provides component ablations, measurements across several models, and additional experiments on Qwen3-1.7B.

Critical strengths
  1. Ranking tokens within each response is a simple improvement supported by experimental evidence.

    Under the CreditFilter configuration, switching to batch-level ranking reduces accuracy by 5.0, 7.4, and 8.3 percentage points on the three benchmarks. In another comparison, changing the original 80/20 method to select the most uncertain 20% of tokens within each response improves accuracy by 8.4, 12.9, and 11.4 percentage points, respectively (Table 2, p. 8; Table 5, p. 15). These two comparisons support the value of within-response ranking from different angles and form one of the paper's more convincing contributions.

  2. The authors carefully measure how choices at different positions affect the final answer and explain the limitations of those measurements.

    The authors try different tokens at the same position and generate repeated continuations to estimate their effects on the probability of answering correctly. In addition to measurements restricted to a small set of candidates, they design a method that includes the remaining candidates, run approximately 3.58 million continuations, and report uncertainty intervals (Appendix E.4, Tables 13–14, pp. 25–27). This provides empirical support for the method. The authors also discuss omitted candidates and continuation-length limits, helping readers assess the scope of the evidence.

Critical weaknesses
  1. A gap remains between what the theory establishes and the weights used in practice; the additional value of exponential weighting is also unclear.

    Proposition 1 states that, for a given problem, averaging the total forking value across all possible responses yields a fixed quantity. It does not state that every individual response has the same total, so it does not directly imply that each response should receive the same mean weight. Appendix D.5 acknowledges this distinction (p. 19).

    Within-response rank also depends on later generated content and response length. The theory assumes that, under certain conditions, this later information does not further change the expected credit at the current position, but this assumption is not tested (Assumption 2, p. 5). Approximating actual importance using the more easily computed fork probability also omits factors related to success probability and fork consequence. The paper does not sufficiently validate these approximations in actual training, and settings such as length penalties fall outside the relevant theoretical results.

    The experiments likewise do not clearly establish that exponential weighting is better than the simpler within-response top-20% method. Under the same training recipe, the top-20% window reaches 38.0% and 28.0% on AIME24 and AIME25, compared with CreditFilter's 33.8% and 27.7% (Table 2). Exponential weighting performs better on MATH500, but it remains difficult to determine how much additional benefit comes from measuring and calibrating the exponential curve. This affects my assessment of the size of the methodological contribution.

  2. More evidence is needed to establish whether the main results are stable and whether the method performs better at comparable computational cost.

    The main comparisons do not report multiple independent training runs. Resampling test problems and averaging results from three neighboring checkpoints cannot replace retraining with different random seeds. Each AIME benchmark also contains only 30 problems, so the stability of the gains should be assessed cautiously.

    The configurations are not fully matched across methods. CreditFilter includes additional stabilization mechanisms, while some baselines differ in clipping parameters, activation conditions, or input templates. Appendix C discloses these differences, but they still affect the claim of consistently outperforming prior methods (pp. 14–15).

    Equal training steps do not necessarily mean equal training cost, because methods may generate different numbers of responses with different lengths. The paper does not report complete training-token totals or GPU time. At evaluation, CreditFilter's average output length is approximately 1.8–3.1 times that of GRPO/DAPO (Table 15). These outputs include text generated after the answer, so the difference cannot simply be treated as longer reasoning. Nevertheless, comparisons under matched inference budgets are still needed. The second-model experiments are a useful addition, but each configuration has only one run, and accuracy gains over the untrained model are small. They therefore do not fully resolve these concerns (Table 16, p. 28).

Questions for authors
  1. Could you further clarify whether the theoretical assumptions hold in actual training?

    In particular, could subsequent response length and uncertainty ranking affect the credit estimate at the current position? How much do the omitted factors matter when fork probability is used in place of actual importance? Also, why does the variance result that holds on average support the per-response budget design? If these connections cannot currently be validated, I suggest presenting the corresponding claims as design motivation.

  2. What is the main advantage of exponential weighting over the simpler within-response top-20% method?

    With damping, fallback, clipping, and mean weight matched, could you report uncertainty intervals for the performance differences and results from multiple independent training runs? In Table 2, the top-20% method performs better on both AIME benchmarks. Please clarify whether exponential calibration mainly benefits accuracy, training stability, or another aspect.

  3. Do the main gains persist at comparable computational cost and across multiple independent training runs?

    Please provide training-token totals, GPU time, and results with different random seeds for CreditFilter, DAPO, and the within-response top-20% method, together with comparisons under matched inference-token budgets. For the STARE and SC-GRPO configuration differences described in Appendix C, please also explain how much these differences might affect the comparisons.

5 comments

Dear Reviewer oTEf, thank you for your detailed review and for recognizing that the experiments support ranking tokens within each response, and the forking-value measurements. We answer in three comments: (1) the independent training runs and the cost, (2) the matched budgets and the matched configurations, (3) the theory and the comparison with the top-20% window. Every change we mention is in the revised PDF, marked in blue.


W2 & Q3 (independent runs): The main comparisons do not report multiple independent training runs. Please provide results with different random seeds for CreditFilter, DAPO and the within-response top-20% method.

A1: We trained two more data seeds for each of the three methods (Appendix F.2, Table 24). The table gives late averages on MATH500 / AIME24 / AIME25.

Seed 201Seed 300Seed 400Mean ±\pm s.d.
CreditFilter87.6 / 33.8 / 27.783.1 / 27.8 / 21.787.2 / 36.0 / 27.986.0 ±\pm 2.5 / 32.6 ±\pm 4.2 / 25.8 ±\pm 3.5
DAPO83.0 / 24.4 / 17.785.8 / 25.6 / 19.266.0 / 16.3 / 13.178.3 ±\pm 10.7 / 22.1 ±\pm 5.0 / 16.7 ±\pm 3.2
Within-response top-20% window85.4 / 38.0 / 28.072.2 / 29.0 / 23.287.7 / 37.2 / 27.281.7 ±\pm 8.4 / 34.7 ±\pm 5.0 / 26.1 ±\pm 2.6

Over the three seeds, the standard deviation of CreditFilter's late averages is 2.5 / 4.2 / 3.5 points, and DAPO's is 10.7 / 5.0 / 3.2. On both AIME sets, each of the six runs with a within-response window is above each of the three DAPO runs. On every set, CreditFilter's margin over DAPO in Table 1 is the middle one of the three seeds.

Paired by seed, CreditFilter minus DAPO is +7.7 ±\pm 12.2 / +10.5 ±\pm 8.8 / +9.1 ±\pm 6.2 points. CreditFilter minus the top-20% window is +4.3 ±\pm 5.9 / $-$2.1 ±\pm 1.7 / $-$0.3 ±\pm 1.1 points. These are means ±\pm standard deviations over the three seeds, computed before rounding. Our third comment compares the two windows (Q2).

W2 & Q3 (cost): Equal training steps do not mean equal training cost. Please provide training-token totals and GPU time. The reported output lengths include text generated after the answer.

A2: Training cost (Appendix F.1, Table 19). Over 400 steps CreditFilter trained on 6.04 billion tokens in about 145 hours, and DAPO on 2.87 billion in about 70, both on 16 H100 GPUs. The top-20% window trained on 6.88 billion tokens, which is about 165 hours at the same time per token. Every step trains on the same number of prompts and responses, in the same number of updates. The extra time therefore comes from longer responses.

Length up to the answer (Table 23). The lengths in the submission's Table 15 (Table 17 in the revision) include text written after the answer. The revision also counts each response only up to where it first states its final answer. By this count CreditFilter writes 2,556 / 7,028 / 6,607 tokens on MATH500 / AIME24 / AIME25, DAPO 905 / 1,969 / 1,549 and GRPO 1,696 / 4,080 / 3,300. CreditFilter's reasoning up to the answer is 2.8 to 4.3 times DAPO's and 1.5 to 2.0 times GRPO's.

W2 & Q3 (matched budgets): Do the main gains persist at comparable training cost and under matched inference-token budgets?

A3: We ran three comparisons. CreditFilter leads at equal training tokens and at equal tokens spent. At matched decoding budgets it leads from 8k tokens upward and trails at 4k.

Equal training tokens (Table 21). We continued the DAPO run beyond step 400. More training does not raise its accuracy. It scores 83.7 / 25.9 / 19.5 at step 400. At its five checkpoints from step 500 to 600 it scores between 82.9 and 84.0 / 25.6 and 27.5 / 18.8 and 21.0. By step 600 it has trained on 1.56 times its tokens at step 400. At 4.2 billion training tokens, CreditFilter's step-300 checkpoint scores 85.0 / 29.9 / 25.0 and DAPO's step-575 checkpoint 82.9 / 25.6 / 20.4.

Matched decoding budget (Table 20). We cut each response at a token budget and grade it as in the paper. The numbers are late averages over steps 350–400.

Decoding budgetCreditFilterDAPOGRPO
4k77.9 / 16.8 / 17.682.7 / 24.2 / 17.980.6 / 20.8 / 16.8
8k86.7 / 28.6 / 25.882.8 / 24.3 / 17.983.2 / 25.3 / 19.3
12k87.5 / 32.6 / 27.482.8 / 24.3 / 17.983.2 / 25.5 / 19.4
20,48087.6 / 33.4 / 27.782.8 / 24.3 / 17.983.2 / 25.5 / 19.4

At 4k CreditFilter trails, because about 70 percent of its AIME responses have not yet written an answer by then. The table samples the responses of the same checkpoints anew, so its full-length row differs from Table 1 by at most 1 point.

Equal tokens spent (Appendix F.1). A majority vote over three DAPO samples spends more tokens than one CreditFilter response. It reaches 85.8 / 26.4 / 19.0, and a vote over eight samples reaches 87.4 / 30.3 / 21.2. One CreditFilter sample scores 87.6 / 33.8 / 27.7 (Table 1).

W2 & Q3 (matched configurations): CreditFilter includes additional stabilization mechanisms, and some baselines differ in clipping, activation conditions or templates. How much might the STARE and SC-GRPO differences affect the comparisons?

A4: The stabilizers do not produce the gain without the window. Two rows of Table 2 separate the two.

  • Within-response ranking without the stabilizers. The within-response cut on the settings of the DAPO row uses DAPO's clip, and neither damping nor the fallback. This is the cleanest matched comparison. Against DAPO it loses 0.5 points on MATH500 and gains 7.8 and 8.6 points on the two AIME sets.
  • The stabilizers without the window. Damping alone, with uniform token weights, scores 71.3 / 16.8 / 13.8, which is below DAPO.

The window and the stabilizers are complementary (Section 5.4). STARE and SC-GRPO ran with their published settings. Appendix C (Table 6) lists how each run differs from its publication.

W1 & Q1: A gap remains between what the theory establishes and the weights used in practice. Proposition 1 holds on average rather than for each response, Assumption 2 is not tested, the fork probability omits factors, and the length penalty falls outside the theory. If these connections cannot be validated, please present the claims as design motivation.

A5: The revision presents the per-response design and the window as design motivation (abstract, Sections 1, 3.1 and 4). Our tests are on the base policy, so the same holds for the policies seen during training. We take your four points in turn.

  • Proposition 1. It fixes the total forking value in expectation over a prompt's responses, not for each response. The revision says so, and presents ranking within each response as a choice that the proposition motivates.
  • Assumption 2. We tested it with the regression that Reviewer RCZq proposed (Appendix E.5, Table 16; the table is also in our reply there). The part that the assumption rules out is within sampling error at the 95% level. The exception is the top 0.25% of the rank in correct responses, where the realized increment is about 1.5 times what Proposition 3 gives.
  • Omitted factors. The value factor barely changes the target's shape over the rank. With and without it, the spectra have squared cosines of 0.97 in correct and 0.98 in wrong responses (Appendix E.6). The fork consequence is the main approximation. Over the top 20% of the rank, where it can be estimated, it is flat apart from a spike in the top 2.5%. The spike is about six times its typical value in the top 0.25%, where the window already puts its largest weight (Appendix E.4).
  • Length penalty. The propositions take the reward to be the 0-1 outcome, without the overlong penalty. Section 4 and the limitations now say so.

Q2: What is the main advantage of exponential weighting over the simpler within-response top-20% method: accuracy, training stability, or another aspect?

A6: In every seed the exponential window finishes more of its responses, generates fewer tokens and trains on fewer tokens. Both windows use the same damping, fallback, clip and class mass. The table gives late averages over the three seeds, exponential / top-20% (Table 22 of the revision, with standard deviations; per-seed accuracies in Table 24).

MATH500AIME24AIME25
Accuracy (%)86.0 / 81.732.6 / 34.725.8 / 26.1
Mean generated tokens6,541 / 7,8948,611 / 11,7307,491 / 10,428
Unanswered at the cap (%)1.2 / 6.44.6 / 12.11.7 / 9.9
  • Fewer tokens. The exponential window generates 17–28% fewer tokens at evaluation. It trains on 15% fewer, 5.57 against 6.58 billion on average.
  • More answers within a budget. On the AIME sets we count only the responses that end within 8k tokens. The exponential window then gives more correct answers in every seed, 27.7 / 22.3 against 22.9 / 19.5 on AIME24 / AIME25. Within 12k tokens the two windows are level, 30.5 / 24.1 against 29.3 / 24.0.
  • The width. What the measurement supplies is the width. Fixed at twice the measured value, the window loses 9 to 15 points (Table 2). CreditFilter-D re-measures the width during training and matches CreditFilter (Section 5.4).

We hope these results address your concerns and are sufficient to reconsider the assessment. We are happy to run any further check during the discussion period.

Reviewer_RCZqOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a new way to reweigh tokens during the policy update of reinforcement learning with verifiable rewards (RLVR). The authors define a token’s forking value as the variance of the final success probability over the policy’s possible next-token choices. They show that forking values along a response sum to the reward variance, which motivates redistributing credit within each response rather than across a batch. Measuring on a base model, they find the fork probability (the softmax-available part of the forking value) decays exponentially with a token’s within-response entropy rank. CreditFilter scales each token’s advantage by an exponential window over that rank, with class-specific widths, a mass-balancing floor, difficulty damping, and an entropy-gated fallback, at no extra forward-pass cost.

Critical strengths
  • Simplicity/Soundness: The idea of redistributing a response’s credit strictly within the response, onto its highest-entropy-rank tokens with an exponential envelope calibrated to measured fork probabilities, is a simple yet sound solution.
  • Good ablation study: the component-wise ablations (window shape, rank coordinate, class mass, stabilizers) cleanly isolate what drives the gain, and the relevant baselines (GRPO, DAPO, 80/20 mask, STARE, SC-GRPO) are reproduced under a matched recipe.
Critical weaknesses
  • Overclaim: The abstract and introduction frame CreditFilter as weighting tokens by their measured impact (the forking value), but the method weights by the fork-probability envelope alone. The body is clear about this, but the abstract should state that the window is calibrated to fork probability under an approximately-holding flatness assumption.
  • Baseline reproduction: The reproduced baseline of 80/20 performs worse than DAPO, contradicting the result from the original paper. This suggests a reproduction issue.
  • Unverified assumptions: Assumption 2 assumes that “given the prefix and the outcome, the number of later tokens is more uncertain than the current one and the response length carries no further information about the current increment.” This is plausibly violated in reasoning traces, where a pivotal wrong step is often followed by a confident, low-entropy continuation, making high rank itself a signal of a large increment. The assumption is what licenses conditioning on a retrospective rank in Prop. 3/5, yet it is never tested
Questions for authors
  • Why does the reproduced 80/20 mask trail its DAPO host by 9 points on MATH500 (0.741 vs 0.830) when Wang et al. report it matching DAPO? Were the published hyperparameters replicated, or only the masking rule on your recipe?
  • Can Assumption 2 be checked on the existing branching corpus, e.g., by regressing the realized increment δ^tδ̂_t on rank within each outcome class while controlling for HtH_t?
  • What was the difference in wall-clock time?
5 comments

Dear Reviewer RCZq, thank you for your review and for recognizing the method's simplicity and the component-wise ablations. Following your suggestions, we retrained the 80/20 baseline with its released training script and ran the regression you proposed to test Assumption 2. We answer in two comments. Every change we mention is in the revised PDF, marked in blue.


W1: The abstract and introduction should state that the window is calibrated to fork probability under an approximately-holding flatness assumption.

A1: The abstract and the introduction (Section 1) now state this. The window is calibrated to the fork-probability envelope, under an approximately flat fork consequence.

W2 & Q1: The reproduced 80/20 mask trails DAPO, contradicting the original paper. Were the published hyperparameters replicated, or only the masking rule on your recipe?

A2: The submitted row used the masking rule on our settings. It took the top-20% cut over the rollout batch, on the training settings of our DAPO row. It is now the batch-cut row of Table 2, and Appendix C says so.

We have now also trained the mask with its authors' released training script. We kept our training set and prompt, with 128 prompts per step for 400 steps (settings in Appendix C, Table 6).

Late average of steps 350–400, the paper's evaluationMATH500AIME24AIME25
80/20 mask, released script82.730.223.6
Top-20% cut over the rollout batch, DAPO settings (submitted row)74.119.314.9
CreditFilter87.633.827.7
DAPO83.024.417.7

Trained with its released script, the mask no longer trails DAPO. It matches DAPO on MATH500 and exceeds it on both AIME sets.

CreditFilter leads it by 4.9, 3.6 and 4.1 points on MATH500, AIME24 and AIME25. Table 1 now reports this version, and the introduction and Section 5.2 state these margins. The submitted row stays in Table 2 as the batch-cut reference.

The ablation inside CreditFilter points the same way. With everything else fixed, ranking across the batch instead of within each response loses 5.0, 7.4 and 8.3 points.

W3 & Q2: Assumption 2 is never tested. Can it be checked on the existing branching corpus, e.g. by regressing the realized increment on rank within each outcome class while controlling for entropy?

A3: We ran this regression on the base policy's branching corpus of Appendix E, with 800 responses and 10,925 positions. Within each outcome class, we regress the realized increment on an indicator of the top 2.5% of the rank, controlling for entropy.

At equal entropy, top-ranked tokens do carry larger increments. The coefficient splits exactly into two parts. The first is what the assumption itself implies, through the prefix and the outcome. The second is a remainder that the assumption rules out. The remainder is the test.

Top 2.5% of the rank, controlling for entropy; 95% intervals in bracketsCorrectWrong
Coefficient+0.0091$-$0.0035
Part implied by the assumption+0.0061 [+0.0037, +0.0088]$-0.0019[0.0019 [-$0.0030, $-$0.0009]
Remainder+0.0030 [$-$0.0014, +0.0072]$-0.0016[0.0016 [-$0.0049, +0.0015]

Two thirds of the coefficient in correct responses, and over half in wrong ones, is what the assumption itself predicts. The remainder is within sampling error at the 95% level in both.

By finer rank band, a remainder shows only in the top 0.25% of the rank in correct responses, with the sign of the advantage. Its effect on the weight's target is small. Counting it, the top 2.5% of the rank holds 28% of the target in correct responses instead of 25%. The approximation thus puts slightly too little weight at the head, where the window already puts its largest weight.

The revision states Assumption 2 as an approximation in Section 4 and reports the full test in Appendix E.5 (Table 16).

Q3: What was the difference in wall-clock time?

A4: A factor of 2.1. CreditFilter's 400 steps took about 145 hours and DAPO's about 70, both on 16 H100 GPUs. The difference comes entirely from longer responses. CreditFilter trained on 6.04 billion tokens and DAPO on 2.87 billion, at about the same time per token. The weight adds no computation (Appendix F.1, Table 19).

Trained on from step 400, DAPO does not gain accuracy. At 4.2 billion training tokens it scores 82.9 / 25.6 / 20.4, and CreditFilter 85.0 / 29.9 / 25.0 (Table 21).

We hope these results address your concerns and are sufficient to reconsider the assessment. We are happy to run any further check during the discussion period.

Dear Reviewer RCZq, the discussion period ends on November 18. Here is how the revision answers each of your points. All scores are accuracies in percent, in the order MATH500 / AIME24 / AIME25.

  • Overclaim (weakness 1). The abstract and the introduction now state what you asked for: the window is calibrated to the fork-probability envelope, under the flatness assumption that you named (an approximately flat fork consequence).
  • Baseline reproduction (weakness 2, question 1). The submitted 80/20 row used only the masking rule, on the training settings of our DAPO run. We have now trained the mask with its authors' released training script, on our training set and for the same 400 steps. Averaged over steps 350–400, it scores 82.7 / 30.2 / 23.6, against 83.0 / 24.4 / 17.7 for DAPO (Table 1 now reports this run). The mask thus matches DAPO on MATH500 and exceeds it on both AIME sets. CreditFilter scores 87.6 / 33.8 / 27.7, which is 4.9 / 3.6 / 4.1 points above the mask.
  • Unverified assumption (weakness 3, question 2). We ran the regression you proposed, within each outcome class and controlling for entropy, on an indicator of the top 2.5% of the rank (Appendix E.5, Table 16). Its coefficient splits into two parts, and the part that Assumption 2 rules out is within sampling error at the 95% level, in correct and in wrong responses. The other part is what the assumption itself predicts. Under the assumption, each token still has an expected increment given the prefix and the outcome, and this expected increment is larger in size for top-ranked tokens. This predicted part is two thirds of the coefficient in correct responses and over half in wrong ones. By finer rank band, a remainder shows only in the top 0.25% of the rank in correct responses. There the approximation puts slightly too little weight at the head, where the window already puts its largest weight.
  • Wall-clock time (question 3). CreditFilter's training took 2.1 times as long as DAPO's. The time per token is about the same, so the difference comes from its longer responses. For a comparison at equal training tokens, we also continued DAPO's training beyond step 400. The scores of this comparison are single checkpoints (Table 21). DAPO scores 83.7 / 25.9 / 19.5 at step 400 and 82.9 / 25.6 / 20.4 at step 575, where it has trained on 4.2 billion tokens. CreditFilter reaches 4.2 billion tokens at step 300 and scores 85.0 / 29.9 / 25.0 there.

Please let us know whether these answers resolve your points. If one remains open, we will answer it within the discussion period.

Reviewer_gZRAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CreditFilter uses within-response entropy ranks to assign soft token weights in RL with verifiable rewards. Forking-value measurements motivate a decaying weighting profile, combined with a positive weight floor and damping of positive updates on easier prompts. The paper reports improvements on MATH500 and AIME and includes detailed ablations and an additional model study.

Critical strengths
  1. The paper goes beyond a generic high-entropy heuristic by measuring how token choices affect success probability. The response-local ranking ablations are particularly useful for understanding where the gains come from.
  2. The method is simple to integrate without additional training-time model forward passes. The reported reasoning gains and the analysis of entropy and response length support its practical relevance.
Critical weaknesses
  1. The theory-to-algorithm link needs a narrower interpretation. The variance decomposition is an expectation over trajectories, rather than a fixed credit budget for each realized response. The rank-based approximation also requires assumptions; it does not establish that entropy rank directly measures token credit in general.
  2. Compute efficiency is not settled by avoiding extra forward passes. Some main comparisons produce substantially longer responses, which changes rollout cost and inference budget. The additional-model length results are encouraging, but do not resolve this issue for the main experiments.
Questions for authors
  1. Can the authors clarify which parts of the entropy-rank weighting are theoretically justified and which are empirical calibration choices?
  2. Do the main accuracy gains remain when training is compared at equal generated-token budgets and evaluation uses matched decoding budgets?
3 comments

Dear Reviewer gZRA, thank you for your careful review and for recognizing the forking-value measurements and the response-local ranking ablations. We answer your points in two comments. Every change we mention is in the revised PDF, marked in blue.


W2 & Q2: Compute efficiency is not settled by avoiding extra forward passes. Do the main accuracy gains remain when training is compared at equal generated-token budgets and evaluation uses matched decoding budgets?

A1: CreditFilter adds no forward pass, but its responses are longer. At equal steps it trains on 2.1 times DAPO's tokens, and Section 5.2 of the revision now says so. We ran the comparisons you ask for. At equal training tokens CreditFilter leads. At matched decoding budgets it leads from 8k tokens upward and trails at 4k.

Matched decoding budget (Table 20 of the revision). We cut each response at a token budget and grade it as in the paper. The numbers are late averages over steps 350–400, on MATH500 / AIME24 / AIME25.

Decoding budgetCreditFilterDAPOGRPO
4k77.9 / 16.8 / 17.682.7 / 24.2 / 17.980.6 / 20.8 / 16.8
8k86.7 / 28.6 / 25.882.8 / 24.3 / 17.983.2 / 25.3 / 19.3
12k87.5 / 32.6 / 27.482.8 / 24.3 / 17.983.2 / 25.5 / 19.4
20,48087.6 / 33.4 / 27.782.8 / 24.3 / 17.983.2 / 25.5 / 19.4

At 4k CreditFilter trails, because about 70 percent of its AIME responses have not yet written an answer by then. DAPO's accuracy stops changing after 4k, and GRPO's after 8k. The table samples the responses of the same checkpoints anew, so its full-length row differs from Table 1 by at most 1 point.

Equal tokens spent (Appendix F.1). DAPO's shorter responses can instead be spent on a majority vote. A vote over three DAPO samples uses 7.3k / 13.7k / 9.6k tokens per problem. One CreditFilter response uses fewer, 6.4k / 9.1k / 8.4k. The vote reaches 85.8 / 26.4 / 19.0, and a vote over eight samples reaches 87.4 / 30.3 / 21.2. One CreditFilter sample scores 87.6 / 33.8 / 27.7 (Table 1). At equal tokens spent the lead remains.

Equal training tokens (Table 21). We continued the DAPO run beyond step 400. More training does not raise its accuracy. It scores 83.7 / 25.9 / 19.5 at step 400. At its five checkpoints from step 500 to 600 it scores between 82.9 and 84.0 / 25.6 and 27.5 / 18.8 and 21.0. By step 600 it has trained on 1.56 times its tokens at step 400. At 4.2 billion training tokens, CreditFilter's step-300 checkpoint scores 85.0 / 29.9 / 25.0 and DAPO's step-575 checkpoint 82.9 / 25.6 / 20.4.

Training cost (Appendix F.1, Table 19). Every step trains on the same number of prompts and responses, in the same number of updates. CreditFilter's training took about 145 hours and DAPO's about 70, both on 16 H100 GPUs. The time per token is about the same, so the difference comes from the longer responses. The extra tokens alone do not carry the accuracy. With a random rank in place of the entropy rank, CreditFilter's settings train on about 5.0 billion tokens, 1.7 times DAPO's. That run scores 84.9 / 25.5 / 21.7 (Table 2).

W1: The theory-to-algorithm link needs a narrower interpretation. The variance decomposition is an expectation over trajectories rather than a fixed credit budget for each realized response, and the rank-based approximation requires assumptions.

A2: The revision narrows both statements (abstract, Sections 1, 3.1 and 4).

  • Ranking within each response. Proposition 1 fixes the total forking value in expectation over a prompt's responses, not for each response. The revision presents ranking within each response as a design choice that Proposition 1 motivates rather than implies.
  • The window. The revision presents it as calibrated to the fork-probability envelope, under an approximately flat fork consequence.
  • The assumption behind the rank. We tested Assumption 2 with the regression that Reviewer RCZq proposed (Appendix E.5, Table 16). Most of the rank's association with the realized increment is explained by the prefix and the outcome. Over the top 2.5% of the rank, the part that is not explained is within sampling error at the 95% level. By finer band it shows only in the top 0.25% of correct responses, with the sign of the advantage.

Q1: Which parts of the entropy-rank weighting are theoretically justified, and which are empirical calibration choices?

A3: Table 8 in Appendix D.5 lists every part with its source. There are four kinds.

  • Derived, under stated conditions. The target of the weight is the forking spectrum (Proposition 3). Corollary 6 states when it reduces to the fork-probability envelope.
  • Measured. A single exponential fits the fork-probability envelope, with widths 0.111 and 0.155 on the base model. The runs use 0.10 and 0.14, the rounded widths of a subset of the responses (Section 3.1).
  • Chosen. Ranking within each response and the exponential form of the window follow from three requirements that do not use the measurement (Appendix D.5). The class-mass floor, the damping and the fallback are also design choices. Each is ablated in Table 2.
  • Approximated. The fork consequence is taken as flat over the rank, and the value factor is left out. The revision reports measurements of both (Appendices E.4 and E.6).

We hope these results address your concerns. We are happy to run any further check during the discussion period.

Dear Reviewer gZRA, the discussion period ends on November 18. Here is how the revision answers each of your points. All scores are accuracies in percent, in the order MATH500 / AIME24 / AIME25.

  • Theory-to-algorithm link (weakness 1). The variance decomposition of Proposition 1 holds in expectation over a prompt's responses. It does not give each response a fixed credit budget. The revision therefore presents ranking within each response as a design choice that Proposition 1 motivates rather than implies (abstract, Sections 1, 3.1 and 4). It also presents Assumption 2, the assumption behind the entropy rank, as an approximation and tests it (Appendix E.5, Table 16). In the test, the prefix and the outcome explain most of the rank's association with the realized increment, as Assumption 2 says.
  • Theory or calibration (question 1). Table 8 in Appendix D.5 marks each part of the method by its source. The target of the weight, the forking spectrum, is derived under stated conditions (Proposition 3). The decay of the fork probability over the rank is measured, and the widths of the weight are calibrated to it. Ranking within each response, the exponential form of the weight, the floor and the damping are design choices. The table also lists the approximations and where each is tested.
  • Compute efficiency (weakness 2, question 2). At equal steps CreditFilter trains on 2.1 times DAPO's tokens, and Section 5.2 now says so. We ran the comparisons you asked for (Appendix F.1).
    • Equal training tokens. CreditFilter leads. To compare at equal tokens, we continued DAPO's training beyond step 400. DAPO scores 83.7 / 25.9 / 19.5 at step 400 and 82.9 / 25.6 / 20.4 at step 575, where it has trained on 4.2 billion tokens. CreditFilter reaches 4.2 billion tokens at step 300 and scores 85.0 / 29.9 / 25.0 there (Table 21).
    • Matched decoding budget. With every response cut at the same token budget, CreditFilter leads from 8k tokens upward. At 4k it trails, because about 70 percent of its AIME responses have not yet written an answer by then (Table 20).
    • Majority vote. A vote over three DAPO samples spends more tokens per problem than one CreditFilter response. CreditFilter is ahead of this vote by 1.8 / 7.4 / 8.7 points (Appendix F.1).

Please let us know whether these answers resolve your points. If one remains open, we will answer it within the discussion period.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response

Dear Reviewers and Area Chair, we sincerely thank you for your time and constructive comments. Below we summarize the major concerns of reviewers and our responses. The revised PDF is uploaded with these comments; every change is marked in blue, and each reply names the section or table in the revision.

1. Training and decoding cost (gZRA, RCZq, oTEf). The reviews ask what CreditFilter costs: it adds no forward pass, but its responses are longer. The revision now reports the cost and compares the methods at matched budgets (Appendix F.1, Tables 19–21).

  • Cost. Over 400 steps CreditFilter trains on 6.04 billion tokens in about 145 hours, and DAPO on 2.87 billion in about 70 hours, on the same 16 H100 GPUs. This factor of about 2$\times$ comes entirely from longer responses. The time per token is about the same, and the weight itself adds no computation.
  • Equal training tokens. We continued the DAPO run beyond step 400. More training does not raise its accuracy. At 4.2 billion training tokens CreditFilter scores 85.0 / 29.9 / 25.0 on MATH500 / AIME24 / AIME25, and DAPO 82.9 / 25.6 / 20.4.
  • Equal decoding budget. We cut every response at a token budget and graded it as before. CreditFilter leads from a budget of 8k tokens upward. At 4k it trails, because about 70 percent of its AIME responses have not yet written an answer by then.
  • Equal tokens spent. A majority vote over three DAPO samples spends more tokens than one CreditFilter response. CreditFilter still leads it, by 1.8 / 7.4 / 8.7 points.

2. Independent training runs (oTEf). The submission reported one training run per method. We trained two more data seeds each for CreditFilter, DAPO and the within-response top-20% window (Section 5.2 and Appendix F.2, Table 24). Mean ±\pm standard deviation over the three seeds, on MATH500 / AIME24 / AIME25:

  • CreditFilter: 86.0 ±\pm 2.5 / 32.6 ±\pm 4.2 / 25.8 ±\pm 3.5
  • DAPO: 78.3 ±\pm 10.7 / 22.1 ±\pm 5.0 / 16.7 ±\pm 3.2
  • Top-20% window: 81.7 ±\pm 8.4 / 34.7 ±\pm 5.0 / 26.1 ±\pm 2.6

On both AIME sets CreditFilter's lowest seed is above DAPO's highest. Paired by seed, CreditFilter exceeds DAPO by +7.7 ±\pm 12.2 / +10.5 ±\pm 8.8 / +9.1 ±\pm 6.2 points. On every set the margin of Table 1 is the middle one of the three seeds.

3. From the theory to the weight (gZRA, RCZq, oTEf). The reviews ask how much of the method the theory supports. We made three changes.

  • Narrower claims. The abstract and Sections 1 and 4 now present ranking within each response as a design choice that the theory motivates. They present the window as calibrated to the fork-probability envelope under an approximately flat fork consequence.
  • A test of Assumption 2. We ran the regression that Reviewer RCZq proposed (Appendix E.5, Table 16). Over the top 2.5% of the rank, the part that the assumption rules out is within sampling error at the 95% level.
  • The parts of the method by source. Table 8 in Appendix D.5 lists which parts are derived, measured, chosen or approximated. Appendix E.6 measures the factor that the window leaves out: it barely changes the shape of the target.

4. The 80/20 baseline (RCZq). In the submission our 80/20 row trailed DAPO, unlike in its paper. We retrained the mask with its authors' released training script. It now scores 82.7 / 30.2 / 23.6: level with DAPO on MATH500 and above it on both AIME sets. CreditFilter stays ahead of it by 4.9 / 3.6 / 4.1 points (Tables 1, 2 and 6).

5. Exponential versus top-20% weighting (oTEf). Reviewer oTEf asks what the exponential window adds over a top-20% cut within each response. We compared the two over three seeds (Section 5.4 and Table 22). In every seed the exponential window finishes more of its responses, generates fewer tokens and trains on fewer tokens. It leaves 1.2 / 4.6 / 1.7% of its samples unanswered at the length cap, against 6.4 / 12.1 / 9.9%, and it generates 17–28% fewer tokens at evaluation.

Each reviewer's own points are answered in the reply under their review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.