- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to distribute a response-level reward more appropriately across the tokens generated by a model. Tokens are the basic units used by the model to generate text. Existing methods typically assign the same advantage signal to all tokens in a response, even though different positions may have very different effects on the final answer.
The authors use “forking value” to measure the importance of a position: how much would the probability of a correct final answer differ if different next tokens were chosen at that position? Measuring this effect directly requires many continuations. The authors therefore use the more easily computed “fork probability” as a proxy and examine its relationship with uncertainty rank within a response.
CreditFilter ranks tokens by uncertainty within each response and gradually reduces their training weights as the rank moves down. It also includes a minimum weight for correct responses, reduced reinforcement of successful responses on easy problems, and a fallback to uniform weights when overall uncertainty is too high. The dynamic variant, CreditFilter-D, updates the rate of weight decay during training.
The main experiments use Qwen3-8B-Base and DAPO-17k. In Table 1, CreditFilter achieves late-checkpoint average accuracies of 87.6%, 33.8%, and 27.7% on MATH500, AIME24, and AIME25, respectively, compared with DAPO's 83.0%, 24.4%, and 17.7%. The paper also provides component ablations, measurements across several models, and additional experiments on Qwen3-1.7B.
Critical strengths
-
Ranking tokens within each response is a simple improvement supported by experimental evidence.
Under the CreditFilter configuration, switching to batch-level ranking reduces accuracy by 5.0, 7.4, and 8.3 percentage points on the three benchmarks. In another comparison, changing the original 80/20 method to select the most uncertain 20% of tokens within each response improves accuracy by 8.4, 12.9, and 11.4 percentage points, respectively (Table 2, p. 8; Table 5, p. 15). These two comparisons support the value of within-response ranking from different angles and form one of the paper's more convincing contributions.
-
The authors carefully measure how choices at different positions affect the final answer and explain the limitations of those measurements.
The authors try different tokens at the same position and generate repeated continuations to estimate their effects on the probability of answering correctly. In addition to measurements restricted to a small set of candidates, they design a method that includes the remaining candidates, run approximately 3.58 million continuations, and report uncertainty intervals (Appendix E.4, Tables 13–14, pp. 25–27). This provides empirical support for the method. The authors also discuss omitted candidates and continuation-length limits, helping readers assess the scope of the evidence.
Critical weaknesses
-
A gap remains between what the theory establishes and the weights used in practice; the additional value of exponential weighting is also unclear.
Proposition 1 states that, for a given problem, averaging the total forking value across all possible responses yields a fixed quantity. It does not state that every individual response has the same total, so it does not directly imply that each response should receive the same mean weight. Appendix D.5 acknowledges this distinction (p. 19).
Within-response rank also depends on later generated content and response length. The theory assumes that, under certain conditions, this later information does not further change the expected credit at the current position, but this assumption is not tested (Assumption 2, p. 5). Approximating actual importance using the more easily computed fork probability also omits factors related to success probability and fork consequence. The paper does not sufficiently validate these approximations in actual training, and settings such as length penalties fall outside the relevant theoretical results.
The experiments likewise do not clearly establish that exponential weighting is better than the simpler within-response top-20% method. Under the same training recipe, the top-20% window reaches 38.0% and 28.0% on AIME24 and AIME25, compared with CreditFilter's 33.8% and 27.7% (Table 2). Exponential weighting performs better on MATH500, but it remains difficult to determine how much additional benefit comes from measuring and calibrating the exponential curve. This affects my assessment of the size of the methodological contribution.
-
More evidence is needed to establish whether the main results are stable and whether the method performs better at comparable computational cost.
The main comparisons do not report multiple independent training runs. Resampling test problems and averaging results from three neighboring checkpoints cannot replace retraining with different random seeds. Each AIME benchmark also contains only 30 problems, so the stability of the gains should be assessed cautiously.
The configurations are not fully matched across methods. CreditFilter includes additional stabilization mechanisms, while some baselines differ in clipping parameters, activation conditions, or input templates. Appendix C discloses these differences, but they still affect the claim of consistently outperforming prior methods (pp. 14–15).
Equal training steps do not necessarily mean equal training cost, because methods may generate different numbers of responses with different lengths. The paper does not report complete training-token totals or GPU time. At evaluation, CreditFilter's average output length is approximately 1.8–3.1 times that of GRPO/DAPO (Table 15). These outputs include text generated after the answer, so the difference cannot simply be treated as longer reasoning. Nevertheless, comparisons under matched inference budgets are still needed. The second-model experiments are a useful addition, but each configuration has only one run, and accuracy gains over the untrained model are small. They therefore do not fully resolve these concerns (Table 16, p. 28).
Questions for authors
-
Could you further clarify whether the theoretical assumptions hold in actual training?
In particular, could subsequent response length and uncertainty ranking affect the credit estimate at the current position? How much do the omitted factors matter when fork probability is used in place of actual importance? Also, why does the variance result that holds on average support the per-response budget design? If these connections cannot currently be validated, I suggest presenting the corresponding claims as design motivation.
-
What is the main advantage of exponential weighting over the simpler within-response top-20% method?
With damping, fallback, clipping, and mean weight matched, could you report uncertainty intervals for the performance differences and results from multiple independent training runs? In Table 2, the top-20% method performs better on both AIME benchmarks. Please clarify whether exponential calibration mainly benefits accuracy, training stability, or another aspect.
-
Do the main gains persist at comparable computational cost and across multiple independent training runs?
Please provide training-token totals, GPU time, and results with different random seeds for CreditFilter, DAPO, and the within-response top-20% method, together with comparisons under matched inference-token budgets. For the STARE and SC-GRPO configuration differences described in Appendix C, please also explain how much these differences might affect the comparisons.
5 comments
W2 & Q3 (matched budgets): Do the main gains persist at comparable training cost and under matched inference-token budgets?
A3: We ran three comparisons. CreditFilter leads at equal training tokens and at equal tokens spent. At matched decoding budgets it leads from 8k tokens upward and trails at 4k.
Equal training tokens (Table 21). We continued the DAPO run beyond step 400. More training does not raise its accuracy. It scores 83.7 / 25.9 / 19.5 at step 400. At its five checkpoints from step 500 to 600 it scores between 82.9 and 84.0 / 25.6 and 27.5 / 18.8 and 21.0. By step 600 it has trained on 1.56 times its tokens at step 400. At 4.2 billion training tokens, CreditFilter's step-300 checkpoint scores 85.0 / 29.9 / 25.0 and DAPO's step-575 checkpoint 82.9 / 25.6 / 20.4.
Matched decoding budget (Table 20). We cut each response at a token budget and grade it as in the paper. The numbers are late averages over steps 350–400.
| Decoding budget | CreditFilter | DAPO | GRPO |
|---|---|---|---|
| 4k | 77.9 / 16.8 / 17.6 | 82.7 / 24.2 / 17.9 | 80.6 / 20.8 / 16.8 |
| 8k | 86.7 / 28.6 / 25.8 | 82.8 / 24.3 / 17.9 | 83.2 / 25.3 / 19.3 |
| 12k | 87.5 / 32.6 / 27.4 | 82.8 / 24.3 / 17.9 | 83.2 / 25.5 / 19.4 |
| 20,480 | 87.6 / 33.4 / 27.7 | 82.8 / 24.3 / 17.9 | 83.2 / 25.5 / 19.4 |
At 4k CreditFilter trails, because about 70 percent of its AIME responses have not yet written an answer by then. The table samples the responses of the same checkpoints anew, so its full-length row differs from Table 1 by at most 1 point.
Equal tokens spent (Appendix F.1). A majority vote over three DAPO samples spends more tokens than one CreditFilter response. It reaches 85.8 / 26.4 / 19.0, and a vote over eight samples reaches 87.4 / 30.3 / 21.2. One CreditFilter sample scores 87.6 / 33.8 / 27.7 (Table 1).
W2 & Q3 (matched configurations): CreditFilter includes additional stabilization mechanisms, and some baselines differ in clipping, activation conditions or templates. How much might the STARE and SC-GRPO differences affect the comparisons?
A4: The stabilizers do not produce the gain without the window. Two rows of Table 2 separate the two.
- Within-response ranking without the stabilizers. The within-response cut on the settings of the DAPO row uses DAPO's clip, and neither damping nor the fallback. This is the cleanest matched comparison. Against DAPO it loses 0.5 points on MATH500 and gains 7.8 and 8.6 points on the two AIME sets.
- The stabilizers without the window. Damping alone, with uniform token weights, scores 71.3 / 16.8 / 13.8, which is below DAPO.
The window and the stabilizers are complementary (Section 5.4). STARE and SC-GRPO ran with their published settings. Appendix C (Table 6) lists how each run differs from its publication.
W1 & Q1: A gap remains between what the theory establishes and the weights used in practice. Proposition 1 holds on average rather than for each response, Assumption 2 is not tested, the fork probability omits factors, and the length penalty falls outside the theory. If these connections cannot be validated, please present the claims as design motivation.
A5: The revision presents the per-response design and the window as design motivation (abstract, Sections 1, 3.1 and 4). Our tests are on the base policy, so the same holds for the policies seen during training. We take your four points in turn.
- Proposition 1. It fixes the total forking value in expectation over a prompt's responses, not for each response. The revision says so, and presents ranking within each response as a choice that the proposition motivates.
- Assumption 2. We tested it with the regression that Reviewer RCZq proposed (Appendix E.5, Table 16; the table is also in our reply there). The part that the assumption rules out is within sampling error at the 95% level. The exception is the top 0.25% of the rank in correct responses, where the realized increment is about 1.5 times what Proposition 3 gives.
- Omitted factors. The value factor barely changes the target's shape over the rank. With and without it, the spectra have squared cosines of 0.97 in correct and 0.98 in wrong responses (Appendix E.6). The fork consequence is the main approximation. Over the top 20% of the rank, where it can be estimated, it is flat apart from a spike in the top 2.5%. The spike is about six times its typical value in the top 0.25%, where the window already puts its largest weight (Appendix E.4).
- Length penalty. The propositions take the reward to be the 0-1 outcome, without the overlong penalty. Section 4 and the limitations now say so.
Q2: What is the main advantage of exponential weighting over the simpler within-response top-20% method: accuracy, training stability, or another aspect?
A6: In every seed the exponential window finishes more of its responses, generates fewer tokens and trains on fewer tokens. Both windows use the same damping, fallback, clip and class mass. The table gives late averages over the three seeds, exponential / top-20% (Table 22 of the revision, with standard deviations; per-seed accuracies in Table 24).
| MATH500 | AIME24 | AIME25 | |
|---|---|---|---|
| Accuracy (%) | 86.0 / 81.7 | 32.6 / 34.7 | 25.8 / 26.1 |
| Mean generated tokens | 6,541 / 7,894 | 8,611 / 11,730 | 7,491 / 10,428 |
| Unanswered at the cap (%) | 1.2 / 6.4 | 4.6 / 12.1 | 1.7 / 9.9 |
- Fewer tokens. The exponential window generates 17–28% fewer tokens at evaluation. It trains on 15% fewer, 5.57 against 6.58 billion on average.
- More answers within a budget. On the AIME sets we count only the responses that end within 8k tokens. The exponential window then gives more correct answers in every seed, 27.7 / 22.3 against 22.9 / 19.5 on AIME24 / AIME25. Within 12k tokens the two windows are level, 30.5 / 24.1 against 29.3 / 24.0.
- The width. What the measurement supplies is the width. Fixed at twice the measured value, the window loses 9 to 15 points (Table 2). CreditFilter-D re-measures the width during training and matches CreditFilter (Section 5.4).
We hope these results address your concerns and are sufficient to reconsider the assessment. We are happy to run any further check during the discussion period.
Dear Reviewer oTEf, thank you for your detailed review and for recognizing that the experiments support ranking tokens within each response, and the forking-value measurements. We answer in three comments: (1) the independent training runs and the cost, (2) the matched budgets and the matched configurations, (3) the theory and the comparison with the top-20% window. Every change we mention is in the revised PDF, marked in blue.
W2 & Q3 (independent runs): The main comparisons do not report multiple independent training runs. Please provide results with different random seeds for CreditFilter, DAPO and the within-response top-20% method.
A1: We trained two more data seeds for each of the three methods (Appendix F.2, Table 24). The table gives late averages on MATH500 / AIME24 / AIME25.
Over the three seeds, the standard deviation of CreditFilter's late averages is 2.5 / 4.2 / 3.5 points, and DAPO's is 10.7 / 5.0 / 3.2. On both AIME sets, each of the six runs with a within-response window is above each of the three DAPO runs. On every set, CreditFilter's margin over DAPO in Table 1 is the middle one of the three seeds.
Paired by seed, CreditFilter minus DAPO is +7.7 ± 12.2 / +10.5 ± 8.8 / +9.1 ± 6.2 points. CreditFilter minus the top-20% window is +4.3 ± 5.9 / $-$2.1 ± 1.7 / $-$0.3 ± 1.1 points. These are means ± standard deviations over the three seeds, computed before rounding. Our third comment compares the two windows (Q2).
W2 & Q3 (cost): Equal training steps do not mean equal training cost. Please provide training-token totals and GPU time. The reported output lengths include text generated after the answer.
A2: Training cost (Appendix F.1, Table 19). Over 400 steps CreditFilter trained on 6.04 billion tokens in about 145 hours, and DAPO on 2.87 billion in about 70, both on 16 H100 GPUs. The top-20% window trained on 6.88 billion tokens, which is about 165 hours at the same time per token. Every step trains on the same number of prompts and responses, in the same number of updates. The extra time therefore comes from longer responses.
Length up to the answer (Table 23). The lengths in the submission's Table 15 (Table 17 in the revision) include text written after the answer. The revision also counts each response only up to where it first states its final answer. By this count CreditFilter writes 2,556 / 7,028 / 6,607 tokens on MATH500 / AIME24 / AIME25, DAPO 905 / 1,969 / 1,549 and GRPO 1,696 / 4,080 / 3,300. CreditFilter's reasoning up to the answer is 2.8 to 4.3 times DAPO's and 1.5 to 2.0 times GRPO's.