acceptodds
Under review as a conference paper at ICLR 2027

DTCA: Denoising Token Credit Assignment for Reinforcement Learning in Diffusion Language Models

Abstract

Diffusion language models (DLMs) generate text through iterative denoising, enabling bidirectional context modeling and parallel token refinement. Recent studies have applied reinforcement learning (RL) to DLMs and improved their reasoning performance. However, the iterative denoising process makes credit assignment more challenging, as final outcome rewards provide limited information about how different tokens contribute to the final output. In this paper, we propose Denoising Token Credit Assignment (DTCA), a framework for token-level credit assignment in reinforcement learning for diffusion language models. DTCA estimates token contributions by measuring the improvement in prediction confidence of remaining masked positions after token commitment and groups these signals according to part-of-speech (POS) categories. These contribution weights are used to select informative denoising timesteps and assign token-level learning signals during RL training. DTCA further calibrates these learning signals according to token-level entropy changes induced by policy updates. Experiments on LLaDA-8B-Instruct show that DTCA consistently improves mathematical reasoning performance across different generation budgets, with gains of up to 9.0 points on MATH500 and 5.3 points on GSM8K over the base model.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Diffusion Large Language Models, Discrete Diffusion Models, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_NH9iOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Denoising Token Credit Assignment (DTCA), a framework for reinforcement learning in diffusion language models. The method addresses the difficulty of assigning sequence-level rewards to intermediate denoising decisions by estimating token contributions from confidence improvements of remaining masked positions after token commitment. These contribution signals are aggregated through POS categories and used for denoising timestep selection and token-level credit assignment. The paper further introduces an entropy-based calibration mechanism to adjust learning signals during RL optimization. Experiments on LLaDA-8B-Instruct across mathematical reasoning benchmarks (MATH500 and GSM8K) show consistent improvements over existing diffusion RL baselines.

Critical strengths

The paper studies an important and relatively under-explored problem in diffusion language model reinforcement learning. Compared with approaches relying only on sequence-level rewards, DTCA attempts to exploit the internal denoising process and provides a more fine-grained perspective for credit assignment.

The proposed framework is evaluated with multiple baselines and component ablations. The experimental results show consistent gains across different generation budgets, suggesting that denoising dynamics contain useful information for RL optimization.

Critical weaknesses

The main concern is that the effectiveness of the proposed token contribution estimation is not sufficiently validated. DTCA assumes that the confidence improvement of remaining masked positions after token commitment is a reliable indicator of token importance, but it remains unclear whether this signal corresponds to the actual causal contribution of tokens to final generation quality.

Furthermore, the contribution of individual components is not fully isolated. The current ablation study evaluates removing major modules, but does not clearly separate the effects of contribution estimation, POS aggregation, timestep selection, and token-level credit assignment. Therefore, it is difficult to determine which component is primarily responsible for the reported improvements.

Questions for authors
  1. The proposed contribution score is used as a proxy for token importance. Have the authors conducted any causal validation, such as token perturbation, masking, or intervention experiments, to verify that tokens with higher contribution scores actually have greater influence on final generation quality?

  2. How stable are the POS-level contribution weights across different datasets, rollout samples, and training stages? Does the contribution table need to be recomputed when applying DTCA to a new task or model?

  3. Could the authors provide a more detailed analysis of the computational overhead introduced by offline contribution estimation, including additional forward passes and total training cost compared with the baseline RL method?

3 comments

Thank you for your careful review and helpful suggestions. We address your comments below.

W1 / Q1: Have token perturbation, masking, or other intervention experiments been used to verify that tokens with higher contribution scores have a greater effect on final generation quality?

A: Thank you for this suggestion. We added a random token replacement experiment. We use 64 questions that LLaDA-8B-Instruct originally answered correctly and select early and middle denoising states. In each trial, we replace the token with the highest contribution score in that step with a random token, then continue generation. All other committed tokens and decoding settings remain unchanged. We run 256 replacement trials in total.

SettingAccuracy (%)Prediction probability of correct answer tokens (%)
No perturbation100.0097.65
Random replacement94.1494.16

After replacement, final answer accuracy drops by 5.86 percentage points, and the mean prediction probability of correct answer tokens before commitment drops by 3.49 percentage points. The mean number of tokens committed in the next step also drops from 2.69 to 1.87. These results show that changing a high-contribution token affects later denoising and the final answer.

W2: The effects of contribution estimation, part-of-speech (POS) aggregation, denoising step selection, and token-level credit assignment are not fully separated.

A: Following your suggestion, we added two controls to the existing ablations: replacing contribution scores with token prediction confidence, and selecting denoising steps based on low confidence. The second control keeps contribution-based token-level credit assignment.

All configurations use the checkpoint at update 500. Other training settings follow Appendix C.2. The generation length and number of denoising steps are both 256. Other evaluation settings follow Appendix C.4.

ConfigurationMATH500 (%)GSM8K (%)
Full DTCA43.285.3
Replace contribution scores with confidence38.883.1
Select denoising steps based on low confidence40.282.5
Without token-level credit assignment39.482.9
Without entropy calibration41.883.9

The two added controls achieve lower accuracy than full DTCA, supporting contribution estimation and contribution-based denoising step selection, respectively. Removing token-level credit assignment or entropy calibration also lowers accuracy, showing the benefit of both components. The settings and results are reported in Appendix D.1, Table 2.

Q2: How stable are the POS weights across datasets, rollout samples, and training stages? Does the table need to be rebuilt for a new task or model?

A: Our results show relatively similar contribution rankings for the main POS categories across math datasets. Weights and rankings are also generally stable across rollout samples. The difference between math and code is larger, so we rebuild the table using data from a new target task. On the same dataset, the two models have relatively similar rankings. When changing models, the table can be rebuilt as needed to adjust the weights.

The table below compares datasets, tasks, and models. Each setting uses 200 questions with 4 responses per question. The dataset and task comparisons use LLaDA-8B-Instruct. The model comparison uses LLaDA and Dream-v0-Instruct-7B on the same GSM8K questions.

ComparisonSettingsRank correlationRank correlation (Figure 1, 8 categories)
Across datasetsGSM8K / MATH (LLaDA)0.6240.881
Across tasksGSM8K / MBPP (LLaDA)0.4140.167
Across modelsLLaDA / Dream (GSM8K)0.7970.810

Differences across tasks. Among the 8 main POS categories in Figure 1, the ADP group moves from 6th on GSM8K to 1st on MBPP. On MBPP, this group mainly contains in, for, and from, with in and for making up 90.1%. These words are often used in loops, membership checks, and imports. Their roles differ from those of prepositions in math responses.

See Appendix D.4 for these analyses, Table 5 for POS rankings, and Figure 11 for the MBPP analysis.

Rollout samples. Using an earlier set of rollouts from 1,000 MATH questions, we randomly sample 200 questions and keep all 8 responses per question. We repeat this 20 times to build POS weight tables. Compared with the table built from all questions, the mean weight difference across all 18 POS categories is 0.0257, and the mean rank correlation is 0.971. The weights and rankings are therefore generally stable across these samples.

Our response continues in Part 2, where we discuss why we use a fixed table during training and the computational cost.

Why we use a fixed table during training. During RL rollouts, multiple tokens are committed in the same step, making it difficult to separate each token's effect on prediction confidence at the remaining masked positions. Replaying states for individual tokens is costly. We therefore estimate token contributions before training and average them within each POS category to build a reusable POS weight table. During training, tokens receive the weights of their POS categories, and the table stays fixed. Appendix C.3 describes how the fixed table is used. Your suggestion motivates us to further explore how to efficiently update the POS table during training to better reflect changes in the model. Thank you for this valuable suggestion.

Q3: How many extra forward passes does offline contribution estimation require, and what is the total training cost compared with the baseline?

A: We use 500 questions from each of the MATH and GSM8K training sets, with 4 responses per question, to build the POS weight table once before training. The table is reused during training. At a generation length of 256, each response requires 128 forward passes for generation and 127×3=381127\times3=381 additional forward passes for contribution estimation, for a total of 509. No gradients are computed.

On 4 NVIDIA B200 GPUs with the same settings, 500 updates take 18.54 hours for DTCA and 17.95 hours for RLDF. DTCA's training time is about 3.3% higher.

Thank you for your response. I would be happy to see this work accepted.

Reviewer_KD9QOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper considers token-level credit assignment for RL fine-tuning of diffusion language models. At each denoising step, multiple tokens are committed at the same time, so it is difficult to tell how much each token contributes to refining the remaining masked tokens. Measuring each token’s contribution requires additional forward passes, which would be expensive to do during RL training.

To mitigate the costs, the authors estimate token contributions before RL training. They measure how much committing an individual types of token(part-of-speech category such as nouns and verbs) improves the model’s confidence in the remaining masked tokens, then average these estimates through timesteps. During RL training, they use these category-level weights to select denoising steps and assign different amounts of credit to tokens. DTCA is evaluated on mathematical reasoning tasks using LLaDA-8B-Instruct.

Critical strengths
  1. The paper addresses token-level credit assignment in diffusion language models, an interesting and underexplored problem.

  2. the main components are evaluated separately through ablations.

Critical weaknesses
  1. The paper feels more like an extension of RLDF than a standalone work. It stays closely within RLDF’s experimental setup instead of evaluating DTCA more broadly as an RL method for diffusion language models.

  2. The paper does not compare DTCA with EGSPO[1], which also studies denoising-step selection and stepwise advantages. This comparison could help clarify how DTCA is different.

  3. The experiments only evaluate mathematical reasoning on MATH500 and GSM8K. Sudoku and Countdown are not included, and it is unclear whether DTCA would also work for code or other logical tasks with different output structures.

  4. It is unclear whether the comparison is fair. The baselines use two-token decoding at each step, while DTCA uses block-wise decoding with a dynamic confidence threshold. Could the authors clarify whether this is the case?

  5. DTCA uses extra forward passes to estimate token contributions before RL training, but the paper does not report how much computation this takes or the total overhead compared with RLDF.

[1] Kunde et al. (2026), Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages, arXiv:2603.12554.

Questions for authors

In addition to the issues raised in the weaknesses section, I have the following questions:

  1. How did you choose the range [0.8,1.2][0.8, 1.2] for normalizing the POS contribution weights? Did you test how sensitive the results are to this choice?

  2. How do you ensure that an increase in confidence reflects better predictions, rather than making the model more confident in an incorrect prediction?

  3. As shown in Fig.1 , the effect of each POS category may vary across denoising steps, but DTCA uses one fixed contribution table for all steps. Did you test whether using step-specific contribution weights would affect the results? Is there evidence that the fixed table does not harm performance?

6 comments

Thank you for your careful review and helpful suggestions. We address your comments below.

W1: DTCA seems like an extension of RLDF and lacks broader evaluation.

A: DTCA introduces a method for measuring token contribution during denoising in diffusion language models (DLMs). RLDF focuses on selecting denoising steps for training. DTCA further uses contribution estimates to distinguish the roles of tokens within the same step and assign them different training weights.

To address your suggestion for broader evaluation, we added code generation experiments. We train on the MBPP training set and evaluate on MBPP and HumanEval. The settings and results are provided in our response to W3.

W2: A comparison with EGSPO is missing, and the differences between the methods need to be explained.

A: EGSPO selects denoising steps for training based on entropy, and its EGSPO-SA variant also uses step-level advantages. DTCA estimates token contributions from confidence changes at the remaining masked positions after token commitment, then averages these estimates within each part-of-speech (POS) category for denoising step selection and token-level credit assignment within the same step.

Following your suggestion, we retrained EGSPO using the same training and evaluation settings as DTCA and added the results and citation to the paper. DTCA achieves higher accuracy on both datasets at all three generation lengths. At a generation length of 256, the gains are 5.4 percentage points on MATH500 and 1.3 on GSM8K.

Generation lengthEGSPO: MATH500 / GSM8K (%)DTCA: MATH500 / GSM8K (%)Gain (percentage points)
12834.8 / 78.639.8 / 80.65.0 / 2.0
25637.8 / 84.043.2 / 85.35.4 / 1.3
51237.2 / 84.144.0 / 85.46.8 / 1.3

W3: Evaluation covers only math tasks, without Sudoku, Countdown, or code generation.

A: Thank you for suggesting broader evaluation. We added code generation experiments, training on the MBPP training set and evaluating on MBPP and HumanEval. The results below use the DTCA checkpoint after 500 training steps and provide initial evidence of its potential for code generation.

DatasetGeneration length 128 (%)Generation length 256 (%)Generation length 512 (%)
MBPP38.244.242.0
HumanEval39.038.437.8

Code has different syntax and structure from natural language. Code elements such as function names and control flow may provide a better basis for grouping tokens than POS categories. We will explore these grouping methods in future work to better capture the roles of tokens in code.

W4: Do the baselines and DTCA use the same decoding settings, and is the comparison fair?

A: In the main experiments, DTCA and all baselines use the same block-wise decoding and dynamic confidence threshold. This threshold determines which tokens are committed at each step. The shared settings in Appendix C.4 are listed below.

ParameterShared setting
Block size32
Dynamic confidence threshold0.9
Temperature0
Responses per question1
Maximum generation length128, 256, 512
Configured denoising stepsSame as the maximum generation length
Prompt templateSame across methods

Committing two tokens per step is used for offline contribution estimation before training. In this process, we commit each token separately to measure its effect on later predictions and build the POS weight table. We will clarify the settings for offline table construction, RL training, and final evaluation separately in the main text.

W5: How much extra computation does offline contribution estimation require, and what is the total overhead compared with RLDF?

A: The POS weight table is built once before training and reused during training. We use 1,000 training questions with 4 responses per question. At a generation length of 256, each response requires 128 forward passes for generation and 127×3=381127\times3=381 additional forward passes for contribution estimation. No gradients are computed.

Under the same training settings on 4 NVIDIA B200 GPUs, 500 updates take 18.54 hours for DTCA and 17.95 hours for RLDF. DTCA's training time is about 3.3% higher.

Q1: How was the POS weight range [0.8, 1.2] chosen, and how sensitive are the results to this range?

A: Our choice of [0.8, 1.2] was inspired by common RL settings, such as PPO's clipping coefficient of 0.2.

To address your question about sensitivity, we added a comparison of three ranges. All runs use the same model and data, 500 training steps, and a generation length of 256.

Weight rangeMATH500 (%)GSM8K (%)
[0.9, 1.1]39.483.3
[0.8, 1.2]43.285.3
[0.7, 1.3]40.285.4

The narrower range lowers MATH500 and GSM8K accuracy by 3.8 and 2.0 percentage points. The wider range improves GSM8K by 0.1 percentage points but lowers MATH500 by 3.0 percentage points. Based on both results, we keep [0.8, 1.2]. These results are reported in Appendix D.1, Table 3.

Q2: Does increased confidence reflect better predictions, or greater confidence in incorrect predictions?

A: Higher confidence can also occur for incorrect predictions. Contribution weights identify tokens with a larger effect on later denoising. The task reward determines which predictions to encourage or discourage based on answer correctness. Contribution weights strengthen learning at these key positions.

Q3: Have fixed weights been compared with weights estimated separately for each denoising step? Does the fixed table affect performance?

A: At each selected denoising step, DTCA divides the weights of the tokens used for training by their mean within that step. As shown in Figure 1(a), POS categories have different token commitment patterns across denoising stages. As the mix of tokens used for training changes, the normalized training weight of the same POS category also changes. Thus, token-level credit assignment reflects differences across denoising stages.

I thank the authors for their detailed response. I address only the points that remain unresolved below.

W2: Why are the original EGSPO results omitted? My understanding is that EGSPO uses two-token decoding. If you use the same settings, you should report the original numbers and acknowledge where DTCA performs worse. Otherwise, specify exactly what you ran and how you selected the checkpoint. With such a high confidence threshold, I would expect decoding to move toward one token per step, potentially improving performance. This substantial drop therefore needs an explanation. Reporting lower reproduced results without explaining the discrepancy is disappointing.

W3: How are POS categories defined and assigned to code tokens? Without explaining how the contribution table was constructed and applied to code, it is unclear how these results were obtained. Please also include the original results from d1, ESPO, and EGSPO as baselines. ESPO may be closer to your setting, as it uses one-token training and decoding, whereas d1 and EGSPO report results with two-token decoding.

W4: With a confidence threshold of 0.9, I suspect many steps commit only one token. Please report the actual generation lengths, number of denoising steps, and average tokens committed per step for DTCA and each baseline.

Reviewer_5JoKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper tackles the problem of reinforcement learning (RL) for masked diffusion language models (DLMs). The authors propose reweighting the advantage function for each token based on the part of speech (POS) for each token. Their intuition for doing this is that different tokens affect how the next tokens are unmasked, they would have liked to reweight the advantage based on this interaction, however, since computing this during online RL is expensive they compute these for offline rollouts and average them by POS so that they can be leveraged as surrogates for the online RL. Furthermore, these weights are used to select which timesteps to perform RL over. Finally they follow OPEFO to introduce entropy based credit calibration to balance entropy increasing/decreasing updates. The experimental results on MATH and GSM8K demonstrate strong empirical performance.

Critical strengths
  • The paper proposes a relatively simple change to improve RL for DLMs which produces significant improvements in performance
  • The paper performs ablations to demonstrate how the different moving pieces of its method interact with each other
  • The paper produces some interesting insights of how models optimized with DTCA compare to other methods
Critical weaknesses
  • The paper doesn't explain how some of the choices are made or how they impact the following results. This is most notable in line 206, where the weighting wcw_c is defined with some magical constants 0.8 and 0.4.
  • Although the experimental results are very compelling, the method itself doesn't carry much novelty. The method is a heuristic re-weighting of the advantage and a carry over from an existing method.
  • From the ablation results, it seems like most of the gains come from the entropy calibration, and not from the contribution credit. This can be seen for example in Figure 3, where (for example) the reward collapses for DTCA w/o Entropy Calibration, while w/o Contribution credit continues to grow, although the plot is cut short. This suggests that the contribution credit may not be key to producing optimal results, which further weakens the method novelty.
  • Although ablations are present, all the work in the paper seems to be limited to up to 500 training steps. However it remains unclear if the reported results are at step 500, furthermore, a better ablation would continue training for more than just 500 steps which is usually the case in practice.
Questions for authors
  • Can you explain the choice of constants 0.8 and 0.4
  • Could you provide more results on what happens when training for longer for all 4 possible configurations (base, DTCA, w/o entropy, w/o calibration) to better understand which component is truly fundamental
5 comments

Thank you for your careful review and helpful suggestions. We address your comments below.

W1 / Q1: How were the constants 0.8 and 0.4 chosen, and how do they affect the results?

A: Our choice of [0.8, 1.2] was inspired by common RL settings, such as PPO's clipping coefficient of 0.2.

To address your question about sensitivity, we added a comparison of three ranges. All runs use the same model and data, 500 training steps, and a generation length of 256.

Weight rangeMATH500 (%)GSM8K (%)
[0.9, 1.1]39.483.3
[0.8, 1.2]43.285.3
[0.7, 1.3]40.285.4

The narrower range lowers MATH500 and GSM8K accuracy by 3.8 and 2.0 percentage points. The wider range improves GSM8K by 0.1 percentage points but lowers MATH500 by 3.0 percentage points. Based on both results, we keep [0.8, 1.2]. These results are reported in Appendix D.1, Table 3.

W2: The method is a heuristic reweighting of the advantage function and builds on existing methods, with limited novelty.

A: Thank you for raising this point. Our core contribution is a method for measuring token contribution during denoising in diffusion language models (DLMs) and using this estimate in RL training. This has two parts:

  1. Measuring token contribution to later denoising. DLMs commit multiple tokens in parallel at each step. A sequence-level reward, shared by the whole response, does not capture the different effects of these tokens on later denoising. We estimate a token's contribution by comparing prediction confidence at the remaining masked positions before and after the token is committed.
  2. Using contribution estimates in training. We estimate token contributions before training and average them within each part-of-speech (POS) category to build a POS weight table. During training, this table is used for denoising step selection and to adjust the training weights of tokens within the same step, enabling token-level credit assignment.

W3 / W4 / Q2: Are the gains mainly due to entropy calibration? Can the variant without token-level credit assignment catch up after more than 500 training steps?

A: Following your suggestion, we trained full DTCA and two ablation variants from scratch for 600 steps and compare them with the original results at step 500. The generation length is 256 for all runs.

Training stepConfigurationMATH500 (%)GSM8K (%)
500Full DTCA43.285.3
500Without token-level credit assignment39.482.9
500Without entropy calibration41.883.9
600Full DTCA42.485.3
600Without token-level credit assignment40.283.2
600Without entropy calibration38.882.6

At step 500, removing token-level credit assignment lowers MATH500 and GSM8K accuracy by 3.8 and 2.4 percentage points. Removing entropy calibration lowers accuracy by 1.4 percentage points on both datasets. At step 600, full DTCA outperforms the variant without token-level credit assignment by 2.2 and 2.1 percentage points. Full DTCA trained for 500 steps also achieves higher accuracy than this variant trained for 600 steps.

W4: Are the main-table results from step 500, and how are checkpoints selected?

A: Yes. The results for DTCA, its two ablation variants, and RLDF in Table 1 use the checkpoints saved after 500 updates. For each method, the same 500-step checkpoint is evaluated on MATH500 and GSM8K at generation lengths of 128, 256, and 512. Training settings are described in Appendix C.2, and the shared evaluation settings are described in Appendix C.4.

Dear authors, thanks for taking the time to address my questions.

I appreciate the new experiments, but I must add that just an extra 100 steps is not enough for a proper ablation, on how more training time interacts with the entropy calibration. The papers scope continues to be limited, by the number of training steps, most RL methods will perform much more than 500 steps and there is currently no evidence that the proposed method is worth pursuing until this becomes clearer.

Thank you for carefully reviewing our additional results and for your quick response. Our experiments show that token-level credit assignment helps achieve higher accuracy with fewer training steps. After 500 training steps, full DTCA achieves 43.2% on MATH500 and 85.3% on GSM8K. DTCA w/o Contribution Credit reaches 40.2% and 83.2% after 600 steps. Even if this variant reaches similar accuracy with longer training, using fewer training steps still has practical value.

Entropy calibration alone adjusts the credit weights of entropy-increasing and entropy-decreasing tokens by the same proportion in opposite directions. Token-level credit assignment gives different credit weights to tokens within each group. These weights reflect each token's contribution to subsequent denoising. Together, token-level credit assignment focuses learning on key positions, while entropy calibration balances the entropy changes caused by these updates to help keep training stable.

Thank you again for your valuable suggestion to extend training. Longer runs would help us better understand how both components affect training dynamics and final performance. Time and computing resources were limited during this discussion period, so we ran the additional experiments for 600 training steps. We would be happy to answer any further questions about this explanation or our earlier responses. We would also be grateful if you could reconsider your assessment and score based on these results.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Clarification in Section 3.1: Token Contribution Estimation

While checking the manuscript, we found that Equation (3) in Section 3.1 mistakenly included the normalization factor $ 1/|\mathcal{M}_t|$. The text also described the sum of probabilities as an average. All experiments used the sum to measure the total change in prediction confidence at the remaining masked positions after a token is committed.This is consistent with Figure 1(b), where the reported “Confidence gain” values exceed 1.

We have removed this factor in the revised manuscript and changed "average probability" to "sum of probabilities" in the text. This correction makes the description match the actual computation. The POS weight table, training procedure, and reported results remain unchanged. We apologize for this writing error.

For ease of review, the corresponding changes in Section 3.1 are marked in red in the revised manuscript.

AuthorsOpenReview ↗

Rebuttal Summary

Rebuttal Summary

We sincerely thank the area chair and all reviewers for their careful reviews and the time and effort they have devoted to this paper. We summarize the main contribution, the key concerns discussed during the rebuttal, and the additional evidence below.

DTCA estimates token contributions during denoising to guide token-level credit assignment in reinforcement learning for diffusion language models (DLMs). We observe that part-of-speech (POS) categories differ in commitment timing and their effects on subsequent prediction confidence. DTCA estimates contributions from changes in prediction confidence at the remaining masked positions before and after token commitment, then averages them within each POS category to build a reusable POS contribution table. This table guides denoising timestep selection and token-level credit assignment during RL training. DTCA outperforms the base model at all three generation lengths, with accuracy gains of up to 9.0 and 5.3 percentage points on MATH500 and GSM8K, respectively.

The reviews mainly focus on the methodological contribution and empirical evidence. KD9Q describes token-level credit assignment as an “interesting and underexplored” problem. NH9i recognizes the finer-grained perspective offered by studying the internal denoising process, and 5JoK acknowledges the performance improvements. Their main concerns are:

  • 5JoK: Method novelty, the role of Contribution Credit, and performance with longer training.
  • KD9Q: Differences from existing methods, fairness of baseline comparisons, and applicability beyond mathematical reasoning.
  • NH9i: Validity of contribution estimation, the separate effects of the components, and stability of POS contribution weights.

We provide the following evidence and clarifications in response to these concerns.

  1. Core contribution and differences from existing methods (5JoK, KD9Q). In response to concerns about novelty and differences from existing methods, we clarified that DTCA estimates token contributions to subsequent denoising and averages them within each POS category. This design uses differences in the refinement effects of POS categories to obtain reusable contribution weights. These weights guide denoising timestep selection and token-level credit assignment within each selected step, connecting POS-level denoising behavior with RL training.

  2. Evidence for contribution estimation and the roles of individual components (5JoK, NH9i). Full DTCA outperforms the variants without Contribution Credit or Entropy Calibration after both 500 and 600 training steps. Full DTCA at 500 steps also exceeds the variant without Contribution Credit at 600 steps. Replacing high-contribution tokens affects subsequent denoising and lowers final-answer accuracy. These results support contribution estimation and show that token-level credit assignment helps achieve higher accuracy with fewer training steps. They also show that full DTCA remains effective in the experiments conducted so far.

  3. Baseline comparisons under matched settings and computational costs (KD9Q, 5JoK, NH9i). Following the reviewers' suggestions, we added an EGSPO comparison using the same training data and dynamic decoding settings, reported separately from the main experiments. DTCA performs better at lengths 256 and 512 and slightly worse at 128. Under the same hardware and training settings, its online training time is approximately 3.3% higher than RLDF's. Offline table construction costs are described separately in our follow-up responses. These additions provide a fuller comparison of performance and computational costs and help us better understand DTCA's potential.

  4. Stability of the POS contribution table and the scope of the fixed table (NH9i, KD9Q). POS contribution weights and rankings are generally stable across repeated samples. On the same selected denoising steps, DTCA's normalization within each denoising state brings credit weights from the fixed table close to those from stage-specific tables. We also added analyses across datasets, tasks, and models. Code generation experiments provide initial evidence of potential beyond mathematical reasoning and motivate exploring token categories better suited to code. These analyses support the use of the fixed table under the current settings and identify directions for further improvement on other tasks.

Finally, we sincerely thank the area chair and all reviewers once again for their thoughtful feedback, which has helped us improve the paper and better understand DTCA's potential. We welcome further comments and discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.