- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Denoising Token Credit Assignment (DTCA), a framework for reinforcement learning in diffusion language models. The method addresses the difficulty of assigning sequence-level rewards to intermediate denoising decisions by estimating token contributions from confidence improvements of remaining masked positions after token commitment. These contribution signals are aggregated through POS categories and used for denoising timestep selection and token-level credit assignment. The paper further introduces an entropy-based calibration mechanism to adjust learning signals during RL optimization. Experiments on LLaDA-8B-Instruct across mathematical reasoning benchmarks (MATH500 and GSM8K) show consistent improvements over existing diffusion RL baselines.
Critical strengths
The paper studies an important and relatively under-explored problem in diffusion language model reinforcement learning. Compared with approaches relying only on sequence-level rewards, DTCA attempts to exploit the internal denoising process and provides a more fine-grained perspective for credit assignment.
The proposed framework is evaluated with multiple baselines and component ablations. The experimental results show consistent gains across different generation budgets, suggesting that denoising dynamics contain useful information for RL optimization.
Critical weaknesses
The main concern is that the effectiveness of the proposed token contribution estimation is not sufficiently validated. DTCA assumes that the confidence improvement of remaining masked positions after token commitment is a reliable indicator of token importance, but it remains unclear whether this signal corresponds to the actual causal contribution of tokens to final generation quality.
Furthermore, the contribution of individual components is not fully isolated. The current ablation study evaluates removing major modules, but does not clearly separate the effects of contribution estimation, POS aggregation, timestep selection, and token-level credit assignment. Therefore, it is difficult to determine which component is primarily responsible for the reported improvements.
Questions for authors
-
The proposed contribution score is used as a proxy for token importance. Have the authors conducted any causal validation, such as token perturbation, masking, or intervention experiments, to verify that tokens with higher contribution scores actually have greater influence on final generation quality?
-
How stable are the POS-level contribution weights across different datasets, rollout samples, and training stages? Does the contribution table need to be recomputed when applying DTCA to a new task or model?
-
Could the authors provide a more detailed analysis of the computational overhead introduced by offline contribution estimation, including additional forward passes and total training cost compared with the baseline RL method?
3 comments
Why we use a fixed table during training. During RL rollouts, multiple tokens are committed in the same step, making it difficult to separate each token's effect on prediction confidence at the remaining masked positions. Replaying states for individual tokens is costly. We therefore estimate token contributions before training and average them within each POS category to build a reusable POS weight table. During training, tokens receive the weights of their POS categories, and the table stays fixed. Appendix C.3 describes how the fixed table is used. Your suggestion motivates us to further explore how to efficiently update the POS table during training to better reflect changes in the model. Thank you for this valuable suggestion.
Q3: How many extra forward passes does offline contribution estimation require, and what is the total training cost compared with the baseline?
A: We use 500 questions from each of the MATH and GSM8K training sets, with 4 responses per question, to build the POS weight table once before training. The table is reused during training. At a generation length of 256, each response requires 128 forward passes for generation and additional forward passes for contribution estimation, for a total of 509. No gradients are computed.
On 4 NVIDIA B200 GPUs with the same settings, 500 updates take 18.54 hours for DTCA and 17.95 hours for RLDF. DTCA's training time is about 3.3% higher.
Thank you for your response. I would be happy to see this work accepted.
Thank you for your careful review and helpful suggestions. We address your comments below.
A: Thank you for this suggestion. We added a random token replacement experiment. We use 64 questions that LLaDA-8B-Instruct originally answered correctly and select early and middle denoising states. In each trial, we replace the token with the highest contribution score in that step with a random token, then continue generation. All other committed tokens and decoding settings remain unchanged. We run 256 replacement trials in total.
After replacement, final answer accuracy drops by 5.86 percentage points, and the mean prediction probability of correct answer tokens before commitment drops by 3.49 percentage points. The mean number of tokens committed in the next step also drops from 2.69 to 1.87. These results show that changing a high-contribution token affects later denoising and the final answer.
A: Following your suggestion, we added two controls to the existing ablations: replacing contribution scores with token prediction confidence, and selecting denoising steps based on low confidence. The second control keeps contribution-based token-level credit assignment.
All configurations use the checkpoint at update 500. Other training settings follow Appendix C.2. The generation length and number of denoising steps are both 256. Other evaluation settings follow Appendix C.4.
The two added controls achieve lower accuracy than full DTCA, supporting contribution estimation and contribution-based denoising step selection, respectively. Removing token-level credit assignment or entropy calibration also lowers accuracy, showing the benefit of both components. The settings and results are reported in Appendix D.1, Table 2.
A: Our results show relatively similar contribution rankings for the main POS categories across math datasets. Weights and rankings are also generally stable across rollout samples. The difference between math and code is larger, so we rebuild the table using data from a new target task. On the same dataset, the two models have relatively similar rankings. When changing models, the table can be rebuilt as needed to adjust the weights.
The table below compares datasets, tasks, and models. Each setting uses 200 questions with 4 responses per question. The dataset and task comparisons use LLaDA-8B-Instruct. The model comparison uses LLaDA and Dream-v0-Instruct-7B on the same GSM8K questions.
Differences across tasks. Among the 8 main POS categories in Figure 1, the ADP group moves from 6th on GSM8K to 1st on MBPP. On MBPP, this group mainly contains
in,for, andfrom, withinandformaking up 90.1%. These words are often used in loops, membership checks, and imports. Their roles differ from those of prepositions in math responses.See Appendix D.4 for these analyses, Table 5 for POS rankings, and Figure 11 for the MBPP analysis.
Rollout samples. Using an earlier set of rollouts from 1,000 MATH questions, we randomly sample 200 questions and keep all 8 responses per question. We repeat this 20 times to build POS weight tables. Compared with the table built from all questions, the mean weight difference across all 18 POS categories is 0.0257, and the mean rank correlation is 0.971. The weights and rankings are therefore generally stable across these samples.
Our response continues in Part 2, where we discuss why we use a fixed table during training and the computational cost.