- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes to use clean state likelihoods generated from the model from partial states (instead of next state) in the policy gradient method for reinforcement learning (known as CSPO) for diffusion language models (DLMs), since there is an inherent lack of exact likelihood of the model generating the particular sequence in DLMs. Furthermore to reduce expensive gradient computations for all steps, the work gives a simple way to pick only k steps from the roll-outs where the model assigned least confidence. Across math and code benchmarks, model trained with CSPO demonstrates good cross-setting generalization.
Critical strengths
- Good results on within-domain cross-dataset generalization.
- "Novel" idea of picking steps based on weight-probability to reduce the expensive computation of gradient on all steps.
Critical weaknesses
-
Not using the 2 tokens/step decoding strategy which more faithfully represents diffusion language modeling. At least, provide the avg. decoded tokens/step in the dynamic decoding strategy.
-
Lack of accuracy results with full clean state loss without token clipping, so as to assess the argument for clean states quantitatively.
-
Picking the steps based on a metric computed from unmasking distributions (confidence, entropy) is not entirely novel. See [1].
[1] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.
Questions for authors
- Can you include a discussion with [1] in the related work, especially about their entropy vs. your weighted probability-based step-selection?
- Can you compare with [1] by evaluating your checkpoints with 2 tokens/step (w/o training)?
- Can you provide the results (at least on any one dataset) for your algorithm CSPO trained with 2 tokens/step w/o token clipping?
[1] Reinforcement Learning for Diffusion LLMs with Entropy-Guided Step Selection and Stepwise Advantages. https://arxiv.org/abs/2603.12554.