- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
CLD studies catastrophic forgetting during continual reinforcement learning of diffusion language models. It preserves historical decoding behaviour through token-distribution distillation and a pairwise confidence-ranking objective on stored intermediate states. Experiments examine retention, adaptation, and generalisation across reasoning tasks, with additional analyses of decoding-order drift.
Critical strengths
- The problem is well motivated: sequential adaptation demonstrably harms previously acquired capabilities, and flexible decoding order provides a relevant perspective for diffusion models. The proposed alignment integrates naturally with existing RL objectives and adds no inference-time parameters or decoding steps.
- The empirical evidence is substantial. Table 1 improves average accuracy from 35.99 for vanilla continual RL and 36.80 for prompt replay to 39.03. Component ablations on two backbones, alternative domain orders, and the three-seed results in Table 8 support the method's practical value and the complementary contribution of both alignment terms.
Critical weaknesses
- Specific value of historical order. The correlations and component ablations support order alignment, but do not fully distinguish preserving historically learned order from a more general benefit of constraining confidence rankings. A simple order target would help establish whether the historical trajectory information is essential to the gains.
- Interpretation of the ranking target. Equation 9 uses confidence scores at stored states, whereas Appendix E.4 stores one midpoint state per block. Confidence can change as subsequent tokens are revealed, and position-index tie-breaking imposes an order on simultaneously decoded tokens. The correspondence between these targets and the actual decoding process therefore needs clarification.
- Replay resource comparison. The stated equal memory budget is insufficiently quantified: prompt replay stores prompts, whereas CLD additionally stores states, probabilities, and orders. Historical processing and runtime are also unspecified, limiting interpretation of the practical trade-off. This is a secondary concern.
Questions for authors
- In one representative setting, how does replacing historical order with fixed positional order affect performance when stored states, token alignment, ranking-loss weight, and training budget are held constant?
- Could you specify the confidence score and valid pair construction, and report how well the historical model's midpoint confidence ranking agrees with its eventual decoding order, treating simultaneous commits as ties?
- Could you clarify the actual storage in bytes, historical sampling/update arrangement, and measured training overhead for CLD and prompt replay, explaining how their budgets are matched?
4 comments
Thank you for the response. The fixed and shuffled order controls clearly show that the historical order target matters, and the confidence definition and cost measurements resolve my other questions. One limitation remains: midpoint confidence agrees with the realised decoding order on about 70–75% of pairs, so the ranking target is an approximate proxy for the historical order.
Overall, I remain positive about this paper and keep my rating of 3 in support of acceptance.
Sincerely thank you again for your thoughtful and encouraging feedback! We really appreciate your time and support.
We totally agree with you that:
- Our current design is intentionally a surrogate objective of the decoding order.
- This is because it's non-trivial to directly optimise the decoding order.
That said, we are now very motivated by your suggestion to design a more direct yet optimisable objective for decoding order. If you would happen to or be willing to share any possible direction or thought, we would be really grateful and try it out immediately!
Thank you again. We wholeheartedly hope you would consider championing our paper during the discussion.
Dear Reviewer 9kEP,
As the discussion deadline approaches, we want to thank you once again for your thoughtful feedback and continued support! We sincerely hope you might consider championing our paper during the final discussion. We would be truly grateful for your support!
Best,
Authors 18080
Response to Reviewer 9kEP
We sincerely thank you for your insightful question, and we have thoroughly addressed all your concerns. We summarise our response and highlight the key information as follows:
Thank you for this insightful suggestion. We follow your suggestion and further examine the value of preserving historical decoding orders through four points:
Figure 3 links higher order similarity with better BWT. Figure 5 and Section 5.5 further show that CLD reduces historical order drift.
Table 2 shows that order-only alignment outperforms token-only alignment on LLaDA (38.40 vs. 36.65) and Dream (41.30 vs. 40.28).
For Fixed Positional Order, we replace historical orders with a fixed left-to-right order.
For Shuffled Historical Order, we randomly shuffle the stored historical orders.
Both controls keep the same stored states, token alignment, ranking loss, and training budget.
CLD outperforms both controls, showing the value of preserving historical decoding orders over arbitrary ordering constraints.
Thank you for this insightful question. We clarify two points:
For both LLaDA and Dream, confidence is defined as ct,i=maxv∈Vpθ(v∣x,st)i. Valid pairs are masked positions within the same block, ordered by their historical commitment steps. Same-step ties are resolved by position index for alignment (Eq. 1).
Following your suggestion, we analyse 8,000 historical midpoint states per backbone to measure agreement between midpoint confidence rankings and eventual decoding orders. Same-step ties are excluded.
Midpoint confidence rankings agree with realised decoding orders for 75.36% of pairs on LLaDA and 70.13% on Dream, substantially exceeding the 50% random-order expectation.
These results show that midpoint confidence provides a useful signal for learning historical decoding orders through differentiable alignment.
Thank you for raising this practical concern. We clarify three points:
Both CLD and Prompt Replay store 100 items per previous domain and sample 12 historical records per update. This ensures the same item budget, although storage in bytes differs.
CLD additionally stores diffusion states, token probabilities, and decoding orders. We measure storage, training time, and peak GPU memory on a single AMD MI355X (288 GB), including evaluation.
Overall, CLD achieves higher ACC (39.03) than Prompt Replay (36.80) and Vanilla (35.99) with modest additional costs.
We sincerely appreciate your valuable suggestion, which has been very helpful in improving our manuscript. We would be grateful for any further questions or feedback you may have.