acceptodds
Under review as a conference paper at ICLR 2027

Decoding Trajectory Forgetting: Continual Reinforcement Learning for Diffusion Large Language Models via Trajectory Alignment

Abstract

Diffusion large language models (dLLMs) introduce a parallel decoding paradigm that resolves multiple masked positions with a flexible order. Recent developments in post-training for dLLMs have substantially improved dLLM reasoning capabilities in single-domain training and inference scenarios with reinforcement learning (RL). Yet, continually adapting dLLMs to evolving real-world tasks remains challenging, as previously learnt capabilities can rapidly degrade after model post-training on new domains. To investigate the reason behind catastrophic forgetting in dLLMs, we examine how previously learnt decoding trajectories evolve during continual RL from two complementary dimensions: (1) token prediction distributions that characterise what to decode, where continual RL can alter prediction behaviour on previous domains, and (2) decoding order patterns that characterise which positions to decode, where continual RL can distort the relative decoding order between token positions. Beyond changes in token predictions, we find that forgotten tasks exhibit more pronounced order drift in the decoding trajectory than preserved ones, revealing decoding trajectory forgetting as an important dimension of catastrophic forgetting in dLLM continual RL. Motivated by these insights, we introduce CLD, a novel continual RL framework for dLLMs that enables continual adaptation while preserving previously learnt decoding trajectories. During continual RL, the native RL objective drives adaptation to the incoming domain, while decoding trajectory alignment provides stability by preserving previously learnt decoding behaviour. Moreover, CLD can be easily integrated with different dLLM RL algorithms in a plug-and-play style. Extensive experiments on 9 benchmarks demonstrate that CLD effectively mitigates catastrophic forgetting in dLLMs. Our code is provided at https://anonymous.4open.science/r/CLD-2026

Primary area:
foundation or frontier models, including LLMs
Keywords:
Continual Reinforcement Learning, Large Language Models, Diffusion Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_9kEPOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CLD studies catastrophic forgetting during continual reinforcement learning of diffusion language models. It preserves historical decoding behaviour through token-distribution distillation and a pairwise confidence-ranking objective on stored intermediate states. Experiments examine retention, adaptation, and generalisation across reasoning tasks, with additional analyses of decoding-order drift.

Critical strengths
  1. The problem is well motivated: sequential adaptation demonstrably harms previously acquired capabilities, and flexible decoding order provides a relevant perspective for diffusion models. The proposed alignment integrates naturally with existing RL objectives and adds no inference-time parameters or decoding steps.
  2. The empirical evidence is substantial. Table 1 improves average accuracy from 35.99 for vanilla continual RL and 36.80 for prompt replay to 39.03. Component ablations on two backbones, alternative domain orders, and the three-seed results in Table 8 support the method's practical value and the complementary contribution of both alignment terms.
Critical weaknesses
  1. Specific value of historical order. The correlations and component ablations support order alignment, but do not fully distinguish preserving historically learned order from a more general benefit of constraining confidence rankings. A simple order target would help establish whether the historical trajectory information is essential to the gains.
  2. Interpretation of the ranking target. Equation 9 uses confidence scores at stored states, whereas Appendix E.4 stores one midpoint state per block. Confidence can change as subsequent tokens are revealed, and position-index tie-breaking imposes an order on simultaneously decoded tokens. The correspondence between these targets and the actual decoding process therefore needs clarification.
  3. Replay resource comparison. The stated equal memory budget is insufficiently quantified: prompt replay stores prompts, whereas CLD additionally stores states, probabilities, and orders. Historical processing and runtime are also unspecified, limiting interpretation of the practical trade-off. This is a secondary concern.
Questions for authors
  1. In one representative setting, how does replacing historical order with fixed positional order affect performance when stored states, token alignment, ranking-loss weight, and training budget are held constant?
  2. Could you specify the confidence score and valid pair construction, and report how well the historical model's midpoint confidence ranking agrees with its eventual decoding order, treating simultaneous commits as ties?
  3. Could you clarify the actual storage in bytes, historical sampling/update arrangement, and measured training overhead for CLD and prompt replay, explaining how their budgets are matched?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 9kEP

We sincerely thank you for your insightful question, and we have thoroughly addressed all your concerns. We summarise our response and highlight the key information as follows:

Re W1&Q1: Fixed Positional Order / Shuffled Historical Order

Thank you for this insightful suggestion. We follow your suggestion and further examine the value of preserving historical decoding orders through four points:

  1. Figure 3 links higher order similarity with better BWT. Figure 5 and Section 5.5 further show that CLD reduces historical order drift.

  2. Table 2 shows that order-only alignment outperforms token-only alignment on LLaDA (38.40 vs. 36.65) and Dream (41.30 vs. 40.28).

  3. For Fixed Positional Order, we replace historical orders with a fixed left-to-right order.

  4. For Shuffled Historical Order, we randomly shuffle the stored historical orders.

Both controls keep the same stored states, token alignment, ranking loss, and training budget.

Ranking TargetLogic CRLMath CRLCode CRLACC↑BWT↑
Fixed positional order47.8235.0025.6036.14-2.75
Shuffled historical order45.5034.6025.8035.30-4.01
Historical order (CLD)52.6936.0028.4039.03+0.29

CLD outperforms both controls, showing the value of preserving historical decoding orders over arbitrary ordering constraints.

Re W2&Q2: Midpoint Confidence Ranking vs Eventual Decoding Order

Thank you for this insightful question. We clarify two points:

  1. For both LLaDA and Dream, confidence is defined as ct,i=max⁡v∈Vpθ(v∣x,st)ic_{t,i}=\max_{v\in\mathcal{V}}p_\theta(v\mid x,s_t)_i. Valid pairs are masked positions within the same block, ordered by their historical commitment steps. Same-step ties are resolved by position index for alignment (Eq. 1).

  2. Following your suggestion, we analyse 8,000 historical midpoint states per backbone to measure agreement between midpoint confidence rankings and eventual decoding orders. Same-step ties are excluded.

BackboneKK (%)Math (%)KodCode (%)Averaged Agreement (%)Tie Ratio (%)
LLaDA-8B70.7083.4471.9375.366.67
Dream-7B59.8273.3877.1870.136.67

Midpoint confidence rankings agree with realised decoding orders for 75.36% of pairs on LLaDA and 70.13% on Dream, substantially exceeding the 50% random-order expectation.

These results show that midpoint confidence provides a useful signal for learning historical decoding orders through differentiable alignment.

Re W3&Q3: Replay Resource Comparison, Storage in Bytes & Training Overhead

Thank you for raising this practical concern. We clarify three points:

  1. Both CLD and Prompt Replay store 100 items per previous domain and sample 12 historical records per update. This ensures the same item budget, although storage in bytes differs.

  2. CLD additionally stores diffusion states, token probabilities, and decoding orders. We measure storage, training time, and peak GPU memory on a single AMD MI355X (288 GB), including evaluation.

MethodItems / DomainStorage (MB / Domain)Historical Samples / UpdateTime / Step (s)Total GPU HoursPeak GPU Memory (GiB)
Vanilla Continual RL00022.120.3188.5
Prompt Replay1006.71223.821.9190.4
CLD10010.81224.622.7194.7
  1. Compared with Vanilla, CLD adds 11.3% training time per step and 6.2 GiB peak GPU memory. Compared with Prompt Replay, it adds only 3.4% time per step and 4.1 MB storage per domain.

Overall, CLD achieves higher ACC (39.03) than Prompt Replay (36.80) and Vanilla (35.99) with modest additional costs.

We sincerely appreciate your valuable suggestion, which has been very helpful in improving our manuscript. We would be grateful for any further questions or feedback you may have.

Thank you for the response. The fixed and shuffled order controls clearly show that the historical order target matters, and the confidence definition and cost measurements resolve my other questions. One limitation remains: midpoint confidence agrees with the realised decoding order on about 70–75% of pairs, so the ranking target is an approximate proxy for the historical order.

Overall, I remain positive about this paper and keep my rating of 3 in support of acceptance.

Sincerely thank you again for your thoughtful and encouraging feedback! We really appreciate your time and support.

We totally agree with you that:

  1. Our current design is intentionally a surrogate objective of the decoding order.
  2. This is because it's non-trivial to directly optimise the decoding order.

That said, we are now very motivated by your suggestion to design a more direct yet optimisable objective for decoding order. If you would happen to or be willing to share any possible direction or thought, we would be really grateful and try it out immediately!

Thank you again. We wholeheartedly hope you would consider championing our paper during the discussion.

Dear Reviewer 9kEP,

As the discussion deadline approaches, we want to thank you once again for your thoughtful feedback and continued support! We sincerely hope you might consider championing our paper during the final discussion. We would be truly grateful for your support!

Best,

Authors 18080

Reviewer_NBuBOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates catastrophic forgetting in continual reinforcement learning for diffusion large language models (dLLMs). The authors identify decoding-order drift as an additional dimension of forgetting beyond changes in token predictions. Based on this observation, they propose CLD, a trajectory-alignment framework that preserves both historical token prediction distributions and decoding-order patterns through differentiable objectives. Experiments across multiple domains and dLLM backbones demonstrate that CLD effectively mitigates catastrophic forgetting while maintaining the ability to learn new tasks.

Critical strengths

The key strength of this paper is the insightful identification of decoding order shift as an important dimension of catastrophic forgetting in diffusion LLMs. Similar to how parameter or weight shifts are studied in classical continual learning, the authors highlight a previously underexplored form of behavioral drift specific to diffusion models. Building on this observation, they develop a clever and effective trajectory alignment method to preserve previously learned decoding patterns. I find both the motivation and the proposed solution insightful and well-designed.

Critical weaknesses

Weakness: Limited understanding of why decoding order alignment mitigates forgetting.

While the paper demonstrates that decoding order alignment effectively mitigates catastrophic forgetting, the underlying mechanism could be further explored. Specifically, the proposed decoding order alignment loss is a differentiable surrogate based on pairwise token confidence scores, rather than directly optimizing the actual decoding order. This loss can continue to encourage changes in confidence margins even when the relative decoding order is already preserved. Therefore, it remains unclear whether the performance improvement primarily comes from preserving previously learned decoding orders or from the additional regularization imposed on token confidence scores. A more detailed analysis disentangling these two effects would provide deeper insights into the role of decoding order in catastrophic forgetting.

Questions for authors

See weakness.

2 comments
AuthorsOpenReview ↗

Response to Reviewer NBuB

We sincerely appreciate your recognition of the decoding order perspective and your suggestion to further investigate its mechanism. We summarise our response and highlight the key information as follows:

Re W1: Historical Decoding Order vs Confidence Regularisation

We fully agree that the current Softplus ranking loss can continue increasing confidence margins even when the relative order is correct. To directly address your concern, we conduct two additional analyses:

  1. We introduce Rank-only Alignment using a zero-margin Hinge Loss, which stops updating confidence margins once the relative decoding order is preserved.

  2. We further compare shuffled historical orders and confidence-only regularisation, which increases pairwise confidence margins without historical order targets. All controls use the same stored states, token alignment, RL settings, and training budget.

MethodLogic CRLMath CRLCode CRLACC↑BWT↑
Confidence-only regularisation42.4333.6026.4034.14-4.44
Shuffled historical order45.5034.6025.8035.30-4.01
Rank-only alignment (CLD, Hinge)51.8035.8028.0038.53+0.14
Historical order (CLD, Softplus)52.6936.0028.4039.03+0.29

Rank-only Alignment achieves results close to CLD, while shuffled and confidence-only controls perform worse. This supports the role of historical order preservation beyond additional confidence regularisation.

We sincerely appreciate your valuable suggestion, which has been very helpful in improving our manuscript. We would be grateful for any further questions or feedback you may have.

Dear Reviewer NBuB,

Thank you so much for your thoughtful review and encouraging feedback! We truly appreciate your recognition of our motivation and approach. We have carefully addressed your concern about the role of historical decoding order preservation versus confidence regularisation, with additional experiments comparing rank-only alignment, shuffled historical orders, and confidence-only regularisation above. Your suggestions have been very helpful in strengthening our paper.

As the discussion deadline approaches, we would be very grateful if you could take another look at our responses and share your thoughts. We are eager to address any remaining concerns and sincerely hope you might consider championing our paper during the final discussion. We wholeheartedly appreciate your support!

Best,

Authors 18080

Reviewer_YGGuOpenReview ↗
Rating
2
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies catastrophic forgetting when a diffusion LLM (mainly LLaDA-8B-Instruct, also Dream-7B and LLaDA2.0-mini) is trained with RL on three domains in sequence (Knights and Knaves logic, then MATH, then KodCode). It separates the forgetting into two parts: drift in the token prediction distributions, and drift in the order in which positions are unmasked within a block, measured with a Kendall-style score S_K. It reports that forgotten tasks show larger order drift than preserved ones (Fig. 3). The proposed method, CLD, stores 100 reward-positive rollouts per finished domain and, while training on later domains, adds a top-K KL term on the stored states and a pairwise loss that keeps the old confidence ranking between positions. With StableDRL as the RL algorithm, CLD reaches ACC 39.03 and BWT +0.29, against 35.99 / -3.07 for plain sequential RL and 36.80 / -2.06 for prompt replay (Table 1). The paper also reports a 3-seed run (Table 8), ablations (Table 2), and results with other RL algorithms and backbones (Tables 9 to 11).

Critical strengths
  1. A forgetting dimension that is specific to diffusion LLMs. Treating the unmasking order as something that can be forgotten is new, and I did not find prior continual-RL work on diffusion LLMs. The ablation supports that it matters: the order-only variant beats the token-only variant, which is essentially replay distillation, on both backbones (38.40 vs 36.65 on LLaDA and 41.30 vs 40.28 on Dream, Table 2). This is the main reason the paper is interesting to me.

  2. The gain over plain sequential RL holds across seeds. In Table 8 (3 seeds) CLD reaches ACC 39.67 vs 35.49 and BWT +0.26 vs -4.42, and the method adds no inference cost. So the effect over the vanilla baseline is not seed noise.

Critical weaknesses
  1. I generally think the algorithm complexity vs gain, make it not to be an ideal approach to be adopted by real practitioners.

In the following I am summerizing the results of cross check of numbers with related paper, I used AI to cross check those numbers.

  1. The baselines are reused, under-trained or mismatched, so the comparison does not establish that CLD beats a properly tuned continual-learning method.

(a) The base, SFT, single-domain and Vanilla Mix RL rows of Table 1, and the LLaDA base row of Table 3, are identical to the numbers in Block-R1 (arXiv 2605.11726, Tables 1 and 4). The paper does not say these numbers are reused, and it does not say on which split those checkpoints were selected.

(b) The baselines are well below their own papers at the same generation length of 256. On MATH500: StableDRL 35.40 here vs 45.2 in its paper (arXiv 2603.06743, which trains 2,000 steps at lr 1e-6, against 1,000 steps at lr 3e-7 here); d1 33.40 vs 38.6 and Diffu-GRPO 33.60 vs 37.2 (arXiv 2504.12216). Only wd1 matches (34.20 vs 34.4, arXiv 2507.08838).

(c) Vanilla Mix RL scores below the untrained base model (26.98 vs 28.00) because it is Block-R1's six-domain mix, which also includes Countdown, GSM8K and Sudoku. It is not a matched multi-task upper bound. Block-R1's own mix result (mean 38.18) is close to CLD's 39.03 and is not reported.

(d) Prompt replay is the only continual-learning baseline, and it falls below vanilla on Dream (35.92 vs 37.35, Table 2). In its source (arXiv 2608.18574) prompt replay with pass-rate based selection reaches multi-task performance, and that selection rule is missing here (App. E.2). Replay with KL to the previous policy, LwF, EWC and CPO (arXiv 2607.04364, which the paper cites and which has public code) are all missing. The reference policy of the beta = 0.04 KL term in the RL loss is never stated.

(e) Tables 9 to 11 have no vanilla rows, and with Diffu-GRPO, d1 and wd1 CLD's BWT is still negative (Table 10). In Table 11 block size alone moves KodCode by 7.2 points, more than CLD's 1.6-point gain over vanilla.

This matters because the central claim is that CLD is a better way to retain old skills than existing tools. With these baselines I cannot tell whether a tuned replay plus KL method would match it.

  1. Some gains do not fit the proposed mechanism, the checkpoint selection appears to use test data (a) In Table 3 CLD raises LLaDA's ARC-E from 78.16 to 93.06 and ARC-C from 75.09 to 84.90 on tasks never trained on. The base scores look suppressed by the evaluation protocol: the LLaDA paper reports ARC-C 88.5 and HumanEval 49.4 (arXiv 2502.09992) against 73.21 and 28.66 here, and Dream reports HumanEval 55.5 and MBPP 58.8 (arXiv 2508.15487) against 24.39 and 33.40 here. Recovered answer format or fewer truncations is a more likely cause than retained decoding trajectories.

(b) CLD also improves the domain currently being learned (last-stage Logic 53.79 vs 48.17 on LLaDA and 51.79 vs 47.26 on Dream, Table 4), and LLaDA2.0-mini's MATH500 rises from 66.40 to 75.80 while it trains on Code (Table 9). A stability term should not do this, which points to format effects or selection variance.

(c) Table 5 says checkpoints are selected on a 10% held-out split, but App. E.5 mentions periodic evaluation, average forgetting is defined on the best test checkpoint, and the reused numbers come from Block-R1 and d1, which both report the best test-set checkpoint.

Other issues:

  • The link between order drift and forgetting (Fig. 3) comes from one run, and D_traj contains (1 - S_K) by construction. Cross-domain S_K is about 0.67 in Fig. 2a but 0.41 to 0.43 in Fig. 7, which is below the random-order value of 0.5.
  • Theorem 1 is the data-processing inequality, and under deterministic remasking the KL can be infinite.
  • The confidence score c_{t,i} that drives the order loss is never defined, including for Dream.
  • The compute overhead of CLD is not reported, and there is no compute-matched vanilla run.
Questions for authors
  1. [Clarification] Which numbers in Tables 1 and 3 are reused from Block-R1, and on which split was each reported checkpoint selected (held-out or test)? If checkpoints were selected on test, the main results would need to be redone, which would change my assessment.

  2. [Clarification] Why is StableDRL about 10 points below its own paper on MATH500? Does CLD still help when the baselines are trained with their original budget?

  3. [Suggestion, affects my score] Can you add replay with KL to the previous policy, LwF, CPO, prompt replay with the original pass-rate selection, and a three-domain mix trained on the same data, each with 3 seeds? Also, what is the reference policy of the beta = 0.04 KL term?

  4. [Clarification] Please report format-failure and truncation rates for Table 3, and add vanilla rows to Tables 9 to 11.

  5. [Clarification] How is the confidence score c_{t,i} defined, including for Dream?

8 comments
AuthorsOpenReview ↗

Response to Reviewer YGGu [1/6]

We sincerely thank you for your valuable questions and feedback. We have carefully and thoroughly addressed all your concerns and updated the manuscript. For clarity, we organise our responses in the following order: Questions, Weaknesses, and Other Issues. We summarise our responses and highlight the key information as follows:

Re Q1: Reused Numbers from Block-R1 & Held-out or Test Checkpoints

Thank you for raising this important concern. We would like to clarify that only reference baseline results are reused from Block-R1, while all continual RL results are independently obtained with validation-based checkpoint selection.

  1. Reused Results: We reuse the Base, SFT, single domain RL, and Mix-domain RL results in Table 1, together with the LLaDA Base results in Table 3, from the unified Block-R1 benchmark. No continual RL results are reused.

  2. Checkpoint Selection: All continual RL methods, including Vanilla, Prompt Replay, and CLD, select checkpoints exclusively using a 10% held-out validation split, with evaluation every 100 steps (Table 7, Algorithm 1, and Appendix E.5). Test data are never used for checkpoint selection.

  3. Zero-shot Evaluation: For MBPP and HumanEval, checkpoints are selected using KodCode validation performance. For other zero shot dataset such as ARC-E and ARC-C, checkpoint selection uses the average validation performance across KK, Math, and KodCode. None of these zero-shot test benchmarks are used for checkpoint selection.

We greatly appreciate your suggestion and have clarified the provenance of the reused results in Section 5 (Lines 369-377) of the revised manuscript.

Re Q2: StableDRL on MATH500 & Original Training Budget

Thank you for raising this concern. We would like to clarify that the reported StableDRL results follow a unified benchmark, while CLD's effectiveness is evaluated under matched continual RL training budgets.

  1. Single-domain Benchmark Comparability: The StableDRL score of 35.40 on MATH500 is taken from the unified Block-R1 benchmark (Table 4, page 19, arxiv:2605.11726), rather than the original StableDRL paper.

  2. Cross-paper differences can be substantial due to different training settings. For example, d1 reports 42.2 versus 34.4 on Countdown (length 512) in its original paper and the StableDRL paper, respectively. Similarly, wd1 reports 76.4 versus 32.1 on Sudoku (length 256).

  3. Continual RL Training Budgets: CLD is evaluated against the corresponding Vanilla continual RL baseline under identical training settings. As shown in Appendix F.9 and Table 12, CLD consistently improves ACC and BWT across different dLLM RL algorithms under matched sequential training.

Therefore, our results support CLD's effectiveness under controlled continual RL comparisons, rather than establishing improvements under every original-paper training budget.

AuthorsOpenReview ↗

Response to Reviewer YGGu [2/6]

Re Q3: Replay with KL, LwF, CPO, Pass Rate Selection, Three-Domain Mix & KL Reference Policy

We fully agree with you that adding more continual learning baselines is important for a stronger evaluation of dLLM continual RL. We have additionally implemented five continual learning baselines, covering three categories:

  • Experience Replay: CPR (Pass-rate Selection) and Replay + Previous-policy KL.
  • Knowledge Distillation: Learning without Forgetting (LwF).
  • Parameter Regularisation: Elastic Weight Consolidation (EWC) and Continual Policy Optimization (CPO).

All five methods retain the same native StableDRL objective as CLD for a fair and consistent comparison, differing only in their forgetting mitigation mechanisms. The details are:

  1. Replay + Previous-policy KL: This key requested baseline replays historical prompts and applies KL regularisation between the previous and current policies' token prediction distributions.
  2. LwF and EWC: LwF preserves previous policy predictions through knowledge distillation, while EWC constrains changes to important parameters. Both do not explicitly address historical decoding order preservation problem in dLLMs.
  3. CPO was originally designed for VLMs for image-based reasoning tasks. We adapt CPO to our dLLM continual RL setting for text-based reasoning tasks and report its results below.
MethodLogic CRLMath CRLCode CRLACC↑BWT↑AF↓
Vanilla Continual RL49.1732.0026.8035.99-3.074.36
CPR (Pass-rate Selection)48.7429.8027.2035.25-3.193.39
Replay + Previous-policy KL47.3627.8022.2032.45-2.082.86
LwF50.5230.8026.0035.77-2.403.18
EWC48.5933.6024.2035.46-2.963.47
CPO44.6633.6028.0035.42-4.334.87
CLD (Ours)52.6936.0028.4039.03+0.290.10

Under the same dLLM continual RL training and zero-shot evaluation settings, CLD achieves the best ACC, BWT, and AF across all compared methods. The other methods still show forgetting, as they do not explicitly address the decoding order trajectory problem unique to dLLMs.

Regarding the KL reference policy: Our default StableDRL objective does not adopt a reference-policy KL penalty. The β = 0.04 in Table 8 applies to other RL algorithms such as d1, where the reference policy in d1 is typically the initial model before RL post-training.

Re Q4: Format-failure and Truncation Rates & Vanilla Rows

Following your suggestion on format issue,

  1. We confirm that all experiments use the same answer parser within the unified evaluation framework.
  2. We additionally count format failure and truncation rates of:
    • Format failure: The answer parser cannot extract a valid answer.
    • Truncation: The response reaches the generation limit without an EOS or end-of-turn token and the answer.

For LLaDA-8B:

ModelMetric (%)MBPPHumanEvalGSM8KCountdownARC-EARC-CAvg.
BaseAccuracy33.6028.6676.1916.8076.9873.2150.91
Format failure10.1112.800.682.344.634.445.83
Truncation0.000.000.000.000.000.000.00
Vanilla CRLAccuracy33.8027.4478.2418.7578.1675.0951.91
Format failure9.3512.210.000.000.080.003.61
Truncation0.000.000.000.000.000.000.00
CLDAccuracy34.4029.8880.2926.5693.0684.9058.18
Format failure8.6011.590.680.000.460.433.63
Truncation0.000.000.000.000.000.000.00

For Dream-7B:

ModelMetric (%)MBPPHumanEvalGSM8KCountdownARC-EARC-CAvg.
BaseAccuracy33.4024.3977.3319.1480.8176.0251.85
Format failure18.4018.900.4524.221.142.4710.93
Truncation0.200.000.080.000.000.000.05
Vanilla CRLAccuracy35.2031.1077.5621.4882.3278.6754.39
Format failure13.4012.800.083.120.040.004.91
Truncation0.000.000.000.000.000.000.00
CLDAccuracy39.6040.8580.5932.4293.9487.2062.43
Format failure13.4010.980.381.560.340.264.48
Truncation0.000.000.000.000.000.000.00
  1. All variants show a reasonable range of format issue percentage.
  2. CLD's improvement is not attributed to format issue improvement, but to the decoding trajectory.

For the Vanilla comparisons, we kindly draw your attention to our response to Re W2(e) below, where we provide matched comparisons.

AuthorsOpenReview ↗

Response to Reviewer YGGu [3/6]

Re W1: Complexity vs Gain

We totally agree with you that for real practitioners, efficiency is very important. Therefore, we already consider this complexity vs gain trade-off in our manuscript.

  1. Low overhead: In Section 4.4, we analyse that CLD (1) uses compact trajectory memory and (2) introduces no additional cost in inference parameters or decoding steps.
  2. Significant gains: Table 1 shows that CLD significantly improves ACC from 35.99 to 39.03 and BWT from -3.07 to +0.29, while reducing AF from 4.36 to 0.10.

CLD achieves a significantly improved performance without additional inference overhead, supporting its practical applicability.

Re W2: Results vs Other Papers

We sincerely thank you for carefully cross checking our reported results. Before all,

  1. We provide the result of key requested baseline: Replay + Previous-policy KL in "Re Q3" above for reference.
  2. We would like to kindly double confirm that our reported results of all baselines/our CLD are fully reproducible with the provided code. We kindly invite the reviewer to run the provided code if there is further concern on the result.
  3. We would like to kindly reassure that we conduct our experiments to strictly ensure fairness and consistency in the experimental setting.

Re W2.a&b: Reuse or Rerun

Thank you for the question.

  1. Our results are fully reproducible from the provided code, which is also based on the evaluation protocol from Block-R1's public repo.
  2. For the main continual RL setting:
  • We rerun the baselines under Block-R1's configuration adapted to continual setting;
  • This is because (1) most baselines are developed in their own single domain setting; and (2) Block-R1's repo does not report continual RL's result.
  • We did this to ensure a fair and consistent continual evaluation protocal.
  1. For the other Base Model, SFT, single domain RL, and Mix-domain RL settings:
  • We reuse the evaluation results from Block-R1's public repo to keep fairness and consistency;
  • This is simply because these results are fully reproducible and provided by the repo.

We have updated this clarification in Line 369 to 377 of Section 5 in the updated manuscript.

Re W2.b: Baselines Number Different from Their Original Papers

We thank the reviewer for this careful observation. We would like to clarify that:

  1. Same generation length does not imply the same training setup. Performance also depends on the training recipe, including the learning rate, number of training steps, and parameter-update strategy.
  2. Our numbers come from a unified reproduction benchmark under a fair/consistent setting, not simply copying from the original papers.
  3. Reproduction under different experimental configurations with different results is not uncommon. For example, StableDRL (ICML 2026) reports:
    • d1 (NeurIPS 2025), Countdown (512): 42.2 (original) vs. 34.4 (StableDRL ICML 2026).
    • wd1 (ICLR 2026), Sudoku (256): 76.4 (original) vs. 32.1 (StableDRL ICML 2026).
    • The gap between the original paper and the reproduction is due to differences in training and implementation settings.
    • Yet these are still fair and consistent comparisons.

We would like to kindly reassure that:

  1. We are not under-reporting the baseline results;
  2. Our experiment setting is strictly fair and consistent;
  3. Every experiment in our manuscript is fully reproducible with the provided code.

Re W2.c: Vanilla Mix RL Results vs Block-R1

We would like to respectfully clarify that:

  1. Block-R1 baseline uses a privileged Block-R1-41K dataset coming with an additional supervision signal to obtain their much closer 38.18 (Section 4 and Section 6.1 in their arXiv version).
  2. Block-R1's privileged setting may not be properly directly comparable.
  3. The reported 26.98 of Vanilla Mix RL is strictly and fairly comparable under a unified and controlled setting as in Section 5 (Lines 331-339) and Table 1 (same dLLM backbone, decoding configuration, and evaluation protocols).

We would also like to kindly clarify that:

  1. We did not (aim to) claim Vanilla Mix RL as an upper bound.
  2. Vanilla Mix RL is just presented as a rigorously compared mix-domain training setting in Table 1.

Comments on the paper

5 comments
~Phileas_Sprecher1OpenReview ↗

Major Concern: Severely Insufficient Continual Learning Baselines

Thank you to the authors for their work. However, I have a serious concern regarding the substantial lack of comparisons with existing continual learning methods, which I consider a fundamental weakness of the experimental evaluation.

Although Table 1 includes several dLLM RL methods, most are evaluated in single-domain or joint-training settings. These methods are not designed to address catastrophic forgetting and therefore do not serve as meaningful continual learning baselines. The actual continual learning comparison consists of only two baselines: Vanilla Continual RL and Continual Prompt Replay. More importantly, Vanilla Continual RL does not introduce any mechanism to mitigate forgetting, leaving Continual Prompt Replay as essentially the only external baseline explicitly designed for CL.

I find this particularly concerning given that the central contribution of the paper is a method for mitigating catastrophic forgetting. Comparing against sequential training and a single replay method is not sufficient to establish the effectiveness of a new CL approach. The paper demonstrates improvements over relatively limited baselines, but does not adequately investigate whether established CL methods could achieve comparable or better results. In my view, this is not a minor experimental omission, but a fundamental limitation that directly affects the strength of the paper's main claims.

There is extensive literature on catastrophic forgetting, and I would expect a paper proposing a new CL method to include at least a representative selection of established approaches, such as:

  • General CL: EWC [1], LwF [2], and DER++ [3], covering parameter regularization, knowledge distillation, and experience replay.
  • CL for LLMs: O-LoRA [4], which investigates parameter-efficient continual adaptation.
  • Continual RL: CLEAR [5], Policy Consolidation [6], and Continual Policy Optimization [7], which specifically investigate catastrophic forgetting during sequential policy learning. These are not obscure or unrelated methods. They represent well-established directions in continual learning, and several are already discussed in the paper's related work. While some require adaptation to diffusion LLMs or RL training, standard techniques such as parameter regularization and knowledge distillation should be reasonably straightforward to implement within the existing training framework. I do not expect the authors to compare against every method, but the absence of representative baselines from these categories is difficult to justify.

My main concern is that the current experiments establish that CLD outperforms naive sequential training and one prompt-replay baseline, but do not establish that CLD offers a meaningful advantage over existing continual learning techniques. For a paper whose primary contribution is mitigating catastrophic forgetting, I consider this a serious gap in the experimental validation and a major obstacle to supporting its claimed contribution.

References

[1] Overcoming Catastrophic Forgetting in Neural Networks, PNAS 2017.

[2] Learning without Forgetting, ECCV 2016.

[3] Dark Experience for General Continual Learning: A Strong, Simple Baseline, NeurIPS 2020.

[4] Orthogonal Subspace Learning for Language Model Continual Learning, Findings of EMNLP 2023.

[5] Experience Replay for Continual Learning, NeurIPS 2019.

[6] Policy Consolidation for Continual Reinforcement Learning, ICML 2019.

[7] RL Forgets! Towards Continual Policy Optimization, arXiv 2026.

Re: Additional Continual Learning Baselines

Thank you for your detailed comment. We fully agree that comparing with more representative continual learning (CL) methods is important for a stronger evaluation of CLD. We would like to friendly clarify 3 points below:

1. Original Baseline Selection. Our original evaluation focused on the unique challenges of continual RL for diffusion large language models (dLLMs), particularly forgetting in iterative denoising and decoding trajectories. Since CLD is designed to work with existing dLLM RL algorithms, we initially selected baselines that could be evaluated under the same RL framework.

2. Additional CL Baselines. Following your suggestion, we have additionally implemented 5 representative CL baselines under the same LLaDA-8B-Instruct backbone, StableDRL algorithm, domain sequence, and evaluation protocol. These methods cover three categories:

  • Experience Replay: CPR (Pass-rate Selection) [1] and Replay + Previous-policy KL, which replay historical prompts and preserve previous policy behaviour.
  • Knowledge Distillation: Learning without Forgetting (LwF) [2], which preserves previous policy predictions.
  • Parameter Regularisation: Elastic Weight Consolidation (EWC) [3] and Continual Policy Optimization (CPO) [4], which constrain parameter changes during continual training.

We would also like to kindly clarify that the other four suggested methods are not directly compatible with our focused dLLM RL setting as:

  • DER++ [5]: Relies on supervised replay with stored logits and labels. Applying it to dLLM RL requires adapting logit replay to intermediate diffusion states.
  • O-LoRA [6]: Uses separate orthogonal parameter subspaces for different tasks. This changes our setting of continually updating a single shared model without task specific adapters.
  • CLEAR [7]: Combines replay with off policy learning and behavioural cloning. Its original training procedure differs from our StableDRL framework.
  • Policy Consolidation [8]: Maintains multiple historical policy networks. Applying this approach to 8B dLLMs would introduce substantial additional memory and computation costs.

We therefore focus on the other 5 representative methods that can be adapted and compared under the same dLLM continual RL settings.

3. Experimental Results. We report all results under the same continual RL training and evaluation settings, using ACC, BWT, and AF.

MethodLogic CRLMath CRLCode CRLACC↑BWT↑AF↓
Vanilla Continual RL49.1732.0026.8035.99-3.074.36
Continual Prompt Replay50.5933.2026.6036.80-2.062.58
CPR (Pass-rate Selection)48.7429.8027.2035.25-3.193.39
Replay + Previous-policy KL47.3627.8022.2032.45-2.082.86
LwF50.5230.8026.0035.77-2.403.18
EWC48.5933.6024.2035.46-2.963.47
CPO44.6633.6028.0035.42-4.334.87
CLD (Ours)52.6936.0028.4039.03+0.290.10

CLD achieves the best ACC, BWT, and AF among all compared methods. Although the other CL methods reduce forgetting in some settings, they do not explicitly preserve historical decoding orders. CLD directly addresses this problem in dLLMs, leading to better performance and less forgetting under our evaluation settings.

We have included these additional comparisons in Table 1 (Lines 349–355) and Section 5 (Lines 368–372) of the revised paper.


We sincerely thank you again for your valuable suggestions, which have greatly helped us improve and strengthen our paper.


References

[1] Luo et al. Continual Reasoning Gym: Diagnosing and Harnessing Shared Reasoning in Continual RLVR. arXiv:2608.18574, 2026.

[2] Li and Hoiem. Learning without Forgetting. ECCV, 2016.

[3] Kirkpatrick et al. Overcoming Catastrophic Forgetting in Neural Networks. PNAS, 2017.

[4] Luo et al. RL Forgets! Towards Continual Policy Optimization. arXiv:2607.04364, 2026.

[5] Buzzega et al. Dark Experience for General Continual Learning: A Strong, Simple Baseline. NeurIPS, 2020.

[6] Wang et al. Orthogonal Subspace Learning for Language Model Continual Learning. Findings of EMNLP, 2023.

[7] Rolnick et al. Experience Replay for Continual Learning. NeurIPS, 2019.

[8] Kaplanis et al. Policy Consolidation for Continual Reinforcement Learning. ICML, 2019.

Dear Reviewers,

With only two days remaining in the discussion period, we kindly encourage reviewers who have not yet responded to the authors' rebuttal or participated in the discussion to do so.

We would greatly appreciate it if you could review the authors' responses and engage in further discussion with them, particularly regarding whether your concerns have been adequately addressed.

Thank you very much for your time and valuable contributions!

AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.