acceptodds
Under review as a conference paper at ICLR 2027

d3LLM-v2: Ultra-Fast Agentic Diffusion LLM via Reward Gating and Online Distillation

Abstract

Diffusion large language models (dLLMs) have emerged as a promising alternative to autoregressive (AR) LLMs, offering potential advantages such as parallel decoding and global context modeling. While many efforts have been devoted to improving dLLMs and enhancing their reasoning capabilities, extending dLLMs to *agentic settings* that involve multiple interaction turns and tackling the inherent *accuracy–parallelism trade-off* in such long-horizon agentic tasks, remain underexplored. In this work, we study on-policy training for agentic dLLMs and propose d3LLM-v2 (*Adaptive-Distilled Diffusion Large Language Model*) framework, striking a balance between accuracy and parallelism in long-horizon agentic tasks. *(i)* We introduce *reward-gated agentic RL* for dLLMs: due to bidirectional attention and parallel decoding, dLLMs are more likely to generate invalid actions or failed tool calls; moreover, ELBO-based KL regularization provides only a coarse approximation, which becomes unreliable across multi-turn interactions. To address this, we propose *validity-based reward gating* that penalizes invalid actions, paired with implicit policy regularization for agentic dLLMs. *(ii)* We further introduce *trajectory-based online self-distillation* to improve the efficiency of dLLMs in an online manner, where the model serves as its own teacher and learns to reproduce its decoding outputs in fewer diffusion steps along its own trajectories, guided by an easy-to-hard curriculum mechanism. Experiments demonstrate that d3LLM-v2 achieves up to 4.4 speedup while maintaining competitive task accuracy.

Primary area:
foundation or frontier models, including LLMs
Keywords:
diffusion LLM, agentic RL, efficient inference, diffusion language model, parallel decoding

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JVVyOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper focus on extending dLLMs to agentic settings where accuracy–parallelism trade-off is important in such long-horizon tasks. It proposes a two-stage post-training for dLLM agents. Stage 1 is on-policy RL with two changes: a turn whose action is invalid gets its share of the episode reward cut, and the KL penalty to a reference model is replaced by GSPO-style narrow clipping with a dual-clip floor. Stage 2 is self-distillation: the model logs the order in which it decoded tokens in its own successful episodes and learns to fill in the later ones in one pass from the earlier ones, with a curriculum that raises how many it fills at once.

Critical strengths
  • The paper focuses on an important problem of extending dLLMs to agentic settings
  • The engineering is solid
Critical weaknesses
  • The central claim rests on an unfair comparison. RL plus distillation is compared against RL only. Since AUP is roughly accuracy times tokens per forward, so the AUP gap is the Stage 2's parallelism that no baseline receives. The performance of RL stage alone is within noise of GDPO
  • The novelty is limited. Each stage is an existing recipe. Stage 2 has limited novelty given d3LLM v1's pseudo-trajectory distillation. v2 mainly swaps the ground-truth targets for the model's own successful rollouts and applies it per turn. Stage 1 has limited novelty given GSPO's narrow clipping without KL, ESPO-style length-normalized ratios, and a validity penalty that the paper's related work credits to LLaDA 2.2. Additionally, the 4.4x speedup in abstract and the 9.1x in introductoin over the AR baseline are tokens-per-forward ratios while the wall-clock measurement in Table 5 is 1.25x over Qwen3-8B and 1.5x over the best dLLM baseline
Questions for authors
  • Could the authors report wall-clock tokens per second for every row of Table 1 at batch sizes 8 to 32, stating whether prefill and environment time are included?
  • Do the baselines receive any validity or format signal?
4 comments

We sincerely appreciate the reviewer's comments. Below, we address each of your concerns.

Q1. "wall-clock TPS at batch sizes 8 to 32..."

A1. Thank you for this question. Following your suggestion, we extend the wall-clock comparison in Table 6 to batch sizes 2 and 4. The setting is the same as Table 6. Results (TPS/TPF) are presented below:

Methodbs=1bs=2bs=4
Qwen3-8B (AR)221.26 / 1.000416.41 / 1.000755.34 / 1.000
SDAR (+prompt)181.22 / 1.447263.79 / 1.526373.38 / 1.738
d1181.30 / 1.550278.96 / 1.524439.26 / 1.591
BGPO186.24 / 1.624292.33 / 1.562454.62 / 1.573
GDPO180.12 / 1.596287.24 / 1.598448.94 / 1.654
ESPO172.97 / 1.494299.87 / 1.649469.58 / 1.657
d3LLM-v2 (ours)276.71 / 2.944455.95 / 3.081724.87 / 3.052

As shown in the table, d3LLM-v2 achieves about 1.5× the TPS of the fastest dLLM baseline at all batch sizes, and is 25.1% and 9.5% faster than Qwen3-8B at batch sizes 1 and 2. However, its advantage over the AR baseline narrows as the batch size increases.

We remark that this is a known limitation of current dLLMs, rather than a specific issue of our d3LLM-v2. Since dLLMs decode multiple tokens per forward pass, they reach the compute-bound regime at smaller batch sizes than AR models, so their speed advantage narrows under high concurrency (also reported in SDAR & Fast-dLLM). We leave this as a direction for future work.


Q2. Do the baselines receive any validity or format signal?

A2. Thank you for your comment. In the main comparison (Tab. 1), validity-based reward gating is a component of d3LLM-v2, while the baselines are trained with their original objectives. To isolate the effect of the validity signal, our Tab. 2 already includes a baseline that receives the validity signal. We restate the results on WebShop with SDAR-8B (b=4) below:

MethodACC↑TPF↑AUP↑
ESPO53.001.49478.78
ESPO + validity-based reward gating49.601.60179.41
d3LLM-v2 (RL stage only)59.801.68396.96
d3LLM-v2 (full)57.202.944168.1

Q3. "unfair comparison..."

A3. Thank you for this question.

  • We respectfully clarify that the comparison is fair: the distillation stage is an integral part of our method, and our contribution lies in designing an effective pipeline that jointly optimizes the accuracy-parallelism trade-off of agentic dLLMs.

  • Following your suggestion, we further apply the same online distillation stage to the existing RL checkpoints of each baseline. The results on WebShop with SDAR-8B (b=4) are presented below:

MethodACC (RL only)↑ACC↑TPF↑AUP↑
d1 + distillation50.2044.003.015132.66
BGPO + distillation52.6035.202.78697.17
GDPO + distillation58.0047.602.847134.19
ESPO + distillation53.0050.003.075133.81
d3LLM-v259.8057.202.944168.1

As shown in the table, distillation raises the TPF of all baselines, but causes clear accuracy drops. In contrast, d3LLM-v2 largely preserves accuracy and achieves 25% higher AUP than the best baseline with distillation.


Q4. Regarding novelty.

A4. Thank you for this question. We would like to take this opportunity to clarify the novelty of d3LLM-v2.

  • Accuracy–parallelism trade-off of agentic dLLMs. Our core contribution is addressing the inherent accuracy–parallelism trade-off of dLLMs in long-horizon agentic tasks. By combining reward-gated RL for accuracy with trajectory-based online self-distillation for parallelism, d3LLM-v2 pushes the accuracy–parallelism frontier of agentic dLLMs.

  • Technical designs tailored to agentic dLLMs. To push the accuracy–parallelism frontier, we propose RL and distillation tailored for agentic dLLMs. For accuracy, our reward-gated RL suppresses invalid actions caused by parallel decoding; unlike the additive format-validity reward in LLaDA2.2, it gates outcome reward of each turn by its action validity, so that invalid turns are not reinforced even within successful trajectories. For parallelism, our on-policy online self-distillation learns from the model's own successful trajectories, which goes beyond the offline distillation in d3LLM.

  • Open-source agentic dLLM training framework. Extending dLLM's RL training/distillation to on-policy multi-turn agentic settings is not a trivial extension. To the best of our knowledge, d3LLM-v2 is the first open-source framework for on-policy agentic RL training of dLLMs. It supports existing dLLM RL methods under a unified interface, providing a useful testbed for agentic dLLMs. We will give proper credit to related work (e.g., LLaDA2.2) and clarify our contributions in the revision. Thank you!


We sincerely appreciate your feedback. We have revised the manuscript accordingly. If our responses have addressed your concerns, we would be grateful if you would reconsider the score, and we would be glad to discuss any further questions. Thank you!

Dear Reviewer JVVy, thank you again for your time.

As the discussion phase is closing soon, we would appreciate to hear if our response has addressed your concerns, and if there is anything else to discuss.

We sincerely hope that our clarifications have been helpful and would be grateful if you could take them into account in your final assessment.

Best Regards,

Authors.

I thank the authors for the response and the new experiment, and I raise my rating given the authors' effort. It remains below acceptance because both my initial concerns are not addressed. First, the added results in Table 3 combining distillation and RL for baselines cover 1 of the roughly 10 experiment settings. This is understandable given the short rebuttal window, but in my view the central claim needs this in every setting to make comparison fair (RL + distillation vs RL baselines is not fair, which is my initial concern that remains). This does not appear easy to address in the rebuttal, but to me is crucial to support the central claim. Second, the novelty response confirms incremental changes over LLaDA2.2 and d3LLM v1.

Thanks for your reply, and we'd like to take this last minute to further clarify the following points.

  • Regarding fairness. In our previous rebuttal, we showed that for the SDAR-8B (b=4) model, our method achieves higher AUP than all RL baselines combined with distillation (please refer to A3 in our previous reply). This result shows that the advantage of our method persists even when the baselines also use distillation. We are working on running RL + distillation for every baseline, but this is computationally expensive, especially for an academic lab. These experiments will take time to complete, and we will include the results in the revised version. More importantly, since distillation is an integral component of our proposed method, we believe comparing our full method against the original baselines is, in fact, reasonable.

  • Regarding novelty. We respectfully clarify that our contribution lies not in any single component, but in a unified pipeline that addresses the accuracy–parallelism trade-off of agentic dLLMs. Each component differs non-trivially from prior work: per-turn validity gating (vs. the additive reward in LLaDA2.2) and on-policy multi-turn self-distillation (vs. offline distillation in d3LLM v1). Together, these yield a notable AUP gain (up to 25%) over the strongest baseline.

As the discussion period is coming to an end, we would be glad to clarify any remaining questions. Thank you again for your time and feedback.

Reviewer_8PDUOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addresses the accuracy-parallelism trade-off efficiently in long-horizon multi-turn diffusion LLM agentic tasks in on-policy methods by using: (1) Validity-based reward-gated RL that discounts the propagated final reward for invalid turns (by (1−λ)(1-\lambda)) and (2) Trajectory-based online distillation, which uses self-distillation loss on the generated trajectories for few-step generation, along with easy-to-hard curriculum on masking ratios (low masking rate to high masking rate). Another interesting trick is implicit regularization using dual-clipping with any KL-term with a reference model. This novel framework is termed d3LLM-v2 (Adaptive-Distilled Diffusion Large Language Model), and is shown to achieve high speed-ups (4.4x) for almost the same performance.

Critical strengths
  • Clear and crisp presentation.
  • Substantive empirical results to support the claim (Table 1 and Figure 2).
Critical weaknesses

None.

Questions for authors
  • Are the clipped GRPO loss and the distillation losses applied together?
  • Can you report two new rows of Table 1 with: (i). Only validity-gated rewards w/o distillation and (ii). Only distillation with no gating (λ=0\lambda=0)?
2 comments

We sincerely appreciate the reviewer's constructive comments and positive rating. Below, we address your questions.

Q1. Are the clipped GRPO loss and the distillation losses applied together?

A1. Thank you for this question. We clarify that the two losses are applied in two sequential stages. In the first stage, we perform agentic RL with the clipped objective and reward gating to improve accuracy. In the second stage, starting from the RL checkpoint, we perform online self-distillation with only the distillation loss to improve parallelism.


Q2. Can you report two new rows of Table 1 with: (i). Only validity-gated rewards w/o distillation and (ii). Only distillation with no gating (lambda=0)?

A2. Thank you for the constructive suggestion. Following your advice, we report these two settings on WebShop with SDAR-8B (b=4). For reference, we also include the RL stage without gating (λ=0) from Table 4. For the distillation-only setting, we apply the same online self-distillation to an RL checkpoint trained without reward gating (ESPO, the first row of Table 2):

MethodGatingDistillationACC↑TPF↑AUP↑
RL only (λ=0)✗✗51.801.65687.08
(i) Validity gating only (λ=1)✓✗59.801.68396.96
(ii) Distillation only (no gating)✗✓49.203.066138.50
d3LLM-v2 (full)✓✓57.202.944168.1

As shown in the table, the two components play complementary roles: validity gating mainly improves accuracy, while distillation mainly improves parallelism. Without gating, distillation also reinforces invalid turns inside successful trajectories, leading to a clear accuracy drop. By combining both, d3LLM-v2 achieves the highest AUP.


We hope the above responses have adequately addressed your concerns. Thank you again for your constructive comments and positive rating!

Reviewer_8PDUOpenReview ↗

Acknowledgement

Thank you for the extensive experiments, which strengthened my views on the paper. I keep my positive score.

Reviewer_f5jKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the accuracy–parallelism trade-off for diffusion language models in multi-turn agentic tasks and proposes a two-stage training framework. The first stage discounts turn-level rewards according to action validity and replaces an explicit reference-policy KL penalty with narrow asymmetric clipping and dual clipping. The second stage performs online self-distillation using successful trajectories generated by the current policy: it records the denoising order, constructs intermediate masked states, and progressively increases the masking ratio through an easy-to-hard curriculum.

Evaluation covers WebShop, Search-R1, and ALFWorld, with the latter reported in the appendix. The five evaluated model configurations consist of different model sizes and block sizes within SDAR and d3LLM-LLaDA, rather than five independent model families. The principal metrics are task accuracy, tokens per forward pass (TPF), and Accuracy Under Parallelism (AUP). The reported ACC and TPF operating points use a confidence threshold of 0.95, whereas AUP is computed separately by sweeping thresholds from 0.95 to 1.00.

The results show substantial improvements in TPF and AUP, although the method does not consistently achieve the highest task accuracy. On WebShop with SDAR-8B and block size 4, it obtains 57.20% ACC, 2.944 TPF, and 168.1 AUP, compared with 58.00%, 1.596, and 93.41 for GDPO (Table 1). In the wall-clock experiment on one B200 GPU with batch size 1, it achieves 276.71 tokens per second, approximately 1.49× the fastest existing dLLM RL baseline and 1.25× the AR baseline (Table 5). These measurements support a decoding-throughput improvement, while the larger TPF ratios should be distinguished from reductions in complete task latency.

Critical strengths

1. The framework addresses concrete agent failure modes with complementary training components

Validity-based gating directly addresses the credit-assignment problem in which invalid intermediate actions within successful trajectories can still receive positive reinforcement. Successful-trajectory distillation then combines environment feedback with the model’s denoising order. The complete RL stage achieves 59.80% ACC and 1.683 TPF; adding the full distillation procedure yields 57.20% ACC and 2.944 TPF, increasing AUP from 96.96 to 168.1. The variants using trajectory masking or curriculum alone both obtain 48.80% accuracy, whereas the combination recovers accuracy to 57.20%. This supports their complementarity in the tested setting. The practical significance is that the framework translates parallel decoding into observable benefits in multi-turn tool interaction.

2. The evaluation includes diverse configurations, repeated runs, and measured throughput

Critical weaknesses

1. Insufficient matched baselines and overstated efficiency claims

The main efficiency gains arise after distillation, yet comparisons focus on RL baselines (d1, BGPO, GDPO, and ESPO), without applying the original d3LLM distillation or related methods such as COPSD to the same agentic RL checkpoint. Existing ablations do not sufficiently isolate the contributions of online trajectory updates, additional training, and success filtering.

Moreover, the reported maximum 4.4× and 9.1× speedups primarily reflect TPF ratios rather than measured wall-clock improvements. Actual throughput gains are considerably smaller (1.49× and 1.25×), while complete-episode latency, tool overhead, and time per successful task remain unreported. The ALFWorld results also show a notable success-rate reduction (69.53% vs. 77.34% for AR).

2. Insufficient evidence for the implicit regularization explanation

The claim that clipping bounds ELBO-ratio estimation errors is theoretically overstated. PPO-style clipping does not guarantee bounded policy ratios or KL divergence, and estimation errors may affect both surrogate gradients and clipping behavior. Dual clipping does not eliminate these discrepancies.

Although the experiments demonstrate improved stability, they do not isolate the effects of narrow clipping and KL regularization or directly measure ELBO estimation errors and policy divergence. Thus, the proposed explanation attributing instability to accumulated multi-turn ELBO-KL errors remains insufficiently supported.

Additionally, the reward-gating argument assumes conditions that do not necessarily hold for all-failure groups or general λ settings. The normalization procedure and zero-variance handling also require clarification.

Questions for authors

1. Matched distillation comparison: How does the method compare with static distillation, original d3LLM, or adapted COPSD under identical RL checkpoints, filtering, and total training budgets? Does the claimed budget matching include RL, rollout, and distillation costs?

2. Speedup claim: What metric and configuration support the reported 4.4× speedup? Can the authors provide accuracy-matched TPS and complete-episode latency, clarifying whether tool/environment waiting is included? If based only on TPF, the claim should be qualified.

3. Regularization mechanism: Can the authors compare narrow and standard clipping with/without KL under matched settings? Does the boundedness claim apply to objective terms, gradients, or actual policy distance? If unsupported theoretically, it should be framed empirically.

4. Implementation and evaluation: How are group statistics computed for variable-length trajectories, and how are all-failure groups, zero variance, and padding handled? Please clarify validity criteria, the WebShop goal split, and the AUP formula, thresholds, and integration boundaries to ensure reproducibility.

3 comments

We sincerely appreciate the reviewer's detailed comments. Below, we address each of your concerns.

Q1. Matched distillation comparison.

A1. Thank you for this question. We address the two parts below.

  • Budget matching. We clarify the budget at two levels. (i) In the RL stage, all methods start from the same checkpoint and use the same number of training steps and rollouts. (ii) d3LLM-v2 additionally performs the online self-distillation stage. To compare all methods under the same distillation stage, we add experiments in Table 3 of the revised manuscript, where we apply the same online self-distillation to each RL baseline.

  • Matched distillation comparison. Following your suggestion, we compare our online distillation with static distillation as in the original d3LLM. The results on WebShop with SDAR-8B (b=4) are:

MethodACC↑TPF↑AUP↑
RL checkpoint (no distillation)59.801.68396.96
RL + Static distillation59.601.946112.05
RL + Online self-distillation (d3LLM-v2)57.202.944168.1

As shown in the table, static distillation preserves accuracy but brings only limited gains in TPF, whereas our online distillation substantially increases TPF with a small accuracy drop and achieves 50% higher AUP, pushing the frontier of accuracy-parallelism trade-off. We have added these results to the revised manuscript.


Q2. Speedup claim.

A2. Thank you for raising this question. The 4.4× speedup is a speedup based on the TPF (tokens per forward) metric. For wall-clock latency, Table 6 reports the TPS (tokens per second) of all methods on WebShop with SDAR-8B (b=4), measured on a single NVIDIA B200 GPU with batch size 1. This measurement excludes prefilling and tool/environment waiting time. With comparable accuracy, d3LLM-v2 achieves 1.49× the TPS of the previous fastest dLLM. We have clarified this in the revised manuscript.


Q3. Insufficient evidence for the implicit regularization explanation.

A3. Thank you for this suggestion. Following your suggestion, we conduct a controlled comparison of the clipping range and the explicit KL penalty on WebShop with SDAR-8B (b=4). All variants share the same SFT initialization, reward gating, and training budget. Three of these configurations are already reported in the paper (Table 2 and Table 4), and we add the missing one (narrow clipping with KL):

ClippingExplicit KLACC↑TPF↑AUP↑
Standard✗53.801.59887.27
Standard✓49.601.60179.41
Narrow✓52.801.63286.97
Narrow (ours)✗59.801.68396.96

Regarding the boundedness claim, we clarify that it applies only to the per-turn objective term: for a given advantage A^\hat{A}, asymmetric clipping and the dual clip bound the resulting per-turn surrogate objective, rather than the ratio ρ\rho or the unclipped product ρA^\rho\hat{A}, so an estimation error in the ratio cannot amplify the contribution of any single turn to the objective beyond this range. However, this does not bound the ELBO estimation error, the gradient, or the actual policy distance, and the estimation error may also change which clipping branch is active; we therefore present implicit policy regularization as an empirically supported property, based on the comparison above, together with Table 2 and Table 4, showing that combining reward gating with implicit policy regularization leads to more stable training. We have revised Remark 1 for clarity.


Q4. Implementation and evaluation.

A4. Thank you for this question. We clarify each point below:

  • Group statistics for variable-length trajectories: All turns within a trajectory share the episode reward, gated by the validity of each action. Advantages are normalized over these turn-level rewards within the group.
  • All-failure / zero-variance groups: Following standard GRPO, groups with identical rewards (e.g., all failures) yield zero advantages and contribute no gradient.
  • Padding: Each update processes exactly one micro-batch of turn-level samples. Samples that do not fill a complete micro-batch are discarded, so no padding is used.
  • Validity criteria: An action is valid if the environment can parse it into an executable action (e.g., with well-formed action tags). Examples of invalid actions are given in Appendix B.6.
  • WebShop goal split: Goals 0-499 form the evaluation set and the rest form the training set. Training goals are randomly sampled, and evaluation uses all 500 evaluation goals.
  • AUP computation: We follow the AUP formulation of d3LLM with α=3\alpha=3, computed over confidence thresholds from 0.95 to 1.00 and integrated from TPF =0=0 to the largest TPF of each method.

We sincerely appreciate your helpful feedback. We have revised the manuscript accordingly. If our responses have addressed your concerns, we would be grateful if you consider raising the score, and we would be glad to discuss any remaining questions. Thank you!

Dear Reviewer f5jK,

Thank you again for your constructive review. As the discussion period is coming to a close, we would like to kindly follow up on whether our responses have addressed your concerns.

We sincerely hope that our clarifications have been helpful and would be grateful if you could take them into account in your final assessment. If any questions remain or further clarification would be helpful, please let us know. We would be happy to continue the discussion and improve our submission accordingly.

Best regards,

Authors

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns, particularly the matched distillation comparisons, the clarification of the speedup metrics, and the additional ablation study on clipping and KL regularization. These additions provide a clearer picture of the proposed method and strengthen the empirical evidence.

The responses have addressed several of my concerns, although some questions regarding the practical implications of the efficiency improvements and the supporting evidence for the proposed regularization mechanism remain. I appreciate the authors' constructive engagement and will take the clarifications and additional results into account in my final assessment.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.