- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper focus on extending dLLMs to agentic settings where accuracy–parallelism trade-off is important in such long-horizon tasks. It proposes a two-stage post-training for dLLM agents. Stage 1 is on-policy RL with two changes: a turn whose action is invalid gets its share of the episode reward cut, and the KL penalty to a reference model is replaced by GSPO-style narrow clipping with a dual-clip floor. Stage 2 is self-distillation: the model logs the order in which it decoded tokens in its own successful episodes and learns to fill in the later ones in one pass from the earlier ones, with a curriculum that raises how many it fills at once.
Critical strengths
- The paper focuses on an important problem of extending dLLMs to agentic settings
- The engineering is solid
Critical weaknesses
- The central claim rests on an unfair comparison. RL plus distillation is compared against RL only. Since AUP is roughly accuracy times tokens per forward, so the AUP gap is the Stage 2's parallelism that no baseline receives. The performance of RL stage alone is within noise of GDPO
- The novelty is limited. Each stage is an existing recipe. Stage 2 has limited novelty given d3LLM v1's pseudo-trajectory distillation. v2 mainly swaps the ground-truth targets for the model's own successful rollouts and applies it per turn. Stage 1 has limited novelty given GSPO's narrow clipping without KL, ESPO-style length-normalized ratios, and a validity penalty that the paper's related work credits to LLaDA 2.2. Additionally, the 4.4x speedup in abstract and the 9.1x in introductoin over the AR baseline are tokens-per-forward ratios while the wall-clock measurement in Table 5 is 1.25x over Qwen3-8B and 1.5x over the best dLLM baseline
Questions for authors
- Could the authors report wall-clock tokens per second for every row of Table 1 at batch sizes 8 to 32, stating whether prefill and environment time are included?
- Do the baselines receive any validity or format signal?
4 comments
Dear Reviewer JVVy, thank you again for your time.
As the discussion phase is closing soon, we would appreciate to hear if our response has addressed your concerns, and if there is anything else to discuss.
We sincerely hope that our clarifications have been helpful and would be grateful if you could take them into account in your final assessment.
Best Regards,
Authors.
I thank the authors for the response and the new experiment, and I raise my rating given the authors' effort. It remains below acceptance because both my initial concerns are not addressed. First, the added results in Table 3 combining distillation and RL for baselines cover 1 of the roughly 10 experiment settings. This is understandable given the short rebuttal window, but in my view the central claim needs this in every setting to make comparison fair (RL + distillation vs RL baselines is not fair, which is my initial concern that remains). This does not appear easy to address in the rebuttal, but to me is crucial to support the central claim. Second, the novelty response confirms incremental changes over LLaDA2.2 and d3LLM v1.
Thanks for your reply, and we'd like to take this last minute to further clarify the following points.
-
Regarding fairness. In our previous rebuttal, we showed that for the SDAR-8B (b=4) model, our method achieves higher AUP than all RL baselines combined with distillation (please refer to A3 in our previous reply). This result shows that the advantage of our method persists even when the baselines also use distillation. We are working on running RL + distillation for every baseline, but this is computationally expensive, especially for an academic lab. These experiments will take time to complete, and we will include the results in the revised version. More importantly, since distillation is an integral component of our proposed method, we believe comparing our full method against the original baselines is, in fact, reasonable.
-
Regarding novelty. We respectfully clarify that our contribution lies not in any single component, but in a unified pipeline that addresses the accuracy–parallelism trade-off of agentic dLLMs. Each component differs non-trivially from prior work: per-turn validity gating (vs. the additive reward in LLaDA2.2) and on-policy multi-turn self-distillation (vs. offline distillation in d3LLM v1). Together, these yield a notable AUP gain (up to 25%) over the strongest baseline.
As the discussion period is coming to an end, we would be glad to clarify any remaining questions. Thank you again for your time and feedback.
We sincerely appreciate the reviewer's comments. Below, we address each of your concerns.
Q1. "wall-clock TPS at batch sizes 8 to 32..."
A1. Thank you for this question. Following your suggestion, we extend the wall-clock comparison in Table 6 to batch sizes 2 and 4. The setting is the same as Table 6. Results (TPS/TPF) are presented below:
As shown in the table, d3LLM-v2 achieves about 1.5× the TPS of the fastest dLLM baseline at all batch sizes, and is 25.1% and 9.5% faster than Qwen3-8B at batch sizes 1 and 2. However, its advantage over the AR baseline narrows as the batch size increases.
We remark that this is a known limitation of current dLLMs, rather than a specific issue of our d3LLM-v2. Since dLLMs decode multiple tokens per forward pass, they reach the compute-bound regime at smaller batch sizes than AR models, so their speed advantage narrows under high concurrency (also reported in SDAR & Fast-dLLM). We leave this as a direction for future work.
Q2. Do the baselines receive any validity or format signal?
A2. Thank you for your comment. In the main comparison (Tab. 1), validity-based reward gating is a component of d3LLM-v2, while the baselines are trained with their original objectives. To isolate the effect of the validity signal, our Tab. 2 already includes a baseline that receives the validity signal. We restate the results on WebShop with SDAR-8B (b=4) below:
Q3. "unfair comparison..."
A3. Thank you for this question.
We respectfully clarify that the comparison is fair: the distillation stage is an integral part of our method, and our contribution lies in designing an effective pipeline that jointly optimizes the accuracy-parallelism trade-off of agentic dLLMs.
Following your suggestion, we further apply the same online distillation stage to the existing RL checkpoints of each baseline. The results on WebShop with SDAR-8B (b=4) are presented below:
As shown in the table, distillation raises the TPF of all baselines, but causes clear accuracy drops. In contrast, d3LLM-v2 largely preserves accuracy and achieves 25% higher AUP than the best baseline with distillation.
Q4. Regarding novelty.
A4. Thank you for this question. We would like to take this opportunity to clarify the novelty of d3LLM-v2.
Accuracy–parallelism trade-off of agentic dLLMs. Our core contribution is addressing the inherent accuracy–parallelism trade-off of dLLMs in long-horizon agentic tasks. By combining reward-gated RL for accuracy with trajectory-based online self-distillation for parallelism, d3LLM-v2 pushes the accuracy–parallelism frontier of agentic dLLMs.
Technical designs tailored to agentic dLLMs. To push the accuracy–parallelism frontier, we propose RL and distillation tailored for agentic dLLMs. For accuracy, our reward-gated RL suppresses invalid actions caused by parallel decoding; unlike the additive format-validity reward in LLaDA2.2, it gates outcome reward of each turn by its action validity, so that invalid turns are not reinforced even within successful trajectories. For parallelism, our on-policy online self-distillation learns from the model's own successful trajectories, which goes beyond the offline distillation in d3LLM.
Open-source agentic dLLM training framework. Extending dLLM's RL training/distillation to on-policy multi-turn agentic settings is not a trivial extension. To the best of our knowledge, d3LLM-v2 is the first open-source framework for on-policy agentic RL training of dLLMs. It supports existing dLLM RL methods under a unified interface, providing a useful testbed for agentic dLLMs. We will give proper credit to related work (e.g., LLaDA2.2) and clarify our contributions in the revision. Thank you!
We sincerely appreciate your feedback. We have revised the manuscript accordingly. If our responses have addressed your concerns, we would be grateful if you would reconsider the score, and we would be glad to discuss any further questions. Thank you!