- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper points out that using only confidence to do decoding for masked diffusion models is not enough. It is helpful to incorporate historical information along the decoding trajectory to improve decoding quality and speed. They present TACG, a decoding method based on this insight. TACG surpasses several baselines on benchmarks for various commonly-used masked diffusion models.
Critical strengths
- The experiments are comprehensive. Empirical results show improvements over selected baselines.
- The method combines signed temporal logit contrast and explicit token-identity persistence. This is well-motivated.
Critical weaknesses
- Similar methods have been proposed last year and the paper does not seem to compare with them. See question 1.
Questions for authors
- There might be missing citations. For example, the icml 2026 paper https://arxiv.org/abs/2512.02044 also proposes a trajectory-aware decoding method. It may be helpful to include a comparison table to compare TACG with relevant methods.
- I am interested to know the authors' opinion on using TACG in RL rollouts. And while being a training-free method, I wonder if specialized training can further improve TACG's performance.
3 comments
[Q2] Using TACG in RL rollouts and potential gains from specialized training.
Thank you for raising both questions—we evaluated TACG under the online group rollout sampling protocol of d1 (diffu-GRPO, NeurIPS 2025) on an uncontended NVIDIA H200 GPU (25 GSM8K prompts group size = 100 stochastic trajectories at , ABBA-interleaved timing):
| Backbone & Online Rollout Sampler () | Trajectory NFE | Latency / Group | Rollout Speedup (vs. d1 / Native) | Rollout Pass@1 |
|---|---|---|---|---|
LLaDA-8B — Original d1 Fixed (256 steps) | 256.00 | 26.87 s | 1.00× / — | 63.0% |
| LLaDA-8B — Native Confidence () | 82.69 | 10.39 s | 2.59× / 1.00× | 63.0% |
| LLaDA-8B — TACG Rollout (Zero-retuning) | 72.04 (-12.9%) | 9.50 s | 2.83× / 1.09× | 69.0% (+6.0 pp) |
LLaDA-1.5 (VRPO) — Original d1 Fixed (256 steps) | 256.00 | 32.72 s | 1.00× / — | 67.0% |
| LLaDA-1.5 (VRPO) — Native Confidence () | 80.15 | 10.57 s | 3.10× / 1.00× | 69.0% |
| LLaDA-1.5 (VRPO) — TACG Rollout (Fast / Conservative) | 68.44 / 74.82 | 9.47 / 10.14 s | 3.46× / 1.12× (fast) | 70.0% / 71.0% (+2.0 pp) |
Because online rollouts dominate GRPO wall-clock cost and synchronize on the slowest trajectory in each group (), TACG accelerates group rollout generation by 2.83×–3.46× over fixed-step d1 and 1.09×–1.12× over Native Confidence without modifying d1's loss formulation (which scores completed responses via resampled masked-ELBO likelihoods; as in confidence-based rollouts, altering commit order shifts the behavior rollout distribution while leaving the masked-ELBO update interface unchanged, and pairs even more cleanly with ratio-free objectives such as wd1; we leave closed-loop RL training dynamics to future work). At the same time, filtering unstable commits during stochastic exploration () raises LLaDA-8B rollout Pass@1 by +6.0 pp (63.0% 69.0%) and group Pass@4 by +12.0 pp (72.0% 84.0%). Moreover, across all 2,240 items of our 4-task benchmark suite on the RL post-trained LLaDA-1.5 checkpoint, conservative TACG reaches 50.41% / 72.09 NFE, improving both accuracy (+0.37 pp) and NFE (-5.7%) over the LLaDA-1.5 default (50.04% / 76.43 NFE).
Regarding specialized training, our results point to clear headroom: comparing our single shared vector (49.62%–49.90%) against task-specific operating points (51.02%) on LLaDA-8B—and zero-retuning (49.62%) against conservative gating (50.41%) on LLaDA-1.5—shows roughly 0.8–1.4 pp of domain- and calibration-dependent headroom. Learning a lightweight context-aware calibration head over TACG's white-box statistics to adapt per block could capture this gain automatically while retaining full interpretability; likewise, adding cross-step consistency regularization during post-training to penalize high-confidence token flips would align the backbone's confidence calibration directly with trajectory-aware commitment at inference time. In the revised manuscript, we have added the online RL rollout and LLaDA-1.5 experiments to Section 5.8 and Appendix E.3 (Table 21) and noted lightweight calibration and RL training extensions in Section 6 (which we will expand further in the final version).
Thank you for the detailed rebuttal and presenting the new experiment results. It is inspiring to know that a good inference-time method like TACG could be beneficial in RL rollout. My main concern is still partially unresolved though - like the other two reviewers mentioned, several previous works also studied using history information to guide dllm inference. TACG's main contribution is a unifying commitment framework. But it seems like there is currently insufficient theoretical or empirical evidence to support this framework, to show that it is superior to previous works. I would lean toward keeping my score unchanged at this stage.
Thank you for bringing this relevant ICML 2026 work (Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models, proposing Coherent Contextual Decoding, CCD / CCD-DS, arXiv:2512.02044) to our attention. In the revised manuscript, we have cited and discussed CCD / CCD-DS in Section 3 (Related Work) and added the structured comparison table below in Appendix B (Table 10; we will also move the table into Section 3 in the final version as space permits), situating it alongside concurrent commitment methods (VSB, TraceLock) and the trajectory-based baselines evaluated under our unified protocol in Table 1 (where TACG achieves 49.62%–49.90% / 67.87–67.96 NFE shared and 51.02% / 67.82 NFE default on LLaDA-8B, compared with DSB at 49.35% / 75.15 NFE, KLASS at 49.03% / 128.45 NFE, AHD at 47.81% / 76.65 NFE, and Native Confidence at 48.00% / 76.59 NFE).
argmaxargmaxCCD-DS)argmaxargmax(learned commit gate)argmaxMechanistically, TACG differs from CCD / CCD-DS along two key axes. First, CCD aggregates historical contexts into a trajectory-rectified marginal distribution—coupling historical evidence with token selection—whereas TACG strictly decouples when to commit from what token to write, writing only the backbone's unmodified current-step
argmaxtoken (as Section 5.3, Table 2 shows, altering written tokens via historical logits hurts accuracy). Second, whileCCD-DSleverages cross-step top-1 consistency and marginal entropy to adapt the step-level unmasking budget, TACG enforces a per-position discrete persistence hard gate (streak≥m, HG) before ranking eligible candidates by candidate-conditioned positive EMA support growth (TILG) under a capped promotion budget (kmax). Within TACG's own component ablation on LLaDA-8B, removing this per-position HG gate while keeping continuous TILG promotions lowers 4-task accuracy from 49.62% to 48.70% (-0.92 pp), confirming that continuous temporal scores alone cannot replace position-wise discrete persistence filtering.