- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces TRACELOCK, a learned token-commitment controller for frozen diffusion language models. Trained using agreement between intermediate proposals and final trace tokens, it combines contextual hidden-state features, a dynamic threshold, and a moving local window to determine which tokens to commit. Experiments evaluate quality–step tradeoffs and reuse across decoding configurations on LLaDA and Dream.
Critical strengths
- The method addresses an important problem: improving diffusion decoding without modifying the backbone.
- Evaluation covers mathematical reasoning, question answering, and code generation on two backbones.
- Cross-configuration experiments and single-token interventions provide useful evidence about controller reuse and commitment behavior.
Critical weaknesses
- Limited novelty over Learn2PD. Both methods learn token acceptance from agreement with final decoded tokens.
- Unclear efficiency objective. The paper should distinguish whether “efficiency” refers to fewer decoding steps, lower wall-clock latency, or improved quality at a fixed computational budget. These objectives are not equivalent, and the reported results do not consistently improve both quality and speed.
- Unclear acceleration at matched quality. TRACELOCK sometimes requires more steps and higher measured latency than existing accelerated decoders, particularly on LLaDA. Since quality also varies with decoding-window size, matched-quality latency comparisons and quality–latency curves across window sizes are needed to establish a better tradeoff.
- Missing relevant strong baselines such as CCD, DAPD, SlowFast.
- Limited generalization evidence. Generalization is demonstrated primarily across decoding configurations within the same backbone and related task families. Robustness to unseen domains and deployment-induced trajectory distribution shift remains unclear.
- Insufficient methodological details. Lines 216–222 do not adequately explain how high-quality training traces are obtained, including the decoding schedules and filtering criteria. Although Appendix K reports 8,000 traces per backbone, task proportions and filtering retention rates remain unspecified.
Accelerating Diffusion Large Language Models with SlowFast Sampling: The Three Golden Principles
DAPD: Dependency-Aware Parallel Decoding via Attention for Diffusion LLMs
Beyond Confidence: Adaptive and Coherent Decoding for Diffusion Language Models
Questions for authors
NAN
6 comments
Response 8Gby part 2
Robustness to commitment-induced trajectory shift
Appendix C.1 directly tests commitment-induced trajectory shifts through counterfactual interventions. From the same decoding state, we compare native generation with a continuation that additionally commits one TraceLock-selected token. This intervention changes a correct answer to an incorrect one in only 2/367 cases on LLaDA and 2/323 on Dream. Together with our full-rollout evaluations, these results support robustness to commitment-induced shifts under the evaluated conditions.
Training-trace construction
For each frozen backbone, we collect 8,000 retained complete decoding trajectories using its native confidence-based decoder with generation-length/window-size pairs of (256, 64), (192, 32), and (128, 32). We apply rejection-sampling-style filtering to the completed trajectories. For coding, we retain only trajectories whose generated programs pass the supplied test cases, rejecting 38% of candidates. For QA, we use LanguageTool-assisted grammatical and formatting checks to filter malformed or repetitive outputs, rejecting 9% of candidates. For math, we reject trajectories with repetitive outputs or without an extractable final answer in \boxed{} format. The final retained training set contains coding, math and QA trajectories in a 3:1:6 ratio.
We hope these additional experiments and clarifications address your concerns and clarify TraceLock’s contribution. We would be happy to answer any further questions or discuss any points that remain unclear.
Dear Authors,
Thank you for the response. The added baselines, the efficiency objective, the quality–latency results, and the training-trace details address my earlier questions.
My remaining concern is the method itself. TraceLock trains a small controller to decide whether to keep the frozen decoder’s proposed token, using whether that token survives to the end of the decoder’s own trace. This needs extra trace collection and training for each backbone. I did not see generalization beyond decoding settings on the same backbone, so maintaining such a controller in real use is less convincing if that generalization is weak. The accept-or-reject rule is also close to speculative decoding, and only deciding whether to keep the current token may not be the best policy. Better ways to choose the committed token are still worth studying. So I remain my scores.
Response 8Gby part 1
Thank you for your review and for recognising the importance of the problem and the breadth of our experiments. We have conducted additional experiments, including comparisons with the baselines you suggested, and address your concerns below.
Novelty over Learn2PD
Although both methods learn from agreement with final decoded tokens, they learn different commitment policies. Learn2PD uses a fixed-width, confidence-based filter. TraceLock instead uses contextual hidden-state features and a dynamic threshold conditioned on the current decoding state. This allows both token scores and the acceptance boundary to adapt throughout decoding, drawing on information beyond scalar token confidence. Figure 8 illustrates how TraceLock’s scores differ from token confidence.
The practical consequence is a reusable controller: TraceLock accommodates changes in decoding window and generation length using a single checkpoint per backbone. Under Learn2PD’s standard training protocol, changing the filter’s block size requires collecting new traces and retraining. Table 2 evaluates reuse without retraining when the decoding window is enlarged, comparing TraceLock with the original Learn2PD filter applied independently to smaller subwindows. Figure 5 further demonstrates reuse across generation lengths and window sizes on HumanEval.
Appendix Table 12 evaluates L2P-Hidden, which retains the Learn2PD-style blockwise interface but replaces confidence inputs with concatenated hidden-state features. On HumanEval with generation length 128 and block size 64, this substitution improves Pass@1 from 32.9% to 46.9% on Dream and from 24.4% to 28.1% on LLaDA. The full TraceLock system achieves 51.2% and 32.32%, respectively. These results support the value of hidden-state information, while showing that feature substitution alone does not match the full TraceLock system in these settings.
Efficiency definition and quality–latency tradeoff
Our objective is to improve the quality–latency Pareto frontier, with particular emphasis on achieving high task accuracy while controlling generation time. We therefore assess task accuracy alongside measured wall-clock latency, with decoding steps reported as a complementary measure.
We have added Figure 3 to the manuscript, comparing TraceLock, Learn2PD, and Fast-dLLM across acceptance thresholds and window sizes 32 and 64 on GSM8K, with generation length 256.
On Dream, TraceLock achieves the highest observed accuracy, reaching 80.80% in 7.73 seconds per sample. At window 64, every evaluated TraceLock configuration achieves higher accuracy and lower latency than every evaluated Learn2PD configuration. TraceLock also achieves 79.00% accuracy in 6.42 seconds, compared with Fast-dLLM’s highest observed accuracy of 76.42% in 6.56 seconds.
On LLaDA, TraceLock at window 32 achieves 79.33% accuracy in 7.39 seconds, exceeding the highest observed accuracy of both Fast-dLLM (78.70% in 8.00 seconds) and Learn2PD (78.30% in 7.44 seconds) across the two windows, at lower measured latency. At window 64, TraceLock further increases accuracy to 80.00% in 8.09 seconds.
These results demonstrate that TraceLock improves the observed quality–latency tradeoff in the higher-accuracy region on both backbones, surpassing the best accuracy achieved by Learn2PD and Fast-dLLM at lower measured latency, while also reaching the highest overall accuracy among the evaluated configurations.
Additional baselines
We added comparisons with the suggested baselines and additional methods: DAPD-Direct, CCD, CCD-DS, SlowFast, and Apple’s learned unmasking policy. All comparisons use Dream on GSM8K, with generation length 256 and window size 64. We report accuracy, average decoding steps, and generation time per sample below. Results are compared within each experimental setup.
TraceLock achieves the highest observed accuracy (80.80%), with 33.8% and 46.6% lower latency than Fast-dLLM and CCD, respectively. DAPD-Direct, CCD-DS, and SlowFast are faster but less accurate. Apple’s learned unmasking policy uses the fewest decoding steps, but has lower accuracy and higher measured latency than TraceLock.