- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes SLICE, a training-free decoding method for Diffusion Language Models (DLMs). SLICE constructs a structural self-contrastive signal by skipping redundant layers to form a "weak branch," which is then contrasted with the full model to guide decoding. The method is evaluated on LLaDA models across multiple benchmarks, showing solid performance improvements.
Critical strengths
- Solid Empirical Results: The experiments are comprehensive and rigorous. The proposed method achieves strong and consistent performance improvements across various benchmarks (understanding, math, coding) compared to the baselines.
- Application in DLMs: To the best of my knowledge, this is the first work to adapt the concepts of "layer contrast" and "layer redundancy" from autoregressive models into DLMs.
Critical weaknesses
- Limited Methodological Novelty: The core ideas of layer contrast and layer redundancy have been extensively explored in the autoregressive literature. Simply transplanting these existing concepts to DLMs, without fundamentally new mechanisms, makes the theoretical innovation somewhat incremental. I suggest the authors elaborate more on why this specific adaptation to DLMs is non-trivial compared to AR models.
- Incomplete Cost Analysis & Scaling Concerns: While the authors claim high computational efficiency, the cost analysis seems incomplete. Does the reported latency explicitly include the overhead of the initial forward pass for layer similarity probing, the layer pruning operations, and the additional GPU memory usage? More importantly, how does this overhead scale with longer sequence lengths? I hope the authors can provide a more holistic evaluation of the total cost to justify whether the performance gains are worth the computational trade-offs at scale.
- Lack of Evaluation on Advanced/Larger Models: Since the proposed method operates purely at the decoding stage and is training-free, it is highly encouraged to validate it on more advanced or larger DLMs. The current experiments are limited to 8B. Testing on newer models, such as the LLaDA 2.x series (especially the mini versions), would significantly strengthen the claim of the method's generalizability and robustness.
Questions for authors
- Clarify the novelty.
- Provide more cost analysis.
- Evaluate more advanced/larger models.
8 comments
Response to Reviewer GFaq (2/2)
W2/Q2: Incomplete Cost Analysis & Scaling Concerns.
We sincerely thank the reviewer for this suggestion and acknowledge that we did not clearly explain how latency was measured or quantify the additional GPU memory usage.
(a) Cost analysis. We do not claim that SLICE is computationally cheaper than standard decoding: Table 2 explicitly reports generation latency of 1.68×/1.81× relative to Standard. Our efficiency claims refer to the accuracy–cost trade-off relative to other inference-enhancement methods and compatibility with acceleration. In particular, Table 5 shows that Fast-dLLM+SLICE improves six-task average accuracy from 33.91% to 38.20% at 0.99× Standard latency. The reported wall-clock generation time includes prompt-time probing, similarity computation, layer selection and bypass operations, both decoding branches, and logit guidance. We have made this explicit in Table 2’s caption (lines 353–354) and Appendix A.4 (lines 855–864).
(b) Scaling Concerns. Both branches share model weights and the additional memory mainly comes from temporary probing and guidance tensors. Table 3 reports accuracy gains across generation lengths, and we have added the cost measurements below in Appendix C.7, Table 15.
| L / N | Standard latency (s/example) | SLICE latency (s/example) | Standard peak allocated (GiB) | SLICE peak allocated (GiB) |
|---|---|---|---|---|
| 128 / 64 | 2.254 | 3.787 | 15.047 | 16.252 |
| 256 / 128 | 6.069 | 10.439 | 15.077 | 16.716 |
| 512 / 256 | 19.680 | 34.431 | 15.137 | 17.240 |
| 1024 / 512 | 71.330 | 128.394 | 15.260 | 17.793 |
As generation length increases (128→1,024), SLICE peak memory grows only 1.095× (16.252→17.793 GiB). Thus, the additional memory grows much more slowly than the generation budget over this measured range. Relative latency remains 1.68–1.80× Standard. For tighter time budgets, Table 5 shows that Fast-dLLM+SLICE improves average accuracy from 33.91% to 38.20% at 0.99× Standard latency.
W3/Q3: Lack of Evaluation on Advanced/Larger Models.
We sincerely thank the reviewer for suggesting evaluations on newer and larger DLMs. We have evaluated SLICE on LLaDA2.0-mini, a newer MoE diffusion model with approximately 16B total parameters and 1.4B active parameters. SLICE improves ARC-C accuracy under both evaluated decoding budgets. We have added these results in Appendix C.6, Table 14:
| B / L / N | Benchmark | Standard (%) | SLICE (%) | Gain (pp) |
|---|---|---|---|---|
| 32 / 128 / 64 | ARC-C | 23.12 | 26.40 | +3.28 |
| 32 / 512 / 256 | ARC-C | 89.42 | 92.36 | +2.94 |
These results demonstrate that SLICE also works on a newer MoE model with more total parameters than the original 8B checkpoints. For cross-family generalization, our response to Reviewer 44ks, W2(b), reports improvements on all three evaluated Dream benchmarks.
We hope the clarified methodological distinctions, additional cost measurements, and LLaDA2.0-mini results address the reviewer’s three main questions. We invite the reviewer to reconsider the assessment in light of these clarifications and results.
Thank you for the authors' efforts and detailed responses. Given the limited granularity of the rating scale, my current score is raised to 2.5. I would be willing to increase it to 3 if the following concerns are addressed. I encourage the AC to take this assessment into consideration.
-
Evaluation on LLaDA-2.0-mini: The current evaluation remains limited. Since SLICE is training-free and operates entirely at inference time, I encourage the authors to additionally evaluate it at least on MATH-500, HumanEval, and Countdown.
-
Layer selection: Please clarify whether the skipped layers are the same as those selected for ARC-C, to better assess the generalizability of the layer selection strategy.
-
Inference overhead: Please report the inference-time overhead on LLaDA-2.0-mini.
Response to Reviewer GFaq (1/2)
We sincerely thank the reviewer for the constructive suggestions. We provide detailed explanations to address your concerns below.
W1/Q1: Limited Methodological Novelty.
We sincerely thank the reviewer for encouraging us to clarify how SLICE differs from prior AR methods and why its adaptation to DLMs requires specific design choices. SLICE contributes both a structural reference construction and a guidance mechanism tailored to masked diffusion decoding.
(a) Structural reference construction. SLICE uses layer similarity to select prompt-adaptive skips and constructs a shared-weight weak branch that retains downstream computation. This turns redundancy into a mechanism for generating contrastive guidance. In comparison, AR layer-contrast methods such as DoLa contrast intermediate- and final-layer predictions.
(b) Guidance tailored to masked diffusion decoding. In DLMs, guidance affects both token predictions and confidence-based position selection, changing subsequent contexts and decoding trajectories. SLICE evaluates both branches at the same partially masked state, independently standardizes their logits, then clips and rescales the contrast before token commitment. These operations address branch-scale differences and control the correction’s magnitude, making structural contrast usable within iterative, multi-position decoding. We have clarified DLM-specific guidance in Section 3.3 (lines 174–187) and the structural-reference rationale in Appendix A.3 (lines 835–841).