- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes Temporal Delta Sparsity (TDS), a training-free inference acceleration method for diffusion language models. Rather than sparsifying activation magnitudes, TDS maintains reference activations and incrementally updates their most influential temporal differences. A weight-informed selection criterion allocates computation across tokens and channels, while an adaptive threshold controls sparsity over denoising steps. The paper establishes accumulator consistency and one-step error-bound minimization. Experiments on LLaDA-8B and Dream-7B show approximately 89% linear-layer MAC savings with near-dense average benchmark performance. Structured implementations additionally achieve 1.46–1.90× end-to-end speedups on a GSM8K subset.
Critical strengths
-
Strong empirical motivation for temporal sparsity. Figure 1 demonstrates substantially lower output approximation errors when sparsifying temporal changes rather than activation magnitudes.
-
Practical acceleration. Beyond MAC accounting, the paper demonstrates measurable end-to-end acceleration onLLaDA and Dream.
Critical weaknesses
- The last-refresh reference and sparse output-update mechanism follow Delta Networks [1], while Eventful Transformers [2] and Chipmunk [3] establish related selective temporal recomputation in Transformers and diffusion models. WINA [4] supplies the weight-informed scoring principle. The conceptual contribution of global token–channel allocation requires clearer differentiation.
- The adaptive controller fixes and . Their influence on threshold dynamics and the robustness of the reported accuracy–compute trade-off requires ablations.
- The strongest accuracy–sparsity results come from unstructured TDS, whereas measured speedups rely on structured variants. At 10% density, row selection underperforms unstructured TDS by 1.9 and 3.5 average points on LLaDA and Dream.
- The adaptive controller uses more MACs than fixed-budget selection at equal target density. As acknowledged by the authors, these comparisons do not isolate the benefits of adaptive allocation at matched realised compute.
- Evaluation focuses on one-token-per-step decoding. The two-token experiment reports 79.3% accuracy for fixed-budget TDS versus 84.0% for dense decoding at approximately 89% sparsity, leaving its effectiveness under more parallel decoding insufficiently characterized.
[1] D. Neil et al., “Delta Networks for Optimized Recurrent Network Computation”.
[2] M. Dutson et al., “Eventful Transformers: Leveraging Temporal Redundancy in Vision Transformers”.
[3] A. Silveria et al., “Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltas”.
[4] S. Chen et al., “WINA: Weight Informed Neuron Activation for Accelerating Large Language Model Inference”.
Questions for authors
- Could the authors provide end-to-end accuracy–latency trade-offs for TDS-row and TDS-chunk on mathematical reasoning and code-generation benchmarks beyond the 200-example GSM8K subset?
- How do end-to-end latency, additional reference-state memory, and accuracy scale with larger batch sizes and longer real generation contexts?
Thank you for the precise review.
1. What is new (W1)
We take the delta update from Delta Networks, the score from WINA (Sec. 3.2, Remark 1) and the chunk layout from Chipmunk; our row variant is close to Eventful Transformers' token gating, which we do not claim. Our contribution is a study of how to allocate delta computation in masked DLMs, with GPU variants. Each row swaps one choice inside TDS for the closest prior rule (GSM8K dev, ρ = 0.10 / 0.05; dense: LLaDA 81.6, Dream 87.9):
Fixed per-step budget; *with controller. A zero reference gives 0.0% (LLaDA) and 0.4% (Dream).
2. Controller hyperparameters (W2)
There is no α–γ sweep; we will state this as a limitation.
3. Structured vs unstructured (W3)
Agreed. The abstract will pair the row variant's speed with its own accuracy and drop "equal or higher accuracy": 89% sparsity, 45.9 vs 47.4 (LLaDA) and 56.3 vs 60.6 (Dream) on the four-task mean, and 1.90× / 1.46× end to end on 200 GSM8K test problems at batch 1. On the four-task mean it beats all six baselines on Dream (best: top-p, 54.8 at 21% sparsity) and is 0.6 / 0.2 points below TEAL (50%) / top-p (17%) on LLaDA.
4. Controller at matched compute (W4)
Agreed for equal target density, but Table 8 allows matched comparisons. At equal accuracy (82.4%) the controller needs about 18% fewer MACs (97.2% vs 96.6% sparsity); at 97.8% vs 97.6% it scores 82.4% vs 77.7%. Up to 97.6%, every fixed-budget point is matched or beaten by a controller point with at least as much sparsity. So the controller is never worse there, but beyond one standard error only near 98%. We will state exactly this.
5. Parallel decoding (W5)
The evidence is one point: 79.3% for TDS vs 84.0% for dense, both at two tokens per step (LLaDA dev, 89% sparsity, fixed budget). Prop. 3 suggests why: with a fixed threshold, refresh work is bounded by total activation change, which k tokens per step pack into about k times fewer steps, so per-step work would grow with k. TDS suits decoders that commit few tokens per step; the abstract and conclusion will state the evaluated setting (answers up to 256 tokens, blocks of 32, one token per step, runtime at batch 1).
Questions
Q1. Row accuracy on all four tasks is in Table 1: ρ = 0.1 gives 45.9 / 56.3 (four tasks) at 1.90× / 1.46× (GSM8K-200); ρ = 0.05 gives 41.7 / 48.7 for only 1.99× on LLaDA, so ρ = 0.1 is the operating point. Row per-step work depends only on n and ρ, and our sequences have n ≈ 350–440 (Sec. 3.2), so per-step speedups should carry over (inferred, not timed). Chunk results beyond GSM8K are not in the paper.
Q2. Memory (Sec. 4.4 formula, public configs): LLaDA 7.3 MB per token (row 4.5 MB), so 3.8 GB (2.3 GB) per sequence at n = 512 and 7.5 GB (4.6 GB) at n = 1024, linear in batch; Dream is similar. Beside 16 GB of weights and ~2 GB of activations, about six row sequences of length 512 fit on a 32 GB GPU, so TDS targets small-batch latency. Latency: per-step row speedup at ρ = 0.1 grows from n = 512 to 1024 (LLaDA 1.98→2.23×, Dream 1.71→2.33×; synthetic inputs, Fig. 3). Accuracy: reference, budget and threshold are per sequence, so batching changes only BF16 rounding, which already flips the correctness of 5–6.5% of dense answers (App. C.1). On LLaDA the error stays roughly flat over 256 teacher-forced steps (Fig. 2a). Not measured: free-running behaviour beyond 256 tokens, and runtime and peak memory at batch > 1.
We hope these answers address your concerns.