- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes TAPER, a training-free rule for speeding up blockwise decoding in masked diffusion language models (LLaDA and Dream). In standard blockwise decoding, every model call predicts all masked positions, but only positions in the current block can be committed. TAPER also commits positions near the start of the next block when the model is already confident about them, using the same threshold as the current block. It needs no extra model calls or stored history. The authors also show with a simple counting identity that the number of tokens committed early is not the number of calls saved. Most early commits only replace tokens that would otherwise have been committed together later in their own block. On four math and code benchmarks, TAPER reduces the number of model calls by a few percent with little change in accuracy, although the authors acknowledge that retuning the threshold of the existing commit rule gives almost the same trade-off.
Critical strengths
The paper is careful about what its method does and does not do. The counting identity is simple, but the point behind it is useful. Methods that commit tokens early should report how many calls are actually saved, not how many tokens were committed early. The paper also includes many controls that try to locate where the saving comes from, such as handling end-of-sequence tokens separately, releasing positions by distance instead of confidence, and splitting the AHD gate into its confidence and history parts. I also appreciate that negative results are reported openly, including that the method does not beat retuning the threshold and that the accuracy non-inferiority is not established.
Critical weaknesses
- The contribution is small. The call reduction is a few percent, and the authors themselves say that retuning the threshold of the existing commit rule gives a similar trade-off. Reusing the block's threshold also does not seem essential, since a plain confidence threshold recovers most of the savings. The counting identity is bookkeeping and is not enough on its own to count as a main contribution.
- All results assume decoding without a key-value cache, where every call costs the same. In practice, caching and partial recomputation are widely used, and in that setting, fewer calls do not necessarily mean less compute or less time. The paper mentions this in the limitations, but does not measure it.
- Several training-free methods that also reduce the number of decoding steps or relax block boundaries are neither cited nor compared, for example, dilated scheduling (Luxembourg et al., 2025), SlowFast (Wei et al., 2025), and CtrlDiff (Huang & Tang, 2025).
- The window size and threshold were chosen on the same benchmarks used for testing, and the advantage over retuning the threshold at length 512 did not hold up in the pre-planned evaluation on development problems. The paper is also hard to read. The main text is packed with numbers and qualifications, and it took me a while to work out what the central claim is.
Questions for authors
- When TAPER is combined with key-value caching or methods that recompute only some positions, how much of the call reduction remains as a reduction in compute or wall-clock time?
- If retuning the threshold gives almost the same trade-off, in what situation would a practitioner choose TAPER over simply adjusting the threshold?
- How does TAPER compare with dilated scheduling, SlowFast or CtrlDiff in accuracy at a matched number of calls?
2 comments
Reply to Reviewer cnUk (part 2 of 2)
Q1 / W2. Does the saving survive a KV cache? Yes (Section 4.6; Tables 6, A23 and A26). Under Fast-dLLM's dual cache (RTX4090, full test sets, paired), release processes 13–26% more query tokens per output at , yet it removes 7.4–10.0% of calls and 3.8–8.2% of wall-clock time there, and 8.6–11.8% and 8.2–13.6% at :
| Model, admission rule | No release: acc / calls / s | Release: acc / calls / s | acc [95% CI] | calls / Δ query / Δ time | Equal query width: acc / calls / Δ time |
|---|---|---|---|---|---|
| LLaDA, margin price | 46.91 / 85.9 / 3.13 | 48.59 / 77.9 / 2.87 | +1.68 [−0.28, +3.70] | −9.3% / +13.0% / −8.2% | +0.06 / −5.9% / −4.8% |
| LLaDA, Fast-dLLM () | 48.63 / 89.4 / 3.22 | 48.86 / 82.0 / 2.99 | +0.23 [−1.61, +2.06] | −8.2% / +14.2% / −7.2% | −0.09 / −5.3% / −4.4% |
| Dream, margin price | 59.78 / 83.2 / 2.36 | 61.86 / 74.9 / 2.20 | +2.08 [+0.13, +4.01] | −10.0% / +22.0% / −6.6% | +0.28 / −7.0% / −7.5% |
| Dream, Fast-dLLM () | 61.90 / 86.4 / 2.48 | 62.51 / 80.1 / 2.38 | +0.61 [−1.30, +2.50] | −7.4% / +26.0% / −3.8% | +0.66 / −5.5% / −5.8% |
| LLaDA, margin price, | 52.56 / 140.3 / 6.31 | 53.17 / 126.7 / 5.77 | +0.61 [−1.25, +2.58] | −9.7% / +8.7% / −8.5% | −0.13 / −6.5% / −6.5% |
| LLaDA, Fast-dLLM (), | 53.15 / 145.5 / 6.69 | 53.32 / 132.9 / 6.15 | +0.17 [−1.87, +2.27] | −8.6% / +9.5% / −8.2% | not run |
| Dream, margin price, | 62.00 / 101.6 / 3.95 | 62.82 / 89.7 / 3.41 | +0.82 [−1.09, +2.77] | −11.8% / −7.9% / −13.6% | −0.10 / −10.3% / −13.8% |
| Dream, Fast-dLLM (), | 63.05 / 104.7 / 4.04 | 64.95 / 93.7 / 3.60 | +1.89 [0.00, +3.79] | −10.5% / −6.3% / −10.9% | not run |
Why the wider query does not eat the saving (cost ledger, Table A26): fitted to the per-output rows (), a full forward costs 1.4–2.2 cached calls, and widening the query by h adds only 0.5–6% to a cached call. Each removed call therefore saves almost a whole cached call, and a block that release fills before it opens skips its full forward. On Dream at , of the 13.2% saving the ledger gives 9.2 points to skipped full forwards (10.9 instead of 16 per output) and 5.5 to fewer cached calls; the wider query costs 1.4. LLaDA's window is shorter than a block, so it saves through fewer calls alone. The last column is a control that widens the query without releasing: it attributes the accuracy rise to the refreshed window (+1.62 and +1.80 points on the margin price), and at equal query width release still removes 4.4–7.5% of time at and 6.5–13.8% at , with every accuracy interval containing zero (Table A23). We tested one cache design, with batch size one.
Q3 / W3. DUS, SlowFast and CtrlDiff at matched calls. All three are now cited (Section 5). DUS and SlowFast are training-free, so we ran them in our pipeline (Table 3; the other block sizes in Table A21):
| Method | LLaDA acc / calls | LLaDA gap | Dream acc / calls | Dream gap |
|---|---|---|---|---|
| TAPER | 47.72 / 71.4 | −0.55 | 61.90 / 74.3 | 0.00 |
| DUS, b = 8 | 43.06 / 96 | – | 51.17 / 96 | – |
| DUS, b = 16 | 41.05 / 64 | −6.70 | 46.71 / 64 | −14.85 |
| DUS, b = 32 | 35.96 / 40 | – | 34.85 / 40 | – |
| SlowFast, official | 41.87 / 87.1 | −7.40 | 32.12 / 80.9 | −29.97 |
| SlowFast, 32-token blocks | 10.87 / 69.4 | −37.30 | 49.15 / 99.6 | – |
| SlowFast, 32-token blocks, full canvas | 49.14 / 99.8 | – | 61.48 / 108.9 | – |
Gap: accuracy minus our frontier of the five admission settings at the method's calls; a dash means the calls lie outside the frontier's range. SlowFast ends most outputs early with 32-token blocks on LLaDA and with its blockless sampler on Dream. The official SlowFast code, run unmodified on 64 GSM8K items, matches our port to within one item. With 32-token blocks the early endings come from the official prefix-only fast-phase calls: computed on the full canvas, SlowFast reaches 49.14% (LLaDA) and 61.48% (Dream), +1.41 [−0.42, +3.19] and −0.42 [−2.24, +1.32] points against TAPER, at 40% and 47% more calls. CtrlDiff trains its own small model from scratch with an RL block-size policy and reports perplexity; we found no public code, so we cite it but cannot run it on LLaDA or Dream.
W4. Settings chosen on the test benchmarks; readability. On the 1,142 development problems alone, the selection rule picks for LLaDA (the test-set choice) and for Dream; the three windows differ by less than one point on either set (Table A27), and the call reduction holds on these problems (Table A7). was also fixed during earlier development on these benchmarks, and Section 2 now says so; release saves 5.6–7.4% of calls at every from 3 to 6 (Table A8). The gain over retuning, which the pre-specified development evaluation did not reproduce (+0.24 and +0.01 points), is no longer claimed. Section 4 now follows four questions, and the numbers are in tables.
Reply to Reviewer cnUk (part 1 of 2)
Thank you for the clear questions.
TL;DR. (Q1) Yes: under Fast-dLLM's dual cache release lowers wall-clock time by 4–14%, and a cost ledger shows why (a removed call saves a cached call; a block filled before it opens skips its full-forward refresh). (Q2) Under a cache, release saves time on a threshold as it stands; against our retuned margin price it is more accurate at equal time on Dream (interval excludes zero) and only in point estimate on LLaDA, and on Fast-dLLM's threshold the gap to retuning is within noise. Without a cache it adds operating points next to retuning. (Q3) DUS and SlowFast's official settings are 6.7–30 points below our frontier at matched calls (Table 3). (W4) Development-only selection picks the same LLaDA window, and the L=512 gain over retuning is no longer claimed.
Q2 / W1. Why choose release over retuning the threshold? We agree with two of your points: our price is not essential, and the counting identity alone is bookkeeping. What remains is a release step that adds no call and no threshold to any threshold rule, the measurement that one early commit is worth about 0.1 calls, and a wall-clock saving of 4–14% under a key–value cache with a cost ledger that explains it.