- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper focuses on a specific and important problem: how inference-time controllers decide which tokens to process in LLM speculative decoding and diffusion language model sampling. These decisions involve trade-offs between computational efficiency and objectives such as throughput and generation fidelity, making the problem worth studying. The proposed target-correction framework is intuitive and supported by experimental results. However, I have some concerns about the experimental model selection: why not also evaluate on models such as LLaDA to further verify the generalizability of the method?
Critical strengths
-
The problem addressed is specific and important, and the paper is properly structured.
-
The proposed idea is straightforward and clear, requiring only relatively small modifications to existing algorithms.
Critical weaknesses
-
Many dLLM papers use LLaDA for evaluation, whereas this paper only adopts DiffusionGemma for its diffusion sampling experiments. From a generalization perspective, evaluating additional models, potentially including different sizes of LLaDA, would help demonstrate the transferability of the method. Although the sampling mechanisms differ, the authors could also clarify why LLaDA was not selected and whether these differences limit the applicability of their approach.
-
Although the paper includes DSpark and Entropy-Bound as two direct baselines, it does not sufficiently compare against compatible competing approaches or demonstrate that its core construction is more effective than simpler alternatives. In other words, the concern is not that the paper lacks baselines, but that it mainly demonstrates improvements over the original controllers without fully establishing the advantages of the proposed unified framework. It would be helpful to compare against simpler alternatives, such as different request-throughput weighting strategies for speculative decoding or a history-based penalty without the wealth term for diffusion sampling. Ablations isolating the roles of wealth, churn divergence, and terminal utility would also help clarify which components are responsible for the observed gains.
-
The connection between the theoretical framework and the concrete algorithmic choices is not always sufficiently clear. The paper provides useful high-level intuition, but the logical chain from the motivating problem and modeling assumptions to the derivation of the value function and the final algorithmic modifications could be better established. As a result, it is sometimes difficult to understand why each design choice is necessary rather than simply a reasonable heuristic. I mean this paper makes me confusing.
Questions for authors
-
Could the authors explain why LLaDA was not selected for the experiments? Are there particular differences in its sampling mechanism that would limit the applicability of the proposed method?
-
Could the authors provide some qualitative examples of actual generations, particularly for the diffusion sampling experiments, to illustrate the differences between the original and corrected methods?
-
The 4% NFE improvement limited, please show the calculating progress.
4 comments
Results
Here is an overall summary for SEB. It averages across the 12 datasets and 7 hyperparameter values (16K+ runs per method).
| Sampler | Stability / History Signal | Accuracy (%) | Mean NFE () | Tokens / NFE () |
|---|---|---|---|---|
| Entropy-Bound (EB) | Stateless ( only) | 59.75 | 42.27 | 13.93 |
| SWD | Forward KL weight | 60.06 | 42.75 | 13.71 |
| KLASS | Reverse KL gate | 59.66 | 43.08 | 13.45 |
| CreditDecoding | Multi-step argmax logit credit | 59.55 | 40.15 | 14.56 |
| Steady Entropy-Bound (SEB) | Target-corrected wealth + mode churn | 59.50 | 35.08 | 15.99 |
But even in individual hyerparameter-paired comparisons, SEB demonstrates much better mean NFE. Here is the more fine-grained comparison of SEB to KLASS. In this sweep, we matched KLASS' hyperparameters to SEB's in a strong manner consistent with KLASS' choices.
| Budget | KLASS Acc (%) | SEB Acc (%) | KLASS Mean NFE () | SEB Mean NFE () | KLASS Tok/NFE () | SEB Tok/NFE () |
|---|---|---|---|---|---|---|
| 0.003 | 60.16 | 59.74 | 49.07 | 37.72 | 12.42 | 15.27 |
| 0.01 | 59.95 | 60.37 | 48.21 | 36.88 | 12.61 | 15.51 |
| 0.03 | 59.61 | 59.49 | 47.51 | 36.57 | 12.76 | 15.69 |
| 0.1 (default) | 60.79 | 59.91 | 44.45 | 36.03 | 13.32 | 15.89 |
| 0.3 | 60.24 | 59.82 | 42.62 | 35.08 | 13.69 | 16.14 |
| 1.0 | 59.66 | 58.81 | 39.96 | 33.85 | 14.15 | 16.50 |
| 20.0 | 57.21 | 58.39 | 29.72 | 29.44 | 16.57 | 17.23 |
| All | 59.66 | 59.50 | 43.08 | 35.08 | 13.45 | 15.99 |
Thank you for the response and additional experiments. For W2, could the authors compare SEB against a simpler baseline that directly incorporates the proposed mode churn statistic into EB, without wealth-based correction? This would help clarify whether the additional wealth mechanism is necessary for the observed improvements. I will also consider the other reviewers' opinions before making my final decision.
Response
Certainly! Recall, in SEB, the target-corrected effective entropy is: Wealth appears in the terms and , not the mode-churn statistic . The most natural ablation of wealth is to just add directly to the entropy, yielding "EB + Churn":
However, as seen in the table below, EB + Churn does not reduce NFE. Without wealth, acts only as a one-sided penalty (): even when a frontier token is repeatedly corroborated with , its entropy is never discounted. Wealth-based target correction enables that.
| Budget | EB Acc (%) | EB + Churn Acc (%) | SEB Acc (%) | EB Mean NFE | EB + Churn Mean NFE | SEB Mean NFE |
|---|---|---|---|---|---|---|
| 0.003 | 60.20 | 60.20 | 59.74 | 50.86 | 53.04 | 37.72 |
| 0.01 | 60.20 | 61.42 | 60.37 | 47.15 | 48.53 | 36.88 |
| 0.03 | 60.29 | 60.58 | 59.49 | 44.52 | 45.79 | 36.57 |
| 0.1 | 59.99 | 60.24 | 59.91 | 42.95 | 43.25 | 36.03 |
| 0.3 | 60.37 | 60.03 | 59.82 | 41.17 | 41.69 | 35.08 |
| 1.0 | 59.28 | 60.41 | 58.81 | 39.24 | 39.18 | 33.85 |
| 20.0 | 57.89 | 57.68 | 58.39 | 29.98 | 30.41 | 29.44 |
| All | 59.75 | 60.08 | 59.50 | 42.27 | 43.13 | 35.08 |
Please let us know if you'd like to see any other ablations.
Response
Thank you, we agree with all this feedback about rigorous comparison with mechanistic insight. We updated our paper to improve in all these areas.
Q3: NFE Improvement. Great news: we substantially improved SEB. We now see 17% reduction in NFE (not just 4%). We accomplished this through a non-homogeneous terminal utility and a new mode churn statistic.
Q2: Generations. Please see page 23 for an interesting pair of real trajectories produced by EB and SEB on MMLU Pro Philosophy. You can see how EB gets stuck on an ultimately-irrelevant token. On this example, SEB improved NFE by 60%.
W2: Baselines. We now compare against four baselines (not just one): EB, CreditDecoding, SWD, and KLASS (on pages 9, 21 and 22). SEB improves upon all of them by more than 12% in NFE. Importantly, these new experiments provide interesting mechanistic insight. The methods based on KL penalties (SWD uses forward, KLASS uses reverse) increase NFE rather than decrease it. Using KL to measure mode churn has the unfortunate property of penalizing sharpening on a correct mode (which is what a denoising algorithm is supposed to do!) Our new SEB avoids this with a different Di (equation 6 on page 7).
W1, Q1: LLaDA. There are two reasons we didn't use LLaDA. SEB is designed for uniform renoising diffusions as in DiffusionGemma. LLaDA 1 is a research model (4K context length); we focus on production codebases, where keeping code changes small is important. LLaDA 2.1's production (SGLang) sampler is much different (based on token-to-token editing).
If you want, we can try target-correcting the LLaDA 2.1 sampler. (After some analysis, we may have identified a linear myopia in it.) If so, please let us know soon, to give us time to implement and run this application. However, please also take into account the following point.
W3: Applying Target Correction. We added extensive new applications to KV cache eviction and chunked prefill. Target correction is not purely "plug and play" (there is indeed some technical work required to design the game and utility for each application), but we hope these additional examples offer more guidance to researchers and inference engineers.