- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the low acceptance rate of Speculative Jacobi Decoding (SJD; ICLR '25), a training-free method that accelerates autoregressive (AR) image generation by drafting and verifying multiple tokens in parallel. Prior SJD variants such as LANTERN and GSD raise acceptance by relaxing the verification criterion, but the drafting step still samples tokens without considering whether they will pass verification. The authors observe that, across Jacobi iterations, tokens whose probabilities increase consecutively are much more likely to be accepted, and also more likely to match the token produced by standard AR decoding. Their goal is to use this signal at the drafting step to predict which tokens will pass verification.
The proposed method, SJD-VP, changes only the drafting step and consists of two parts. First, it keeps each token's probabilities over the last few Jacobi iterations, computes an exponentially weighted reference, and scores each token by its log-probability gain over this reference. A mask keeps the score only for tokens that have grown for N consecutive iterations. Second, it treats this score as a prior and the current drafting probability as a likelihood, and reweights the top-k candidates to form the distribution from which draft tokens are sampled. Verification follows the original SJD rule, so the method can be attached to other SJD variants. The paper also gives a first-order analysis showing that the total variation distance between the draft and verification distributions decreases when the weighted directional accuracy Q_w exceeds 0.5.
Experiments use Lumina-mGPT on Parti-prompts and MS-COCO 2017, with additional results on LlamaGen, Emu3, and Janus-Pro. On Parti-prompts, SJD-VP reduces the NFE of SJD from 1035 to 721 and raises the latency speedup over AR decoding from 2.20× to 2.94×. On MS-COCO, FID stays at 30.77, compared with 30.78 for SJD. When combined with five other SJD variants, including LANTERN, GSD, and SJD-SV, it reduces NFE in every case. These results show that the drafting-side change consistently improves acceptance, although the latency gain is much smaller than the NFE gain on some models.
Critical strengths
-
A simple and interesting observation about the dynamics of Jacobi decoding. Recent SJD variants improve acceptance by changing how tokens are verified. This paper instead looks at how each token's probability evolves across Jacobi iterations, which is information that SJD already computes but discards. It finds that most accepted tokens show consecutive probability growth before acceptance, 91.4% on average across Lumina-mGPT, LlamaGen, and Emu3. Tokens that keep growing are also likely to match the AR reference token, reaching 96.35% after three steps of growth. This moves attention from the verification step to the drafting step, and may motivate further work that exploits Jacobi trajectories.
-
Consistent acceleration as a plug-in, with little change in generation quality. The method only changes how draft tokens are sampled and requires no training or additional model. On Lumina-mGPT, it reduces the NFE of SJD by about 30% on both benchmarks. It also reduces NFE when combined with each of five other SJD variants and on three other AR models.
Critical weaknesses
-
The memory and computational overhead of the method is not properly characterized.
- SJD-VP stores the probabilities of every vocabulary entry at every window position over the last L iterations. Its memory cost therefore grows with the vocabulary size, the window size, the history length, and the batch size. Recent AR image models use large vocabularies, so this overhead is non-negligible in large-batch settings. The paper reports no memory measurement.
- The only overhead reported is a per-step time of 1.026 ms (Table 9). Yet the latencies in Table 5 imply a larger and model-dependent cost. The time per NFE rises from 34.9 to 42.3 ms on LlamaGen and from 53.7 to 67.5 ms on Emu3.
- This overhead erodes the speedup. On LlamaGen, a 20% reduction in NFE yields only a 3% reduction in latency.
- The overhead also grows with the window size. In Table 6, the time per NFE of SJD-VP rises from 37.5 ms at W = 16 to 46.6 ms at W = 64, while SJD stays at 34.8 ms. Since the strongest results use W = 64, the cost matters most exactly where the method looks best.
-
The core observation and its theoretical justification are weakly grounded.
- The method rests on the empirical finding in Fig. 2 that tokens with rising probabilities are likely to be accepted, but the paper does not explain why. Part of the effect may follow from the SJD acceptance rule itself, min(1, q/p), which always accepts a token whose probability has risen. Without the growth rate among rejected tokens, it is hard to judge how informative the signal is.
- The conclusion of Sec. 3.4 is trivial. It states that the distance to the verification distribution decreases if Q_w exceeds 0.5, that is, if the predictor is better than a random guess. This restates the first-order condition and says nothing about how much acceptance improves.
- The first-order Taylor expansion in Eq. 8 assumes a small shift m. However, the reweighting in Eq. 4 multiplies the probability of each growing token by p_t/p̄_t, which is not a small perturbation. If the shift were small enough for the expansion to hold, the correction would be too small to change acceptance. Appendix A.5 itself concedes that the analysis gives no guarantee for large shifts.
- The analysis also does not describe the method as implemented. Draft tokens are sampled from p'_t but verified with the original p_t (Appendix A.3.1), so the acceptance rate is not governed by the distance between p'_t and q, as Appendix A.5 acknowledges. This mismatch also means that SJD-VP no longer preserves the target distribution, which the main text does not state.
Questions for authors
- Please see the weaknesses. Could the authors report the peak memory and the time per NFE of SJD and SJD-VP for each model and window size, including batched settings?
- Is the link between rising probability and acceptance specific to SJD, or is it also observed in general speculative decoding frameworks that use independent draft models?
- Could the authors report results for a variant that verifies draft tokens with p'_t, which would preserve the target distribution?
13 comments
For W1.2. Runtime Overhead Discrepancy
We apologize for not clearly explaining the measurement settings in Table 9, which may have caused confusion. Specifically, Table 9 reports the runtime of the VP module measured independently, without executing the AR backbone. In contrast, Table 5 reports end-to-end inference performance, where the actual overhead of VP also depends on model-specific computational workloads and memory operations. Therefore, the isolated measurement of 1.026 ms in Table 9 does not directly reflect the additional time per NFE during actual model inference.
To clarify this discrepancy, we further profile the individual VP components during actual inference on three different models, as shown in Table 2.
Table 2: Runtime Overhead of SJD-VP during Actual Model Inference. (W=16)
| Model | History Tracking (ms/NFE) | Prior Estimation (ms/NFE) | Bayesian Fusion (ms/NFE) | Total VP (ms/NFE) |
|---|---|---|---|---|
| Lumina-mGPT | 0.1 | 0.8 | 1.7 | 2.6 |
| LlamaGen | 0.2 | 1.2 | 5.6 | 7.0 |
| Emu3 | 0.2 | 1.2 | 11.2 | 12.6 |
As shown in Table 2, Bayesian fusion accounts for most of the additional computation, while history tracking and prior estimation introduce relatively small overhead. The measured total overheads of 2.6, 7.0, and 12.6 ms/NFE closely align with the increases observed in Table 5 (approximately 2.6, 7.4, and 13.8 ms/NFE), explaining the previously noted discrepancy. We will clarify the two measurement settings and include this profiling analysis in the revised manuscript.
W1.3. NFE Reduction and Wall-Clock Speedup
We would like to clarify the discrepancy between NFE reduction and wall-clock speedup on LlamaGen. The relatively modest wall-clock improvement on LlamaGen is mainly due to the additional computation introduced by VP. Although SJD-VP reduces NFE by approximately 20%, the increased time per NFE partially offsets this benefit, resulting in a 3% latency reduction.
As shown in Table 2 of W1.2, the measured VP overhead is 7.0 ms/NFE, closely matching the 7.4 ms/NFE increase observed in Table 5 of the original manuscript. This explains the difference between NFE reduction and actual latency improvement. We will clarify this in the revised manuscript.
For W1.1. Memory Overhead
We thank the reviewer for highlighting the importance of memory overhead. SJD-VP maintains probability histories across the last L Jacobi iterations for token positions within the decoding window. Therefore, the additional memory consumption depends on the vocabulary size, window size, history length, and batch size.
To quantify this overhead, we measure the peak GPU memory of SJD and SJD-VP during actual image generation under identical inference settings, with a batch size of 1.
Table 1: Peak GPU Memory and Time per NFE across Different Models and Window Sizes.
As shown in Table 1, memory overhead increases with the window size, reaching 0.84 GB (approximately 4.5% of SJD peak memory) for Lumina-mGPT at W=64. These results indicate that SJD-VP introduces relatively modest additional memory consumption under the evaluated settings. We will include these measurements in the revised manuscript.