- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes LatticeSpec, a verification extension for speculative decoding of masked diffusion language models. It relaxes local ordering constraints in an existing draft and shares verification states reached through different update orders. A node budget limits the additional computation. The main configuration allows one adjacent pair to swap order and reports an average 5.7% throughput improvement over SSD across six model configurations and five tasks.
Critical strengths
-
The controlled analyses help explain the gains. Table 2 measures how much progress order relaxation recovers from the same candidates, while Table 3 separately measures the benefit of sharing repeated states.
-
The method reports consistent throughput improvements across all 30 model–task settings, including online selection and verification costs. The paper also acknowledges that larger structures and more elaborate selection rules do not always help.
Critical weaknesses
-
The benefit over prior work remains unclear. SSD already studies mix-order verification, and Spiffy uses shared-state graphs with budgeted pruning, but neither alternative is directly evaluated. Comparisons with the SSD chain and an unmerged tree do not establish the advantage over these methods. A simple threshold with fixed-position selection also performs within 1% of the main calibrated method (Table 9).
-
Output preservation needs clarification. Appendix D.1 reports different intermediate trajectories on 7/124 problems, without explaining the cause or whether final outputs differ. Agreement with the serial reference should be checked for the main configurations. Given the modest speedups, repeated timings and uncertainty estimates would also strengthen the results.
Questions for authors
-
How does LatticeSpec compare with SSD mix-order and Spiffy under matched decoding settings and verification budgets?
-
What causes the trajectory differences on 7/124 problems in Appendix D.1? Do final outputs differ, and how often do the main configurations match the serial reference exactly?
-
Can you provide repeated, interleaved timings and uncertainty estimates for the main SSD comparisons, especially those with the smallest gains?
6 comments
Response to Reviewer JpZo (2/3)
R3-Table 3: Comparison with a fixed Spiffy reference configuration at matched mean parallel verification-state counts
Following Spiffy’s main experimental configuration, we use an offline draft-graph budget of and dynamically prune it to during inference, with separate calibration for each model. LatticeSpec uses one model-level hyperparameter configuration shared by GSM8K and HumanEval.
| Model | Task | LatticeSpec | Spiffy | LatticeSpec TPS | Spiffy TPS | TPS gain | |
|---|---|---|---|---|---|---|---|
| LLaDA-8B, B16 | GSM8K | 3.8308 | 3.9358 | −2.67% | 58.312 | 56.129 | +3.89% |
| LLaDA-8B, B16 | HumanEval | 3.8282 | 3.9369 | −2.76% | 56.299 | 54.438 | +3.42% |
| Dream-7B, B16 | GSM8K | 4.1038 | 3.9352 | +4.28% | 86.359 | 80.064 | +7.86% |
| Dream-7B, B16 | HumanEval | 4.1111 | 3.9332 | +4.52% | 82.881 | 77.113 | +7.48% |
| SDAR-4B, B4 | GSM8K | 3.1536 | 3.1341 | +0.62% | 18.258 | 15.832 | +15.32% |
| SDAR-4B, B4 | HumanEval | 3.1583 | 3.1099 | +1.56% | 17.989 | 15.769 | +14.08% |
| SDAR-4B, B8 | GSM8K | 3.8749 | 3.8492 | +0.67% | 23.468 | 23.403 | +0.28% |
| SDAR-4B, B8 | HumanEval | 3.8716 | 3.8479 | +0.62% | 23.380 | 23.218 | +0.70% |
LatticeSpec improves observed TPS by 3.42%–3.89% on LLaDA, 7.48%–7.86% on Dream, and 14.08%–15.32% on SDAR block-4. On LLaDA, it also evaluates fewer parallel states (2.67% fewer on GSM8K and 2.76% fewer on HumanEval). SDAR block-8 is close to parity (+0.28% and +0.70%).
These results quantify LatticeSpec’s practical gains against existing mix-order and shared-state methods at comparable mean verification-state counts.
Q2: Output preservation and the 7/124 trajectory differences
A2: Thank you for asking us to clarify both the trajectory differences and final-output agreement. All seven cases in Appendix D.1 retain identical final output tokens and stopping behavior. The differences concern intermediate trajectories under finite-precision batched verification. Finite-precision batch sensitivity is an implementation consideration for methods that batch verification states, including but not limited to SSD, Spiffy, and LatticeSpec. Note that LatticeSpec focuses on efficient verification structures, while numerical precision is a separate implementation issue.
We also already account for output agreement in the main evaluation: all timing comparisons use a common exact-match subset, on which every compared method produces the same output tokens and stopping behavior as the one-token reference. The reported speedups therefore compare identical generated outputs and stopping behavior. To investigate the precision explanation, we compared output agreement at BF16, FP16, and FP32 in a separate precision comparison using LLaDA-8B-Instruct on 64 GSM8K problems. The table reports exact agreement with the one-token reference at the same precision across four repeated runs.
R3-Table 4: Exact output agreement with the one-token reference across model precisions
| Method | BF16 | FP16 | FP32 |
|---|---|---|---|
| SSD | 46/64 | 60/64 | 64/64 |
| LatticeSpec | 46/64 | 60/64 | 64/64 |
| Unmerged tree | 49/64 | 60/64 | 64/64 |
Agreement improves from BF16 to FP16 and reaches 64/64 for all three methods in FP32. These results demonstrate that the observed output discrepancies disappear in FP32 for both SSD and LatticeSpec on these 64 examples, confirming that the phenomenon is not specific to LatticeSpec.
For your question about how often the main configurations match the serial reference, the table below gives the number and proportion of evaluation examples on which all compared methods match exactly. These are the common subsets used for the reported timing comparisons.
R3-Table 5: Common exact-match subsets used in the main experiments
| Model / configuration | GSM8K (131) | HumanEval (100) | MBPP (500) | MT-Bench (80) |
|---|---|---|---|---|
| LLaDA-8B-Instruct (B16) | 113 (86.3%) | 85 (85.0%) | 425 (85.0%) | 56 (70.0%) |
| Dream-7B (B16) | 109 (83.2%) | 84 (84.0%) | 424 (84.8%) | 47 (58.8%) |
| SDAR-4B-Chat (B4) | 128 (97.7%) | 97 (97.0%) | 496 (99.2%) | 61 (76.2%) |
| SDAR-4B-Chat-b8 (B8) | 125 (95.4%) | 93 (93.0%) | 487 (97.4%) | 62 (77.5%) |
| SDAR-8B-Chat (B4) | 130 (99.2%) | 96 (96.0%) | 494 (98.8%) | 74 (92.5%) |
| SDAR-8B-Chat-b8 (B8) | 131 (100.0%) | 97 (97.0%) | 488 (97.6%) | 65 (81.2%) |
Each cell reports the common exact-match subset size and its percentage of the evaluation examples (totals in column headers). B denotes generation block length.
Response to Reviewer JpZo (3/3)
Q3: Repeated, interleaved timings and uncertainty
A3: Thank you for asking whether the smaller throughput gains persist across repeated measurements. We measured LatticeSpec three times on A100 GPUs in FP16 using the main evaluation settings and compared its throughput with the SSD baseline measurements.
R3-Table 6: Mean TPS gains over SSD across three measurements
| Model | Measurement 1 | Measurement 2 | Measurement 3 | SD (pp) |
|---|---|---|---|---|
| LLaDA-8B, B16 | +7.55% | +7.81% | +7.56% | 0.15 |
| Dream-7B, B16 | +4.04% | +4.33% | +4.16% | 0.15 |
| SDAR-4B, B4 | +6.13% | +6.09% | +6.85% | 0.43 |
| SDAR-4B, B8 | +4.96% | +4.95% | +5.19% | 0.13 |
| SDAR-8B, B4 | +6.28% | +6.25% | +5.98% | 0.16 |
| SDAR-8B, B8 | +4.89% | +4.91% | +5.14% | 0.14 |
Each measurement reports the arithmetic mean of per-task TPS gains over SSD for the corresponding model configuration. SD is the sample standard deviation across the three reported gains, expressed in percentage points (pp), and describes their observed variability. Comparisons use their respective common exact-match subsets.
LatticeSpec achieves higher TPS than SSD in all 30 model–task settings in each measurement, with consistent gains across model configurations.
Q4: The benefit of calibration over a simpler alternative
A4: Thank you for this constructive suggestion to clarify what our calibrated configuration contributes beyond a simpler alternative. Addressing this question requires distinguishing the roles of construction and selection, which can be chosen independently in LatticeSpec. Construction determines which alternative orders are allowed, while selection chooses which candidate blocks to relax within the verification budget. These components are modular: calibrated rank intervals are one construction option, rather than a requirement of LatticeSpec. We can therefore retain simple threshold construction and test whether gain-based selection improves on fixed-position selection.
Gain-based selection follows a simple principle: prioritize candidate blocks by their estimated acceptance gain per additional verification state. Our additional experiments show that this simple selection rule can improve throughput even when paired with simple threshold construction. This separates the benefit of selection from the choice of calibrated rank intervals.
Table 9 compares different combinations of construction and selection and shows that a simple threshold with fixed-position selection is competitive in the evaluated settings. Building on Table 9, the additional experiments examine whether the same simple construction can benefit from gain-based selection across different decoding settings. We keep threshold construction fixed and vary only the selection rule, while exploring different generation windows.
The window comparison below uses LLaDA and Dream on GSM8K and HumanEval with generation windows of 16 and 32. Both selection rules use threshold 0.1, draft length 4, at most one selected pair, and the same verification budget.
R3-Table 7: Fixed-position versus gain-based selection under the same threshold construction
| Model | Task | Window | Fixed-position TPS | Gain-based TPS | TPS gain |
|---|---|---|---|---|---|
| LLaDA-8B | GSM8K | 16 | 61.950 | 62.728 | +1.26% |
| LLaDA-8B | GSM8K | 32 | 58.517 | 59.140 | +1.07% |
| LLaDA-8B | HumanEval | 16 | 58.614 | 59.168 | +0.95% |
| LLaDA-8B | HumanEval | 32 | 55.730 | 56.467 | +1.32% |
| Dream-7B | GSM8K | 16 | 86.293 | 86.637 | +0.40% |
| Dream-7B | GSM8K | 32 | 78.154 | 79.353 | +1.53% |
| Dream-7B | HumanEval | 16 | 82.629 | 83.008 | +0.46% |
| Dream-7B | HumanEval | 32 | 73.920 | 74.158 | +0.32% |
With the same threshold construction, gain-based selection increases accepted tokens per verification round and reduces model forward calls in all eight settings, with observed TPS gains of 0.32%–1.53%. Confidence intervals estimated by resampling evaluation examples support positive gains in five of the eight settings.
These controlled comparisons show that gain-based selection can improve throughput over fixed-position selection under the same threshold construction. The observed gains highlight the effectiveness of combining simple threshold construction with gain-based selection. This supports LatticeSpec’s modular design: a simple threshold construction can be combined with gain-based selection while retaining the same shared-state verification framework.
Response to Reviewer JpZo (1/3)
Thank you for recognizing the controlled mechanism analyses and the inclusion of online costs. We appreciate your suggestion to extend the evaluation to existing mix-order and shared-state verifiers. Below we address those comparisons, the trajectory differences, and the reliability of the reported runtime gains.
Q1: SSD mix-order and Spiffy under comparable verification budgets
A1: Thank you for suggesting direct comparisons with SSD mix-order and Spiffy. The new results below evaluate LatticeSpec against these existing alternatives under matched mean parallel verification-state counts.
Comparison protocol. We add comparisons on GSM8K and HumanEval using LLaDA-8B-Instruct (block size 16), Dream-v0-Instruct-7B (block size 16), and SDAR-4B-Chat configurations (block sizes 4 and 8). Following the experimental settings in Table 1, all runs use FP16 and a maximum generation length of 256.
All three methods support alternative verification orders, but their default configurations evaluate different numbers of states in parallel. Evaluating more states in parallel can itself accelerate decoding by allowing more candidate updates to be accepted in each round, giving methods with larger verification budgets an advantage. To ensure a fair comparison, we keep each baseline's reference configuration fixed and adjust LatticeSpec so that its mean count differs from the baseline's by at most 5%, measured relative to the baseline:
ΔS=SbaselineSLatticeSpec−Sbaseline×100
Here, S is the mean number of states in verification forwards with batch size greater than one. Keeping S similar makes the comparison fairer by preventing a method from gaining an advantage simply by verifying more candidate states in parallel.
Strengthening SSD mix-order with rejoining. The original mix-order mechanism in SSD, Section 5.2 and Figure 4(b) does not support rejoining: its additional out-of-order branches are leaves, with no continuation back to the main chain. To strengthen this baseline, we additionally implement rejoining, which lets a side branch accept the previously skipped update, return to the equivalent main-chain state, and reuse its predictions and remaining path within the same verification round. This extension adds connections without adding verification states. It brings SSD mix-order closer to LatticeSpec's reuse of shared states across alternative orders.
Across these six comparisons, LatticeSpec improves TPS by 6.79%–8.60% over original SSD mix-order and by 0.97%–6.12% over its rejoining extension. On LLaDA and Dream, these improvements are achieved with fewer parallel verification states.
Each model uses one LatticeSpec configuration across both tasks, with both methods evaluated on the same hardware using the same inference implementation. We did not find an SDAR block-4 configuration allowing a fair comparison with SSD mix-order, as even the closest configuration we found had a mean parallel state count about 28% lower than the baseline, so we excluded it from these comparisons.