- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This papers studies acceleration method for diffusion language model. It tries to improve over the block-wise decoding. Prior method stops the next block decoding until the current block is fully finished. This work tries to relax this constraint and provide speedup with performance unchanged. It proposes to use a private lane that can advanced the decoding of next block while its content doesn't affect main lane decoding. At the end of the current block decoding, the next block content will be merged into the main lanes.
Critical strengths
- I think the motivation is clear and the idea is interesting. It proposed a reasonable engineering improvement to overlapping two adjacenet blocks decoding, the introduction of private lane likes adding a local casual masks.
- The experiments are completed, covering three models, and 5 benchmarks, 3 baselines. The proposed method are generally effective in improving decoding speedup. Also, the ablation studies the design choices of early opening, lookahead prefilling.
Critical weaknesses
- Missing comparisoin with simple baseline. One comparision i found necessary is to comparing the method design two lanes with doubling block size while controlling other tricks to be the same. The proposed method is to allow adjacet blocks to be processed concurently, and this can be simply achieved by increasing the block size. It is not sure if the private lane can help reduce the error compared to 2 block-length decoding.
- I think the author should clarify the configuration/hyperparameter selection method. There are a bunch of hyperparameters and looks like the optimal setting varies over the benchmarks and generation length. It would be helpful to check the sensitivity of hyper or if there is a cheap/principle way to select. Also, paper mentions "sometimes using evaluation examples"which is a bit confusing, does it mean test set is involved?
Questions for authors
- The proposed method needs to maintain two state of decoding, what is the GPU overhead for this? This could be a poential concern for serving.
- From figure 4 and 5, it seems that the prediction reuse reduces more lantency compared to early opening. The main headlne results mixed these two components. I was wondering if prediction reuse the main contributor instead of the two-lane mechaism. I expect some clarification on this cause this might be my reading issue.
7 comments
W2:How the hyperparameters are selected, whether they are sensitive, and whether the test set was used
Thank you for the suggestion. The wording in Section 4.1 was not precise enough. We sincerely apologize for the confusion it caused, and thank you for giving us the opportunity to clarify. These configurations were determined during development, and the selection did not refer to results on the test set. In the original text, "using evaluation examples" refers to the sensitivity analysis we conducted after all main experiments had been completed, in order to identify the patterns in how each parameter affects quality and latency. Overall, these parameters fall into two groups:
- The parameters introduced by AsyncLane can be kept fixed. These parameters show stable behavior and do not need to be tuned.
- the content prefill horizon H;
- the private lane's fallback acceptance floor τfloor, and its private-candidate threshold τpriv;
- the fallback block length ℓ0;
- the content-prefill threshold τLA;
- the opening threshold ρ for starting the private lane;
- the delimiter-confidence threshold τbd, which follows AdaBlock's setting (0.3 for LLaDA and 0.5 for Dream); our analysis shows that changing τbd between these two values does not have a large effect on the results, so it is fixed at 0.3 in the table below.
- A parameter that follows a pattern: the main-lane commit threshold τmain. It is an original parameter of the base decoding rule, and the usual pattern is that a larger value gives slower decoding and higher quality.
Through the analysis, we found general, balanced fixed values for these parameters; unless otherwise specified, all experiments use this unified hyperparameter configuration:
| ρ | τpriv | τfloor | ℓ0 | τLA | H | τbd | τmain |
|---|---|---|---|---|---|---|---|
| 0.7 | 0.9 | 0.5 | 32 | 0.95 | 32 | 0.3 | 0.7-0.9 |
Revision. In the revised version, we will correct the wording in Section 4.1, list all settings in one place in Appendix E, add the sensitivity analysis of the corresponding parameters to Appendix H, and state the selection method described above.
Q1:GPU overhead of maintaining two decoding states, and the impact on serving
Thank you for raising this question. To answer briefly first: the GPU memory and compute overheads of maintaining the second state, i.e., the private lane, are both relatively small.
Peak GPU memory. The private lane increases peak GPU memory by only 0.63–1.51 GiB (LLaDA) and 0.10–0.19 GiB (Dream). The increase consists mainly of the private lane’s KV cache copy and the activations produced when computing the private lane. LLaDA’s increment is larger than Dream’s because LLaDA uses multi-head attention, whereas Dream uses grouped-query attention.
Compute overhead. The private lane does not add model calls: it and the main lane are two input rows in the same model call, so the number of model calls per request is essentially unchanged (a difference of −0.02 to +1.04 calls). The private lane also adds very little computation. Its KV cache is copied directly from the main lane; it does not need to run another forward pass over the entire sequence by itself. In addition, it exists only while the current block and the next block overlap; in each call, it also computes only over the tokens in these two blocks, and the rest of the sequence directly reuses the cache. Therefore, the total number of processed tokens increases by only 1%–6%. Even if, during serving, the GPU is already fully loaded and throughput depends on the amount of computation, the extra computation does not exceed this proportion. At batch size 1, the latency per request increases by only 0.8%–2.9%.
Table R14: Overhead introduced by the private lane
| Metric | Change |
|---|---|
| Peak GPU memory | LLaDA +0.63–1.51 GiB; Dream +0.10–0.19 GiB |
| Number of model calls per request | −0.02 to +1.04 |
| Total number of processed tokens | +1%–6% |
| Latency per request | +0.8%–2.9% |
Conclusion. Both the memory and the compute overhead of the second state are small. Because the private lane computes only the current and next blocks, it still adds little compute even when the GPU is already saturated during serving and throughput is compute-bound.
Revision. We will add this analysis to the new revised version.
Q2:The relationship between prediction reuse (LA) and the two-lane mechanism
Thank you for carefully comparing Figures 4 and 5 and raising this question. In short, prediction reuse (LA) and the two-lane mechanism are two parts of the same design: both start filling subsequent positions without waiting for the current block to be completed, and they also use predictions from the same forward pass. Their effects are complementary: when the other part is held fixed, each can further reduce latency on its own.
Both belong to advancement. Standard block decoding waits until all positions in the current block have been decoded before it starts filling subsequent positions. AsyncLane separates these two things, which is also what “Decoupling Refinement from Advancement” in the title means: the main lane is responsible for completing the current block (refinement), and subsequent positions start being filled before the current block is completed (advancement). This part of the work is divided between the private lane and LA: the private lane is responsible for the next block. LA is responsible for farther positions, i.e., beyond the opened blocks.
The two parts are complementary, and each makes its own contribution. Tables 1–2 report the complete AsyncLane, i.e., with the two-lane mechanism and LA both enabled. The effect of each part is measured separately by the two ablations in Section 4.3: each time, one part is fixed and only the other is changed. As you noticed in Figures 4 and 5, LA’s reduction is larger. However, the effects of the two parts stack, rather than one replacing the other: with LA fixed, early opening still reduces latency by 4.63–11.62%. The two parts also share the same opening forward: while continuing to complete the current block, one full-sequence forward pass also provides the private lane with the initial candidates for the next block and provides LA with predictions at farther positions, so adding LA requires no extra model calls.
Revision. We will add this analysis to the new revised version.
Thank you for reading our paper carefully and for your detailed comments, and would like to respond to the raised points.
W1:Compared with a baseline that directly doubles the block length and keeps the other settings unchanged, can the private lane reduce errors?
We thank the reviewer for the suggestion; we ran this comparison on the full test sets. In short, AsyncLane has speed comparable to the doubled-block-length baseline you suggested, while achieving higher accuracy.
More importantly, increasing the block length cannot replace AsyncLane; the two can be combined. AsyncLane schedules the work between blocks rather than the work within a block, so it can provide further speedup at any block length. After the block length is doubled to 2ℓ₀, adding AsyncLane still makes decoding significantly faster, and accuracy does not drop significantly.
Setup. we cross two factors:
Both Wait-2ℓ₀ and AsyncLane-ℓ₀ advance two adjacent blocks at the same time. AsyncLane can keep the second segment from affecting the first before the first is completed, whereas Wait-2ℓ₀ can be understood as two blocks ℓ₀ that affect each other.
Table R12: Accuracy (%) / mean request latency (s) of the four configurations. Bold: significant at the 95% level.
Table R13: Each row compares two configurations; a speedup > 1 or a ΔAcc > 0 favors the former.
The comparisons in rows 1–2 can demonstrate the effect of AsyncLane’s two lanes, and those in rows 3–4 can demonstrate the effect of directly doubling the block length.
Conclusion. The private lane can indeed reduce errors: at comparable speed, AsyncLane is more accurate than the 2ℓ₀ baseline you suggested on most tasks. Meanwhile, increasing the block length cannot replace the role of AsyncLane either, because the two can be combined.
Revision. We will add this comparison to the new revised version.