- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Context and problem. Autoregressive LLMs generate one token per forward pass, so latency grows with output length. The paper observes that many responses contain intrinsic parallelism: segments such as list items or independent points do not depend on each other and could be generated at the same time. Exploiting this raises three challenges. The first is identifying which segments are truly independent. The second is keeping branches isolated during generation while still producing a coherent result. The third is assigning position IDs when branch lengths are not known in advance. Prior work in this line has clear limitations. SoT relies on prompting and re-prefills the context for each point. APAR discards branch KV caches when merging. PASTA pre-allocates position ranges from predicted branch lengths.
Method. The paper proposes Adaptive Serial-Parallel Decoding (ASPD), which fine-tunes the target model itself and uses no draft model. It has three parts:
- (1) Parallel data pipeline: an LLM rewrites existing responses into an explicit branch format, with three candidates per response. Branch independence, structural integrity, and answer correctness are each checked by LLM majority voting. Among the candidates that pass, the most parallel one is kept.
- (2) Internal parallelization: branches are generated in a single sequence. A branch-invisible attention mask lets each branch see only the main branch and itself. Tokens generated at the same step in different branches share the same position ID, so each branch looks like normal autoregressive decoding from its own perspective.
- (3) Hybrid Decoding Engine: the model emits branch titles and a special token to enter parallel mode, decodes all branches together, and then returns to serial mode. The following text sees all branches, reuses their KV caches, and resumes at the true sequence position.
The paper also gives a likelihood analysis: the approximation error of treating branches as independent equals their conditional total correlation.
Contributions. (i) An automatic pipeline that builds parallel training data from existing responses with independence and integrity checks. (ii) A single-sequence parallel decoding scheme with branch-invisible masks and shared position IDs, which allows serial–parallel switching without re-prefill. (iii) An empirical evaluation on general and RAG tasks.
Validation. The paper fine-tunes Vicuna-v1.3-7B and Qwen2.5-7B-Instruct on ShareGPT data processed by the pipeline. It evaluates on MT-Bench, Vicuna Bench, and an out-of-domain RAG Bench (200 questions) at batch size 1, with quality scored by Qwen3-235B-A22B as an LLM judge. Baselines are the original model, a model fine-tuned serially on the same data, SoT, APAR, and APAR with improved data.
- On Vicuna-7B, ASPD improves tokens per second by 1.30×, 1.82×, and 1.46× over the original model on MT-Bench, Vicuna Bench, and RAG Bench. Speedups on individual categories range from 1.0× to 3.10×.
- Quality is close to the serially fine-tuned model: 5.59 vs. 5.59 on MT-Bench and 7.74 vs. 7.70 on Vicuna Bench for Vicuna-7B; 8.15 vs. 7.98 and 9.03 vs. 9.11 for Qwen2.5-7B. It is higher than APAR and SoT.
- Wall-clock latency drops by 33.9–44.9% relative to the serially fine-tuned model.
- Ablations compare data pipelines (APAR, PASTA), attention mask visibility, and position-ID schemes.
- Additional results cover batched serving with SGLang (batch size 1 to 128), internal MRC benchmarks (1.35× speedup), and math tasks, where gains are smaller (1.17× on MATH500, 1.08× on AIME).
Critical strengths
1. A clean single-sequence design for serial–parallel switching. Prior work in this line each has a concrete failure point. SoT re-prefills the context for every point, APAR discards branch KV caches when merging, and PASTA relies on predicted branch lengths to allocate position IDs. ASPD avoids all three. Branches are decoded in the same sequence with a branch-invisible mask. Tokens at the same step share a position ID, so no length prediction is needed. When decoding returns to serial mode, the following text attends to all branch KV caches and resumes at the true position, with no re-prefill. The position-ID ablation in Table 3 supports this choice: the shared, sequential scheme (Same-Seq) clearly beats length prediction (Predict), with a score of 7.64 vs. 6.75 and 104.21 vs. 59.11 tokens/s. Why it matters: the design is simple, removes known sources of overhead and quality loss in earlier methods, and is a reasonable foundation for this type of parallel decoding.
2. An automatic data pipeline with explicit verification. Instead of rule-based splitting as in APAR, the pipeline rewrites existing responses with an LLM and then checks branch independence, structural integrity, and answer correctness by majority voting. Groups that fail are reverted to serial form. In the data-pipeline ablation (Table 3), this gives a much higher score than APAR's or PASTA's data (7.64 vs. 5.81 and 4.98) at similar or higher tokens per second. Why it matters: deciding which segments can safely be parallelized is the main difficulty of this approach, and the pipeline provides a practical, model-agnostic way to build such data.
Critical weaknesses
1. The central claim, speedup without quality loss, is not reliably supported by the evidence.
- Speed and output length. Two metrics disagree. The token-throughput speedups in Fig. 4 are 1.30×, 1.82×, and 1.46× on MT-Bench, Vicuna Bench, and RAG Bench. The latency reductions in Table 5 correspond to about 1.81×, 1.51×, and 1.60×. The two use different baselines (the original model vs. the serially fine-tuned model). However, the size of the gap, and the fact that its direction varies across benchmarks, suggest that ASPD's outputs differ in length from the baselines. No length statistics are reported. Length affects both throughput-based speedup and LLM-judge scores.
- Source of the quality gain. Most of the improvement over the original model comes from the rewritten training data. Serial fine-tuning on the same data alone raises Vicuna Bench from 6.21 to 7.70. APAR trained on improved data (APAR*) reaches 7.62, against 7.74 for ASPD. The reported 26.89% gain over APAR therefore mainly reflects data quality, not the parallel decoding method.
- Evaluation protocol. Quality is measured by a single LLM judge on small benchmarks (80 questions each for MT-Bench and Vicuna Bench), with sampling at temperature 0.7 and no variance across runs.
- Internal inconsistencies. Section 4.3.2 states that Shared masks consistently outperform Indep masks, but Table 3 shows the opposite (7.64 vs. 4.64 with Seq position IDs). Table 3's baseline score (6.21) matches the original model on Vicuna Bench, yet ASPD's score there (7.64) does not match Table 1 (7.74). Its throughput ratio (104.21/53.19 ≈ 1.96×) also differs from the reported 1.82×.
Why it matters: these issues bear directly on the paper's main claim. Without length statistics, multiple runs, and consistent numbers, it is hard to tell how much of the speedup and quality preservation is genuine.
2. Limited evaluation scope and insufficient detail for reproducibility. The method is evaluated only on two older 7B models (Vicuna-v1.3-7B and Qwen2.5-7B-Instruct). Several key details are missing: the hardware, the inference framework for the main results, how the autoregressive baseline is implemented, whether fine-tuning is full-parameter or LoRA, and which LLM performs the data rewriting. Yet the main metrics (tokens per second and latency) depend directly on hardware and implementation. In addition, the batched-serving study (Table 2) reports ASPD's speedup at batch size relative to ASPD itself at batch size 1. This shows only that batching helps ASPD, not whether ASPD still beats autoregressive decoding under batching, where spare compute shrinks. Why it matters: it is unclear whether the speedups reproduce or hold in more realistic settings with larger models or batched serving.
Questions for authors
-
Output length and the speed metrics. Could the authors report the average output length of ASPD, the original model, and the serially fine-tuned model on each benchmark? Please also explain why the token-throughput speedups in Fig. 4 (1.30×, 1.82×, 1.46×) and the latency reductions in Table 5 (about 1.81×, 1.51×, 1.60×) differ so much, with the direction of the gap varying across benchmarks. If lengths differ, how does this affect the LLM-judge scores?
-
Inconsistencies between the text and tables. Which benchmark does Table 3 use? Its baseline score (6.21) matches the original model on Vicuna Bench, but ASPD's score (7.64) and throughput ratio (about 1.96×) do not match Table 1 (7.74) or the reported 1.82×. Section 4.3.2 also states that Shared masks outperform Indep masks, whereas Table 3 shows the opposite. Could the authors clarify which is correct?
-
Experimental setup and batched serving. What hardware and inference framework are used for the main results, and how is the autoregressive baseline implemented? Is fine-tuning full-parameter or LoRA, and which LLM performs the data rewriting? For Table 2, could the authors compare ASPD against autoregressive decoding at the same batch sizes, rather than against ASPD at batch size 1?