acceptodds
Under review as a conference paper at ICLR 2027

ASPD: Unlocking Adaptive Serial-Parallel Decoding by Exploring Intrinsic Parallelism in LLMs

Abstract

The growing scale and complexity of large language models (LLMs) lead to high inference latency, primarily due to their autoregressive decoding characterized by the next-token prediction paradigm. By re-examining the outputs of autoregressive models, we find that some segments exhibit parallelizable structures, which we term intrinsic parallelism. Decoding these segments simultaneously (i.e. parallel decoding) can substantially accelerate LLM inference speed. In this paper, we propose an Adaptive Serial-Parallel Decoding (ASPD), which addresses two core challenges: automated construction of parallelizable data and efficient parallel decoding mechanism. To empower efficient adaptive serial-parallel decoding, we implement a Hybrid Decoding Engine which enables seamless transitions between serial and parallel decoding modes while maintaining a reusable KV cache, maximizing computational efficiency. Extensive evaluations across General Tasks and Retrieval-Augmented Generation demonstrate that ASPD achieves unprecedented performance in both effectiveness and efficiency. Notably, on Vicuna Bench, our method achieves up to 3.10x speedup while maintaining response quality within 1% difference compared to autoregressive models, realizing significant acceleration without compromising generation quality.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Parallel decoding, LLM

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2sndOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Context and problem. Autoregressive LLMs generate one token per forward pass, so latency grows with output length. The paper observes that many responses contain intrinsic parallelism: segments such as list items or independent points do not depend on each other and could be generated at the same time. Exploiting this raises three challenges. The first is identifying which segments are truly independent. The second is keeping branches isolated during generation while still producing a coherent result. The third is assigning position IDs when branch lengths are not known in advance. Prior work in this line has clear limitations. SoT relies on prompting and re-prefills the context for each point. APAR discards branch KV caches when merging. PASTA pre-allocates position ranges from predicted branch lengths.

Method. The paper proposes Adaptive Serial-Parallel Decoding (ASPD), which fine-tunes the target model itself and uses no draft model. It has three parts:

  • (1) Parallel data pipeline: an LLM rewrites existing responses into an explicit branch format, with three candidates per response. Branch independence, structural integrity, and answer correctness are each checked by LLM majority voting. Among the candidates that pass, the most parallel one is kept.
  • (2) Internal parallelization: branches are generated in a single sequence. A branch-invisible attention mask lets each branch see only the main branch and itself. Tokens generated at the same step in different branches share the same position ID, so each branch looks like normal autoregressive decoding from its own perspective.
  • (3) Hybrid Decoding Engine: the model emits branch titles and a special token to enter parallel mode, decodes all branches together, and then returns to serial mode. The following text sees all branches, reuses their KV caches, and resumes at the true sequence position.

The paper also gives a likelihood analysis: the approximation error of treating branches as independent equals their conditional total correlation.

Contributions. (i) An automatic pipeline that builds parallel training data from existing responses with independence and integrity checks. (ii) A single-sequence parallel decoding scheme with branch-invisible masks and shared position IDs, which allows serial–parallel switching without re-prefill. (iii) An empirical evaluation on general and RAG tasks.

Validation. The paper fine-tunes Vicuna-v1.3-7B and Qwen2.5-7B-Instruct on ShareGPT data processed by the pipeline. It evaluates on MT-Bench, Vicuna Bench, and an out-of-domain RAG Bench (200 questions) at batch size 1, with quality scored by Qwen3-235B-A22B as an LLM judge. Baselines are the original model, a model fine-tuned serially on the same data, SoT, APAR, and APAR with improved data.

  • On Vicuna-7B, ASPD improves tokens per second by 1.30×, 1.82×, and 1.46× over the original model on MT-Bench, Vicuna Bench, and RAG Bench. Speedups on individual categories range from 1.0× to 3.10×.
  • Quality is close to the serially fine-tuned model: 5.59 vs. 5.59 on MT-Bench and 7.74 vs. 7.70 on Vicuna Bench for Vicuna-7B; 8.15 vs. 7.98 and 9.03 vs. 9.11 for Qwen2.5-7B. It is higher than APAR and SoT.
  • Wall-clock latency drops by 33.9–44.9% relative to the serially fine-tuned model.
  • Ablations compare data pipelines (APAR, PASTA), attention mask visibility, and position-ID schemes.
  • Additional results cover batched serving with SGLang (batch size 1 to 128), internal MRC benchmarks (1.35× speedup), and math tasks, where gains are smaller (1.17× on MATH500, 1.08× on AIME).
Critical strengths

1. A clean single-sequence design for serial–parallel switching. Prior work in this line each has a concrete failure point. SoT re-prefills the context for every point, APAR discards branch KV caches when merging, and PASTA relies on predicted branch lengths to allocate position IDs. ASPD avoids all three. Branches are decoded in the same sequence with a branch-invisible mask. Tokens at the same step share a position ID, so no length prediction is needed. When decoding returns to serial mode, the following text attends to all branch KV caches and resumes at the true position, with no re-prefill. The position-ID ablation in Table 3 supports this choice: the shared, sequential scheme (Same-Seq) clearly beats length prediction (Predict), with a score of 7.64 vs. 6.75 and 104.21 vs. 59.11 tokens/s. Why it matters: the design is simple, removes known sources of overhead and quality loss in earlier methods, and is a reasonable foundation for this type of parallel decoding.

2. An automatic data pipeline with explicit verification. Instead of rule-based splitting as in APAR, the pipeline rewrites existing responses with an LLM and then checks branch independence, structural integrity, and answer correctness by majority voting. Groups that fail are reverted to serial form. In the data-pipeline ablation (Table 3), this gives a much higher score than APAR's or PASTA's data (7.64 vs. 5.81 and 4.98) at similar or higher tokens per second. Why it matters: deciding which segments can safely be parallelized is the main difficulty of this approach, and the pipeline provides a practical, model-agnostic way to build such data.

Critical weaknesses

1. The central claim, speedup without quality loss, is not reliably supported by the evidence.

  • Speed and output length. Two metrics disagree. The token-throughput speedups in Fig. 4 are 1.30×, 1.82×, and 1.46× on MT-Bench, Vicuna Bench, and RAG Bench. The latency reductions in Table 5 correspond to about 1.81×, 1.51×, and 1.60×. The two use different baselines (the original model vs. the serially fine-tuned model). However, the size of the gap, and the fact that its direction varies across benchmarks, suggest that ASPD's outputs differ in length from the baselines. No length statistics are reported. Length affects both throughput-based speedup and LLM-judge scores.
  • Source of the quality gain. Most of the improvement over the original model comes from the rewritten training data. Serial fine-tuning on the same data alone raises Vicuna Bench from 6.21 to 7.70. APAR trained on improved data (APAR*) reaches 7.62, against 7.74 for ASPD. The reported 26.89% gain over APAR therefore mainly reflects data quality, not the parallel decoding method.
  • Evaluation protocol. Quality is measured by a single LLM judge on small benchmarks (80 questions each for MT-Bench and Vicuna Bench), with sampling at temperature 0.7 and no variance across runs.
  • Internal inconsistencies. Section 4.3.2 states that Shared masks consistently outperform Indep masks, but Table 3 shows the opposite (7.64 vs. 4.64 with Seq position IDs). Table 3's baseline score (6.21) matches the original model on Vicuna Bench, yet ASPD's score there (7.64) does not match Table 1 (7.74). Its throughput ratio (104.21/53.19 ≈ 1.96×) also differs from the reported 1.82×.

Why it matters: these issues bear directly on the paper's main claim. Without length statistics, multiple runs, and consistent numbers, it is hard to tell how much of the speedup and quality preservation is genuine.

2. Limited evaluation scope and insufficient detail for reproducibility. The method is evaluated only on two older 7B models (Vicuna-v1.3-7B and Qwen2.5-7B-Instruct). Several key details are missing: the hardware, the inference framework for the main results, how the autoregressive baseline is implemented, whether fine-tuning is full-parameter or LoRA, and which LLM performs the data rewriting. Yet the main metrics (tokens per second and latency) depend directly on hardware and implementation. In addition, the batched-serving study (Table 2) reports ASPD's speedup at batch size kk relative to ASPD itself at batch size 1. This shows only that batching helps ASPD, not whether ASPD still beats autoregressive decoding under batching, where spare compute shrinks. Why it matters: it is unclear whether the speedups reproduce or hold in more realistic settings with larger models or batched serving.

Questions for authors
  1. Output length and the speed metrics. Could the authors report the average output length of ASPD, the original model, and the serially fine-tuned model on each benchmark? Please also explain why the token-throughput speedups in Fig. 4 (1.30×, 1.82×, 1.46×) and the latency reductions in Table 5 (about 1.81×, 1.51×, 1.60×) differ so much, with the direction of the gap varying across benchmarks. If lengths differ, how does this affect the LLM-judge scores?

  2. Inconsistencies between the text and tables. Which benchmark does Table 3 use? Its baseline score (6.21) matches the original model on Vicuna Bench, but ASPD's score (7.64) and throughput ratio (about 1.96×) do not match Table 1 (7.74) or the reported 1.82×. Section 4.3.2 also states that Shared masks outperform Indep masks, whereas Table 3 shows the opposite. Could the authors clarify which is correct?

  3. Experimental setup and batched serving. What hardware and inference framework are used for the main results, and how is the autoregressive baseline implemented? Is fine-tuning full-parameter or LoRA, and which LLM performs the data rewriting? For Table 2, could the authors compare ASPD against autoregressive decoding at the same batch sizes, rather than against ASPD at batch size 1?

Reviewer_Y3agOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces ASPD, a fine-tuning and inference framework that exploits intrinsic parallelism in LLM outputs by identifying semantically independent parts in text with a custom serving engine. It proposes a dataset processing pipeline facilitated by an LLM to uncover independent branches in text. Evaluation results show that ASPD can speed up inference without significant accuracy loss.

Critical strengths
  1. The paper proposes a new direction for trained parallel generation, where branch-level parallelism is the focus instead of token-level like dLLM or semi-AR models.
  2. The framework is quite complete and self-sustained, with an algorithm design, data curation pipeline, and a serving engine. It covers end-to-end deployment from training to inference.
  3. The speedup is impressive, upto 3.1 times speedup without significant accuracy loss
Critical weaknesses
  1. The paper does not do ablation on the custom serving engine. It is also not clear what significant contributions the hybrid decoding engine has made and what challenges it solves. It looks like integration engineering work into existing serving engines.
  2. Table 2 does not show how scalable ASPD is. It looks like the high speedup at large batch sizes are mainly from the increase in batch size itself. It is helpful to separate the gain from ASPD from batching, like adding another baseline with batching but without ASPD.
Questions for authors

See weakness

Reviewer_JWUcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that many LLM responses contain independent segments, such as list items, that can be decoded at the same time. ASPD uses an LLM to rewrite serial responses into a tagged parallel format with LLM-voted independence checks, fine-tunes the model with a mask that hides branches from each other and with shared position ids, and decodes serial and parallel parts inside one sequence while keeping the KV cache. On Vicuna-7B the authors report 1.30x to 1.82x tokens-per-second speedups with quality close to a serially fine-tuned model.

Critical strengths
  • The speedup comes without a visible quality loss. Across three benchmarks and two base models, ASPD scores close to the serially fine-tuned model, and Table 5 shows 34% to 45% lower wall-clock latency. This shows the model can learn when to parallelize on general chat tasks.

  • The design is complete and ablated. Keeping all branches in one sequence avoids re-prefilling, and Table 3 compares four position id schemes, which supports the Same-Seq choice.

Critical weaknesses
  • The overlap with Multiverse (Yang et al., 2025b) is not discussed. As I read it, Multiverse already masks parallel paths from each other, starts them from the same position id, reuses the KV cache at merge, and builds training data by LLM rewriting with automatic checks. Section 2 describes it only as math-focused work on SGLang, and there is no experimental comparison. The new parts seem to be Same-Seq after merging (Multiverse uses the maximum position, i.e. Same-Max), the verification steps, and the chat and RAG domain. The paper should state this difference and compare directly, otherwise the contribution is hard to judge.

  • The speed evidence is inconsistent. Against V-Seq, TPS gives 1.30x on MT Bench and 1.82x on Vicuna Bench, while wall-clock latency in Table 5 gives 1.81x and 1.51x. Appendix A.2 calls these consistent. The gap suggests different output lengths, which are not reported. Table 2 has no serial baseline at the same batch size, so it does not show a speedup under batching. The abstract's "up to 3.10x" comes from one category (Generic), where V-ASPD scores 8.80 vs 9.20 for V-Seq.

Questions for authors
  • What is new compared with Multiverse, and can you compare against it, or against Same-Max with your data?

  • What are the output lengths and per-question latencies for each model, and what is the speedup over serial decoding at the same batch size in SGLang?

  • What setting does Table 3 use?

I would consider raising my score if the Multiverse comparison and the speed and consistency issues are addressed.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.