acceptodds
Under review as a conference paper at ICLR 2027

StrataSpec: Self-Speculative Decoding with Structured Routes for Efficient LLM Serving

Abstract

Efficient LLM serving under limited compute depends on batching, which amortizes each model pass across several requests. Self-speculative decoding (SSD) reduces proposal cost further without a separate draft model by drafting through a route, a subset of the target model’s attention and MLP sub-layers, before full-model verification. Existing methods adapt this route to each request from verifier feedback, but assess each choice in isolation. A shared proposal pass executes the union of those routes. Because independently chosen routes rarely skip the same sub-layers, their union can reactivate almost the full model and give different row sets to its sub-layers. Much of the per-request saving then disappears at the batch level. We introduce StrataSpec, which retains request-specific route choice within one compact, regular proposal pass. Its routes form a nested family in which every richer route contains the preceding one. The union of any mixture is exactly its richest assigned route, and sorting requests by route gives each sub-layer a contiguous suffix of rows. A batch planner chooses routes and draft length from the incremental work they add to the active batch. StrataSpec improves throughput by up to 4.40× over the strongest of three recent adaptive SSD baselines and sustains 98.7–99.3% SLO attainment under continuous arrivals, while baseline queues grow by orders of magnitude.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
self-speculative decoding, LLM serving, continuous batching, layer skipping

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_im3GOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies batched serving for training-free, request-adaptive self-speculative decoding (SSD). The authors observe that independently selected request-level routes compose poorly under batching: their union can reactivate most skipped sub-layers and produce irregular active-row layouts. StrataSpec addresses this issue by organizing routes into a nested frontier, which bounds the active union and induces a shared suffix order for execution. A batch-aware planner then jointly selects request tiers and draft length based on estimated committed progress and marginal batch cost.

Critical strengths
  1. The proposed system is technically coherent. The nested-route design simultaneously controls the batch-level route union and enables contiguous-suffix execution, and the batch-aware planner is well aligned with this structure.

  2. The component analysis provides useful evidence for the proposed mechanism. In particular, the paper separates gains from route composition and execution layout, rather than relying only on end-to-end throughput numbers.

Critical weaknesses
  1. My main concern is the practical significance of the problem setting. The paper does not sufficiently justify why training-free, request-adaptive SSD is an important design point for throughput-oriented batched serving. The motivating problem arises because independently selected layer-skipping routes interact poorly under batching. However, this routing paradigm was originally motivated by avoiding an auxiliary drafter and additional training, rather than by maximizing serving throughput under batching and concurrency constraints. Once the objective shifts to batched serving throughput, it is unclear why SSD is preferable to using a more conventional speculative-decoding architecture. As a result, although StrataSpec provides a coherent solution to the batching inefficiency of adaptive SSD, the paper does not establish why solving this particular inefficiency is important for practical LLM serving.

  2. The evaluation is restricted to other training-free adaptive SSD methods, which makes it difficult to assess StrataSpec as a general serving solution. Since the paper does not establish adaptive SSD as a necessary or generally preferable design point for batched serving, comparisons only within this family are insufficient to assess the practical value of StrataSpec. The evaluation should also include broader serving alternatives, such as vanilla autoregressive decoding and representative mainstream speculative-decoding methods, under the same hardware and memory budget. In particular, it would be important to compare the memory saved by avoiding an auxiliary drafter and the corresponding throughput trade-off.

  3. The empirical coverage is relatively narrow for a serving paper. All experiments use a single RTX 4090 and 3B–8B models, although the proposed cost model and benefits depend strongly on weight activation, active-row count, and KV-cache cost. The paper itself also shows that a much simpler common-route design is only about 22–30% below StrataSpec, suggesting that a substantial fraction of the headline gains comes from correcting the poor batching behavior of independent routing.

Questions for authors
  1. What is the intended advantage of training-free, request-adaptive SSD in a throughput-oriented batched serving setting, compared with using a lightweight auxiliary drafter? In particular, could the authors clarify which deployment constraints make preserving this SSD design point preferable?

  2. Could the authors compare StrataSpec with vanilla autoregressive decoding and representative mainstream speculative-decoding methods such as EAGLE/MTP under the same hardware and memory budget? Reporting both memory usage and throughput would help quantify the trade-off introduced by avoiding an auxiliary drafter.

Reviewer_Xx5AOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reveals and quantifies the problems of path union inflation and row layout degradation in batch self speculation decoding: the paper points out that when multiple independent adaptive paths of requests are merged and executed in the same draft stage, the set of sub layers actually activated by the batch is equal to the union of all request paths. At the same time, the paper decomposes the cost of batch drafts into functions of shared weight activation, active request line count, and key value cache page count. The nested path front structure and continuous suffix execution mechanism were proposed, and a dual time scale batch planning algorithm and online dynamic verification evaluation were designed in this paper. The author implemented an end-to-end system based on vLLM and achieved significant throughput improvement and higher service level target achievement rate compared to KnapSpec, CLaSp, and SWIFT on HumanEval, GovReport, AIME, and ShareGPT online loads.

Critical strengths

Revealed and quantified the problems of path union inflation and row layout degradation in batch self speculation decoding: The paper points out that when multiple independent adaptive paths of requests are merged and executed in the same draft stage, the set of sub layers actually activated by the batch is equal to the union of all request paths. At the same time, the paper decomposes the cost of batch drafts into functions of shared weight activation, active request lines, and key value cache pages, formalizing the batch cost model from the perspective of marginal workload. Proposed nested path front structure and continuous suffix execution mechanism: The paper organizes candidate paths into nested hierarchical sequences that strictly satisfy containment relationships, so that the merging of batch paths under any request level allocation is strictly limited to the highest level path allocated in the current batch. By non decreasing sorting of requests within the batch by hierarchical index, single kernel call and unified memory view reuse are achieved without the need to perform layer by layer decentralized collection, reassembly or splitting of repeated kernel calls

Critical weaknesses

1、Contradiction in Nested Frontier Construction: The introduction explicitly criticizes post-hoc inclusion ordering for removing critical layers, advocating for joint containment enforcement from the outset. However, Appendix B.1 adopts KnapSpec’s DP to independently generate states per skip budget, relying on a posteriori filtering to form the nested chain. Since DP state transitions lack containment constraints, this post-hoc selection lacks theoretical guarantees and risks empty sets or forced full-model fallback when adjacent budgets are incompatible

2、Unverified Dual-KV and Materialization Overheads: The dual-KV cache architecture and layer promotion mechanisms (Appendix B.2) lack critical memory footprint and latency analyses on resource-constrained hardware (e.g., 24GB GPUs). The physical cost of materializing skipped layers is omitted from the cost models.

3、Anomalous Baseline Latency and Hyperparameter Omission: Table 3 exhibits internal contradictions: KnapSpec’s proposal latency inexplicably decreases from B=4B=4 to B=16B=16 (e.g., dropping from 14.24/16.52 ms to 9.55/4.20 ms) despite batch-level sparsity degradation. Furthermore, core hyperparameters (e.g., planning fraction, thresholds, scan budgets) are entirely omitted, and without real-world millisecond-level profiling of frontier refreshes, the flat P95 tail-latency curves in Figure 6 lack credibility and reproducibility.

Questions for authors

Please clarify in detail the specific physical storage relationship between the draft key value cache and the target key value cache, explain the specific implementation operation of the materialized attention missing sub layer cache block during hierarchical promotion, and provide a comparison table of peak memory usage for different batch sizes and context lengths under 24GB of video memory

Reviewer_jUHhOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes to improve the performance of self-speculative decoding (SSD) by more intelligently selecting routes for requests within a batch together. The paper shows that independently choosing a route (a subset of its own attention/MLP sub-layers) for each request can lead to significant inefficiency because a layer cannot be skipped even if only one request uses it. Instead, the proposed approach, StrataSpec, constrains routes to form a nested tiers and choose the tier more properly considering the true cost when running a batch. The experimental results on an RTX 4090 with Qwen3 & Llama-3.2 show that StrataSpec can significantly (1.73-4.4x) improve performance compared to the previous work.

Critical strengths

(+) The paper addresses an important problem of improving the performance of a speculative decoding. The main idea of planning the routes considering the overlap among the requests within a batch is a good observation and the experiments show significant performance gains.

(+) The paper does a good job describing the key idea and provide detailed experimental results.

Critical weaknesses

(-) The paper mainly studied old/small models (Qwen3, Llama-3) on a desktop GPU (RTX 4090) even though SSD will make most sense on a data-center GPU that need to process requests from many users together. In that sense it is not clear how much benefit this idea will provide for more recent and future-generation hardware and models.

(-) The experiments are performed only for synthetic arrival patterns instead of more realistic workload patterns.

(-) The paper compares StrataSpec with the three baselines. However there is no comparison with the additional previous work the paper references that seem closely related - BatchCond and DREX. Also, it is not clear how the SSD-based techniques such as StrataSpec compare with other types of adaptive speculative decoding techniques that the paper mentions in the related section - such as TETRIS and FASER.

Questions for authors
  • How does StrataSpec perform for newer generation datacenter GPUs and more modern/larger models?

  • How does StrataSpec perform for a more realistic workload (arrival distribution)?

  • How does StrataSpec's performance compare with more recent proposal (BatchCond and DREX) and other types of adaptive speculative decoding?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.