acceptodds
Withdrawn from ICLR 2027

SpecSys: Selective Speculative Decoding and Completion-Aware Scheduling for Efficient Agentic LLM Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large language model (LLM)-powered agents often invoke multiple LLM requests, making LLM inference a major contributor to end-to-end task completion time. Our measurements reveal that the dominant bottleneck shifts with serving load: from decoding at low concurrency to queueing at high concurrency. Speculative decoding (SD) is a state-of-the-art technique for accelerating LLM inference, but its benefits diminish as serving load increases and can even become negative under high load. Existing adaptive SD methods cannot eliminate this problem because they lack native non-SD execution, while conventional serving schedulers do not fully exploit workflow structure and completion progress of agent tasks. To address these challenges, we design SpecSys, an efficient inference system for agent workloads that enables load-aware selective SD with completion-aware scheduling. SpecSys exposes an externally controllable runtime interface that enables complete iteration-level switching between native SD and non-SD execution. It also employs a lightweight history-based estimator to predict the remaining steps of each agent task based on its workflow state and request characteristics. SpecSys adapts these mechanisms with serving load: it selectively enables SD to reduce decoding latency under low load, while prioritizing tasks with fewer predicted remaining steps under high load to reduce queueing delay. Across three LLMs, four agent workloads, three agent workflows, and two hardware platforms, SpecSys consistently reduces average job completion time (JCT) over strong baselines, including ThunderAgent, vLLM Dynamic SD, and SGLang Adaptive SD, achieving up to a 2.10 speedup under resource-constrained settings. The source code of SpecSys is anonymously available at https://anonymous.4open.science/r/SpecSys-Code-E03D/.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Large Language Models, LLM Inference, LLM Serving, LLM Agents, Speculative Decoding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oTQ1OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces SpecSys, an LLM inference system designed to reduce the end-to-end Job Completion Time (JCT) for multi-step agent workloads. The study finds that performance bottlenecks shift depending on the level of concurrency: at low concurrency, the primary bottleneck is the decoding process, whereas at high concurrency, it shifts to queuing latency. To address this, SpecSys integrates three mechanisms: (1) an externally controllable interface that enables switching between "speculative decoding" (SD) and "non-speculative decoding" execution paths at the iteration level; (2) a lightweight, history-based estimator that leverages workflow information and execution state to predict the remaining steps for an agent task; and (3) a load-aware scheduling strategy that selectively enables speculative decoding under low-load conditions, while prioritizing tasks with fewer predicted remaining steps under high-load conditions.

Critical strengths
  1. The research motivation is clear, and the work makes a tangible contribution at the system level.

This paper identifies a critical challenge in agentic large language model (LLM) services: optimal inference strategies vary according to system load. Consequently, optimizing individual LLM calls in isolation does not necessarily minimize the end-to-end completion time for agentic tasks. The implementation of this approach across two mainstream inference engines further underscores its practical significance.

  1. Significant improvements in end-to-end performance, supported by ablation studies.

Experimental results demonstrate a substantial reduction in job completion time (JCT), particularly in resource-constrained scenarios. Ablation studies reveal that the switching mechanism for speculative decoding and the "completion-aware scheduling" strategy offer complementary benefits. The evaluation encompasses a diverse range of models, hardware configurations, and workload types, with additional metrics provided for tail latency and throughput. These findings compellingly demonstrate that the system extends beyond simple decoding optimizations, effectively enhancing the overall efficiency of agentic services.

Critical weaknesses
  1. There is limited innovation at the methodological level, and the approach lacks a comprehensive comparison with other scheduling strategies.

The two core optimization principles proposed—disabling "speculative decoding" when its overhead outweighs its benefits, and prioritizing tasks with a smaller remaining workload—are essentially direct extensions of existing concepts: "adaptive inference" and "shortest-remaining-processing-time" (SRPT) scheduling. Furthermore, the method used to estimate the "number of remaining steps" relies primarily on simple historical statistics and rule-based state extraction.

  1. Concerns regarding the authenticity of the evaluation and the comparability of baselines raise questions about whether the reported performance gains can generalize to broader, real-world scenarios.

According to Appendix K, the end-to-end experiments reconstruct 4,096 agent tasks by cyclically reusing segments from a collection of 1,000 recorded traces (selected from a held-out dataset). Although the experiments incorporate real-world request arrival timestamps, the actual execution patterns of the agents are "replayed" from existing traces rather than generated dynamically through real-time, closed-loop agent interactions. This repetitive nature may affect workload diversity, the effectiveness of prefix caching, and the predictability of remaining execution steps. Consequently, it remains unclear whether the reported performance improvements would generalize to more diverse and dynamically evolving agent workloads.

Questions for authors

Q1. Could the authors compare the proposed "remaining-step-aware" scheduling strategy against simpler task-level scheduling strategies—specifically those based on "predicted remaining tokens" or "predicted execution time"? Furthermore, how sensitive are the gains from this scheduling approach to variations in the computational cost of individual agent steps?

Q2. Appendix K notes that the end-to-end evaluation constructs 4,096 tasks by cyclically reusing pre-recorded agent trajectories. How does this reuse affect the behavior of the "prefix cache" and the accuracy of the "remaining-step" predictions?

Q3. Could the authors clarify the rationale for choosing to display the best-performing results from either SpecSys-vLLM or SpecSys-SGLang in Figure 4, rather than presenting comparisons based on matched backends? Additionally, given that Table 10 involves different versions of vLLM, how do the authors ensure that the observed performance differences are indeed attributable to SpecSys, rather than to variations in the underlying inference engine versions?

Reviewer_5GjfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SpecSys, an adaptive inference system for LLM-based agents that addresses changing performance bottlenecks under different serving loads. The authors observe that decoding dominates latency at low concurrency, while request queueing becomes more significant under high concurrency, where speculative decoding (SD) may introduce additional overhead.

SpecSys combines two mechanisms: load-aware switching between native SD and non-SD execution paths, and completion-aware scheduling that prioritizes agent tasks with fewer predicted remaining steps. The remaining-step estimator uses workflow information, historical execution statistics, and prompt-derived state features.

The system is implemented on vLLM and SGLang and evaluated across multiple LLMs, agent workflows, workloads, and GPU platforms. The results demonstrate improvements in average job completion time (JCT), achieving up to a 2.10× speedup over the evaluated baselines. Ablation studies further examine the contributions of the two mechanisms.

Critical strengths

S1. Practical load-adaptive inference optimization.

The paper addresses a meaningful systems problem by considering how serving bottlenecks change with concurrency. Its implementation of native SD/non-SD execution-path switching provides a practical way to avoid unnecessary speculative decoding overhead under high load. Combining this mechanism with agent-level scheduling is useful for improving overall task completion time.

S2. Solid implementation and encouraging experimental results.

SpecSys is implemented on two widely used inference frameworks and evaluated across different models, hardware platforms, workloads, and serving traces. The results show consistent JCT improvements, while the ablation studies demonstrate that both SD switching and completion-aware scheduling contribute to performance. The additional evaluation of prediction accuracy, tail latency, and throughput also strengthens the empirical evidence.

Critical weaknesses

W1. Limited technical novelty and insufficient evidence of independent contributions.

The paper mainly combines load-adaptive speculative decoding with remaining-work-based scheduling, both of which have been extensively explored in prior research. Selective SD is closely related to existing adaptive SD methods, while completion-aware scheduling largely follows the classical shortest-remaining-work-first principle. The remaining-step estimator also relies primarily on historical statistics and relatively simple heuristic features. Although the native SD/non-SD path switching is a useful engineering improvement, the overall approach appears to be largely an integration of established techniques. Moreover, the ablation results indicate that completion-aware scheduling accounts for a substantial portion of the performance gains, yet the paper lacks sufficient comparisons with simpler remaining-work-based schedulers or other relevant agent-aware scheduling methods. This makes the independent novelty and contribution of the proposed scheduling mechanism unclear.

W2. Limited validation in realistic dynamic agent workflows.

The end-to-end evaluation primarily relies on replaying previously collected agent execution traces rather than running agents that dynamically determine their next actions based on real-time model outputs and tool feedback. The evaluated workflows also focus on structured, sequential multi-step execution patterns. While trace replay provides controlled comparisons, it does not fully capture the variability of real-world agent systems, including changing execution paths, unpredictable tool behavior, and shifting task distributions. Furthermore, the remaining-step predictor depends on predefined workflow types and historical execution patterns, making its robustness to unseen or changing workflows uncertain. Consequently, the current evaluation provides limited evidence of how well SpecSys generalizes to more dynamic agent-serving environments.

Questions for authors

Q1. Independent contribution of the scheduling mechanism.

Completion-aware scheduling closely resembles classical shortest-remaining-work-first scheduling, and the ablation results show that this component contributes substantially to the overall speedup. Could the authors clarify its key differences from existing agent-aware scheduling methods? Would simpler approaches based on historical average task length or estimated remaining work achieve comparable results?

Q2. Effectiveness in real-world dynamic agent execution.

The end-to-end evaluation mainly uses pre-collected agent traces. How would SpecSys perform when agents dynamically change their execution paths based on real-time tool feedback or model outputs? Can the authors provide evidence that the reported improvements would remain effective under such conditions?

Q3. Reliability and generalizability of remaining-step prediction.

SpecSys uses the predicted number of remaining steps for scheduling, although different steps may have substantially different execution costs. Why is remaining-step count preferred over estimated remaining execution time? How robust is the history-based estimator when workload distributions or workflow structures change?

Reviewer_F4F1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents an LLM inference system that combines load-aware selective speculative decoding with completion-aware scheduling for agentic workloads. The authors observe that the dominant performance bottleneck shifts from decoding latency at low concurrency to queueing delay at high concurrency, where SD can even become detrimental. To address this, SpecSys introduces an externally controllable interface for switching between native SD and non-SD execution paths at inference-iteration boundaries. It also employs a lightweight history-based predictor to estimate the remaining execution steps of agent tasks and prioritize those closer to completion. Across three different models, four agent workloads, three workflows, and two GPU platforms, SpecSys achieves 1.18–2.10× improvement in average job completion time (JCT) over their best evaluated baselines. Furthermore their remaining-step predictor helps their end-to-end system perform within 3.5% of an oracle scheduler.

Critical strengths
  1. Practical problem formulation: The paper addresses an important systems challenge in agentic LLM serving, where optimizing individual inference requests does not necessarily minimize end-to-end agent completion time. Combining inference-level optimizations with agent-level scheduling is practically relevant.

  2. Efficient runtime switching mechanism: The externally controllable interface and separate native SD/non-SD execution paths enable low-overhead switching without retaining unnecessary SD execution overhead. The implementation is demonstrated on both vLLM and SGLang.

  3. Extensive evaluation and useful ablations: The authors evaluate multiple models, hardware configurations, agent workloads, and serving traces. The ablations distinguish the contributions of SD switching and completion-aware scheduling, and the oracle comparison helps establish the effectiveness of the remaining-step predictor.

Critical weaknesses
  1. Limited novelty: The observation that speculative decoding (SD) becomes less effective under high serving loads, motivating dynamic switching between SD and autoregressive decoding, has already been explored in works such as SmartSpec/TurboSpec. Similarly, agent-aware scheduling has been studied in prior systems. The main novelty appears to lie in the efficient execution-path switching implementation and remaining-step estimation.

  2. Limited adaptive SD baselines: Although vLLM Dynamic SD and SGLang Adaptive SD support more flexible draft-length adaptation, the authors restrict both to a binary policy (draft length 0 or 3). This makes it difficult to assess the benefits of SpecSys over fully optimized adaptive SD methods.

  3. Simplistic scheduling heuristic: The scheduler prioritizes tasks based on predicted remaining steps rather than remaining execution time. Since agent steps may vary substantially in generation length and computational cost, this heuristic may not generalize well to heterogeneous workloads.

  4. Limited generalizability: The evaluation focuses on EAGLE-3 with a fixed draft length, dense target models, and replayed agent trajectories. It is unclear whether the improvements extend to other SD methods, MoE architectures, or more complex reasoning-heavy agent workloads.

Questions for authors

See weakness section.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.