acceptodds
Under review as a conference paper at ICLR 2027

SpecRoute: Multi-Step Speculation for Efficient LLM Agent Serving

Abstract

LLM agents interleave model reasoning with tool execution, forming observation-dependent sequences of model calls that incur substantial end-to-end latency. Conventional serving optimizations focus on individual LLM calls, while single-step action speculation overlaps the next tool invocation with ongoing reasoning. Both leave the agent’s multi-turn execution chain largely on the critical path. We present SpecRoute, a serving runtime that accelerates LLM agents through multi-step speculative execution. A lightweight draft agent speculates on future tool actions, interleaving tool execution with draft-model inference while the target processes its current context. At target tool-call boundaries, the target performs incremental, prefill-based verification of speculative observations and incorporates accepted results with KV reuse. A slack-aware scheduler coordinates batch formation and target–draft engine switching on a shared GPU, balancing speculative progress against interference with target execution. Across six agent benchmarks, SpecRoute achieves 1.28–1.56× single-request speedups over vLLM with comparable task performance. Under online serving, mean completion time decreases by 28–36% at low load and 9–20% at the highest tested loads.

Primary area:
infrastructure, software libraries, hardware, systems, etc.
Keywords:
Agent Serving, Speculative Execution, GPU Scheduling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_L24dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents SpecRoute, a multi-step speculative framework for LLM agents. It uses a draft agent to execute future tool calls and allows the target model to verify the observations, reducing intermediate reasoning steps. A slack-aware scheduler coordinates draft and target execution on a shared GPU. Experiments on six benchmarks show 1.28–1.56× speedups over vLLM with comparable accuracy.

Critical strengths

1.The paper is well-written and easy to follow, with clear motivation: the challenges of multi-step speculative execution. 2.The evaluation demonstrate end-to-end improvements, achieving up to 1.56× speedup over vLLM.

Critical weaknesses
  1. Limited novelty. Most components of SpecRoute have already been explored in prior work, including multi-step speculative execution, semantic verification, and shared-GPU scheduling. In the Related Work section, the authors briefly describe existing approaches without clearly explaining the fundamental differences and challenges addressed by SpecRoute. I believe the paper will benefit from sufficient comparisons or evaluations against closely related work, such as DSP, DualSpec, and SPAgent. Furthermore, the Motivation section should explain why existing approaches cannot adequately address these challenges.
  2. Concerns about speculation accuracy and effectiveness. The target LLM directly judges whether to accept speculative observations. This raises the following concerns. First, the capability gap between the small drafter and the target model may lead to low acceptance rates on harder tasks when involving multiple steps. Second, LLM-based verification may introduce incorrect acceptance decisions due to hallucinations. However, the paper does not sufficiently evaluate this part. All six benchmarks focus on relatively simple information-retrieval tasks. It remains unclear whether the approach generalizes to more complex agentic workloads (e.g. coding, scientific benchmarks).
  3. Concerns about experimental results. Table 1 shows that SpecRoute even improves answer accuracy on some benchmarks, which is counterintuitive. The author didnot give a sufficient explanation. It is unclear whether these improvements are simply due to experimental variability. Moreover, the paper lacks a detailed evaluation of acceptance rates across different benchmarks and their impact on accuracy and speedup, which I consider one of the most important evaluations for the proposed method. Figure 5(b) is somewhat unclear, and the results for two benchmarks appear to be missing.
  4. Table 1 also only compares accuracy against vLLM. It is unclear whether SpecRoute achieves a better accuracy-efficiency trade-off than other baselines.
Questions for authors

Unless the concerns are fundamentally addressed, I am unlikely to change my rating.

  1. The authors should more explicitly discuss how SpecRoute differs from existing work and how it addresses the challenges identified in the Motivation part.
  2. I suggest strengthening the baseline comparisons. New baselines should including standard speculative decoding in vLLM, existing general speculative execution frameworks, DSP [1], and DualSpec [2]. At a minimum, the paper should clearly explain its qualitative contributions over existing work.
  3. I suggest evaluating more diverse and challenging workloads, such as coding agents or complex deep-research tasks. This would help identify whether increasing task difficulty reduces the speedup.
  4. More analysis of task accuracy and draft accuracy is needed. I think a more interesting report should be when and where speculation fails, how these failures affect accuracy and latency, where may the large model gives wrong verification, and how they influence the choice of speculation depth and program behavior.
  5. Table 1 should include the accuracy of other baselines and provide a clearer accuracy-efficiency comparison. The authors should also explain why speculative execution sometimes improves answer accuracy.
  6. Could the authors also provide a direct comparison with the standalone small model (2B) on these benchmarks? I am concerned that the smaller model alone may already achieve sufficient accuracy, which would weaken the contribution for the proposed method.

[1] Dynamic Speculative Agent Planning

[2] DualSpec: Accelerating Deep Research Agents via Dual-Process Action Speculation

1 comment

Thank you for the detailed rebuttal and additional experiments. I appreciate the authors' efforts. The new results address several of my concerns.

However, two major concerns still remain.

  1. Speculative draft correctness. My primary concern is the correctness of the speculative drafts generated by the 2B model. The authors only present its standalone end-to-end task accuracy. The reported acceptance rates do not directly establish draft correctness or verification reliability. I understand that time is limited, but I believe this issue deserves at least a clear explanation and discussion, as it is central to the proposed method.

  2. Novelty. I consider feedback-driven drafting and shared-GPU scheduling primarily system design choices, and are relatively straightforward. The key potential contribution is observation-level verification. However, its differences from existing methods, such as SPAgent, is unclear. The key innovation of technical mechanisms should be sufficiently explained.

I am therefore inclined to maintain my original rating.

Reviewer_3ACxOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper explores multi-step speculation on tool calls (multiple rounds of tool calling) in agentic workloads. They identify the the opportunity to anticipate tool calls while the target model is thinking and then propose speculatively executing the tool calls, conditioning on the output, and then ultimately conditioning on the tools in the speculation loop. They show significant gains when applying the technique in vLLM.

Critical strengths
  1. The framing of the problem is clear and the paper is generally well written.
  2. The proposed method is relatively straightforward and one could imagine implementing it broadly
  3. The results are strong and support the adoption of the method.
Critical weaknesses
  1. The paper doesn't address the implications of tool-calls/actions that might be state-mutating. Even things like touching files may change the behavior of future tool calls (e.g., which files were most recently opened).
  2. The tasks look mostly like agentic RAG which while relevant is only a small part of most agent workloads. I would be curious how these techniques might apply to terminal bench.
  3. I wonder how tool call length really affects the gains here.
Questions for authors
  1. I was confused about how overlapped thinking is incorporated into the context with the tool calls. I believe the paper introduced an aa which is the thinking context.
  2. Could you talk about how you might address state-modifying actions?
Reviewer_icydOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SpecRoute, a framework that accelerates LLM agents through multi-step speculative tool execution. It combines semantic observation verification, KV reuse, and shared-GPU scheduling to reduce target reasoning and tool latency. Experiments on six search-based benchmarks report 1.28–1.56× speedups over vLLM with comparable answer accuracy.

Critical strengths
  1. Meaningful problem and practical design. The paper addresses the substantial latency caused by sequential reasoning and tool interactions in LLM agents. Its multi-step speculation framework provides a practical approach to reducing target reasoning overhead.
  2. Promising end-to-end performance. SpecRoute achieves 1.28–1.56× speedups across six benchmarks and reduces target reasoning iterations by 40–55%. The improvements also persist under concurrent serving workloads.
Critical weaknesses
  1. Insufficient comparisons with closely related baselines. The paper mainly compares against vLLM, Continuum, and SpecActions, while recent approaches such as SPAgent, DualSpec, and SpecHop have explored similar ideas in speculative agent execution. Without direct comparisons with these methods, it is difficult to assess the novelty and practical advantages of SpecRoute over existing approaches.
  2. Missing ablations on the benefits of multi-step speculation. Although multi-step speculation is the central contribution, the comparison with SpecActions simultaneously changes the speculation depth, verification mechanism, and scheduling strategy. A single-step baseline with the same semantic verification and scheduling mechanisms is needed to isolate the benefits of multi-step speculation and justify its additional complexity.
  3. Limited diversity of end-to-end evaluation. Although six benchmarks are evaluated, all experiments focus on search-based agents, leaving the applicability to other agent workflows unclear. Furthermore, the experiments use only one target/draft model pair and one GPU configuration, without analyzing how performance changes across different models, tool latencies, or hardware settings.
Questions for authors
  1. Could the authors compare SpecRoute with closely related approaches such as SPAgent, DualSpec, or SpecHop under comparable settings?
  2. How much additional speedup does multi-step speculation achieve over single-step speculation when the verification and scheduling mechanisms are kept identical?
  3. How does SpecRoute perform across different target/draft model configurations, tool latencies, and agent workloads?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.