- Rating
- 3
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents AGSpec, a framework for improving retrieval-based speculative decoding in coding-agent pipelines. The key observation is that existing retrieval-based drafters do not fully exploit the structure of agent workloads. Reusable text may reside in previous model calls or repository files rather than only in the current prompt, while suitable draft lengths vary substantially across agent roles and execution stages. AGSpec introduces two main components. First, it maintains separate global, session, and workspace corpora. Workspace artifacts are indexed in both their original form and the form in which an agent is expected to emit them, such as unified-diff format. AGSpec also uses source-aware policies to select among candidate continuations. Second, it controls draft length using an offline-profiled, agent-specific cap together with an online scaling factor updated from verification feedback. The method is evaluated on SWE-bench Verified and TeamBench using three 24–27B models, two retrieval engines, batch sizes of 1 and 16, and several retrieval-based and model-based speculative-decoding baselines. Additional experiments are conducted on LiveCodeBench and Terminal-Bench. On the two main repository-level benchmarks, AGSpec achieves 2.27–4.37× the throughput of autoregressive decoding at batch size 1 and 1.08–4.76× at batch size 16. It improves throughput by 18.0% on average over the fastest prior method in each setting. The ablations indicate that cross-call session history provides most of the retrieval gain, while workspace and emission-form indexing and adaptive draft-length control provide further improvements.
Critical strengths
1.The method is well aligned with the structure of coding-agent workloads.
AGSpec identifies two concrete limitations of existing retrieval-based speculative decoding: incomplete access to reusable context and context-insensitive draft lengths. It addresses both with relatively simple and well-motivated mechanisms. In particular, organizing text into global, session, and workspace corpora and indexing repository artifacts in agent-emission form are well suited to iterative code-editing workflows. This is consequential because the resulting design is applicable to multiple retrieval engines rather than being tied to a particular underlying retrieval data structure.
2.The evaluation is broad and provides useful explanations for the observed improvements.
The paper evaluates three target models, four benchmarks, two retrieval engines, two batch-size regimes, and several strong baselines. The component analyses also help identify the sources of improvement. Retaining session history substantially increases accept length, workspace indexing provides additional gains, and online draft-length adaptation sharply reduces rejected tokens at high batch sizes. The overhead analysis further suggests that AGSpec itself accounts for only a small fraction of decoding-step time.
Critical weaknesses
1.The current evaluation does not establish end-to-end latency improvements for complete coding-agent tasks.
The primary metric is generation throughput, explicitly excluding prefill. It also excludes tool execution, environment interaction, scheduling, and other agent-framework overheads. Coding-agent sessions can involve long input contexts and expensive tool calls, so a large decode-only speedup may translate into a much smaller reduction in user-visible completion time. Since the paper is primarily motivated as an acceleration method for complete agent pipelines, the absence of end-to-end latency measurements limits the practical significance of its central claim.
2.The comparison may be sensitive to asymmetric workload-specific tuning.
AGSpec uses per-model and per-benchmark corpora, offline agent-specific draft caps, and a hardware-derived break-even threshold. In contrast, several baselines use a single fixed draft length or their published/default vLLM configurations, without reported parameter sweeps on the evaluated hardware and workloads. Because the average improvement over the fastest prior method is 18%, reasonable retuning of baseline draft lengths or speculation factors could materially affect the quantitative ranking. The evaluation also does not show how well AGSpec’s global corpus and offline profiles transfer across benchmarks, repository distributions, or agent pipelines.
Questions for authors
1.Clarification that could affect my assessment: Can the authors report end-to-end wall-clock session latency, including prefill and agent-framework overhead, for at least SWE-bench Verified and TeamBench? A breakdown into prefill, decoding, retrieval/proposal, and tool/environment time would help establish how much of the reported generation-throughput gain is visible to users.
2.Clarification that could affect my assessment: Were baseline draft lengths and speculation factors tuned or swept on the same hardware and workloads? Please provide either an oracle-tuned or swept comparison for the strongest baselines, or sensitivity results demonstrating that the reported configurations are close to optimal.
3.Clarification that could affect my assessment: How does AGSpec perform in a cold-start or transfer setting, for example when using a global corpus and draft-length caps constructed from a different benchmark, repository distribution, or agent scaffold? This would clarify whether representative workload-specific trajectories must be collected before deployment.
4 comments
W2 / Q2. The comparison may be sensitive to asymmetric tuning. Were baseline draft lengths and speculation factors tuned on the same hardware and workloads?
Sweep at batch size 1. We swept the draft length K of every chain baseline, and the speculation factor of SuffixDecoding, on TeamBench with Gemma3 at batch size 1. The table reports end-to-end speedup over autoregressive decoding at the best value (30 sessions).
| Baseline | Values swept | Best value | Speedup at best value |
|---|---|---|---|
| EAGLE-3 | K = 3, 5, 7 | K = 5 | 1.08× |
| REST | K = 4, 10, 16, 32 | K = 16 | 1.62× |
| PLD | K = 4, 10, 16, 32 | K = 16 | 1.96× |
| FastCoder | K = 4, 10, 16, 32 | K = 16 | 2.15× |
| SAM-Decoding | K = 4, 10, 16, 32, 64 | K = 16 | 2.46× |
| SuffixDecoding | speculation factor 1, 2, 4 | factor 4 | 2.70× |
| AgSpec (SAM) | — | — | 2.86× |
Tuning raises some baselines by up to 16%, but none reaches AgSpec. The strongest tuned baseline, SuffixDecoding with a speculation factor of 4, stays 6% behind. With 30 sessions, the paired session-level 95% CI of this gap is [0.81, 1.32].
Sweep at batch size 16. We also swept K at batch size 16 with prefix caching on (TeamBench, Gemma3, 64 sessions).
| Method | K = 10 | K = 16 | K = 32 | AgSpec on the same engine |
|---|---|---|---|---|
| SAM-Decoding | 2.03× | 1.80× | 1.31× | 2.56× |
| SuffixDecoding | 2.60× | 2.70× | 2.69× | 2.83× |
Long fixed drafts hurt under batching, and no single K suits both engines. On each engine, AgSpec beats the best-tuned baseline.
All main settings. Across all main settings, we also compare the faster AgSpec variant with the faster of SAM-Decoding and SuffixDecoding in each setting. The table reports end-to-end speedup with paired session-level 95% CIs (batch size 1, 30 sessions, default baseline configurations).
| Benchmark | Devstral | Gemma3 | Qwen3.6 |
|---|---|---|---|
| TeamBench | 1.17× [1.01, 1.41] | 1.19× [1.08, 1.33] | 1.14× [1.07, 1.21] |
| SWE-bench Verified | 1.06× [1.03, 1.09] | 1.15× [1.07, 1.25] | 1.11× [1.02, 1.22] |
AgSpec is 6–19% faster than the fastest prior method in every setting, and every CI excludes 1.
Native trees. We also ran native tree configurations of the baselines on TeamBench at batch size 1. None of them closes the gap.
| Method | Configuration | Devstral | Gemma3 |
|---|---|---|---|
| EAGLE-3 | static tree, 60 nodes | 1.02× | 1.48× |
| EAGLE-3 | dynamic tree | 0.92× | 1.06× |
| FastCoder | tree, 64 nodes | 1.30× | 2.20× |
| REST | tree, 64 nodes | 0.79× | 1.88× |
| AgSpec (SAM) | chain | 2.42× | 2.86× |
Q3. How does AgSpec perform in a cold-start or transfer setting, with a global corpus and caps built from a different benchmark?
Cold start. Only the global corpus needs prior data. The session and workspace corpora are built online from the first request.
Transfer. To test transfer, we ran AgSpec on TeamBench with offline material from SWE-bench Verified, a benchmark with different repositories and a different agent pipeline. The global corpus comes from SWE-bench Verified traces. The caps come from SWE-bench agents mapped to the closest TeamBench roles. The workspace and session corpora are built online as usual (batch size 1, 30 sessions).
| Configuration (Gemma3) | Speedup over AR |
|---|---|
| SAM-Decoding | 2.41× |
| SuffixDecoding | 2.32× |
| AgSpec, SWE-bench corpus and caps | 2.49× |
| AgSpec, TeamBench corpus and caps | 2.86× |
AgSpec beats both baselines without any trajectory from the target workload. Material from the same workload adds a further 15%, which a deployment gains as its own history accumulates.
We will add these results to the revision.
Gentle Reminder to Review Our Responses
Dear Reviewer jxur,
We appreciate the important questions you raised regarding the practical effectiveness and fairness of our evaluation. To address them, we have provided (1) end-to-end latency measurements for complete agent tasks, (2) baseline hyperparameter sweeps showing AgSpec's advantage over tuned baselines, and (3) cross-benchmark transfer experiments for the global corpus and draft-length caps.
We would welcome any further feedback on our responses and hope they have satisfactorily addressed your concerns.
Sincerely, The Authors
Thank you for the detailed response and the additional experiments. The end-to-end latency breakdown, the baseline parameter sweeps at both batch sizes, and the cold-start and cross-benchmark transfer results have addressed my main concerns. These results provide stronger evidence for both the practical benefit and robustness of AGSpec. I encourage the authors to include these results and clarifications in the revised paper.
I will maintain my current positive score.
Thank you for the thoughtful questions. We address each point below.
We now measure end-to-end session latency on both benchmarks at batch size 1 (30 sessions). The tables give the mean time per session in seconds, broken into phases. Speedup is end-to-end over autoregressive decoding, with paired session-level 95% bootstrap CIs for AgSpec. TeamBench times exclude benchmark setup and grading, so that table has no test column.
TeamBench
SWE-bench Verified
Speculative decoding shortens only the decoding phase. Prefill stays the same, and retrieval costs at most 2 s per session. On TeamBench, decoding takes 86–94% of an autoregressive session, so most of the gain reaches the user. On SWE-bench Verified, tool calls and Docker tests take about 80–120 s per session for every method. This time bounds the end-to-end gain. Devstral decodes fastest, so tools dominate its session and its gain is 1.49–1.54×. AgSpec still cuts a Gemma3 session from 518 s to 207 s and a Qwen3.6 session from 527 s to 231 s.