acceptodds
Under review as a conference paper at ICLR 2027

AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines

Abstract

Retrieval-based speculative decoding (SD) drafts tokens by copying continuations from existing text, which suits coding agents that repeatedly reproduce code, logs, and earlier attempts. Yet existing methods fall short in agent pipelines: much of the reusable text is missing from their corpora or stored in a form that differs from what the agent emits, and their draft lengths ignore that accept length varies across agents and drifts over turns. We present AgSpec, a framework that supplies the corpus and draft-length policies that existing retrieval engines lack in coding-agent pipelines. AgSpec retrieves from session, workspace, and global corpora, retaining the ongoing session trajectory and indexing opened files in the agent's emission format. It bounds each agent's draft length with an offline-profiled cap and adapts the length online from verification feedback. On two repository-level multi-agent coding benchmarks, AgSpec outperforms five retrieval-based drafters and EAGLE-3 in most evaluated settings, raising generation throughput over autoregressive decoding up to 4.37 at batch size 1 and 4.76 at batch size 16. AgSpec also remains effective on benchmarks without a repository or a multi-agent pipeline, showing that its gains generalize to coding agents broadly.

Primary area:
foundation or frontier models, including LLMs
Keywords:
speculative decoding, LLM inference efficiency, retrieval, coding agents, LLM serving

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_jxurOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents AGSpec, a framework for improving retrieval-based speculative decoding in coding-agent pipelines. The key observation is that existing retrieval-based drafters do not fully exploit the structure of agent workloads. Reusable text may reside in previous model calls or repository files rather than only in the current prompt, while suitable draft lengths vary substantially across agent roles and execution stages. AGSpec introduces two main components. First, it maintains separate global, session, and workspace corpora. Workspace artifacts are indexed in both their original form and the form in which an agent is expected to emit them, such as unified-diff format. AGSpec also uses source-aware policies to select among candidate continuations. Second, it controls draft length using an offline-profiled, agent-specific cap together with an online scaling factor updated from verification feedback. The method is evaluated on SWE-bench Verified and TeamBench using three 24–27B models, two retrieval engines, batch sizes of 1 and 16, and several retrieval-based and model-based speculative-decoding baselines. Additional experiments are conducted on LiveCodeBench and Terminal-Bench. On the two main repository-level benchmarks, AGSpec achieves 2.27–4.37× the throughput of autoregressive decoding at batch size 1 and 1.08–4.76× at batch size 16. It improves throughput by 18.0% on average over the fastest prior method in each setting. The ablations indicate that cross-call session history provides most of the retrieval gain, while workspace and emission-form indexing and adaptive draft-length control provide further improvements.

Critical strengths

1.The method is well aligned with the structure of coding-agent workloads.

AGSpec identifies two concrete limitations of existing retrieval-based speculative decoding: incomplete access to reusable context and context-insensitive draft lengths. It addresses both with relatively simple and well-motivated mechanisms. In particular, organizing text into global, session, and workspace corpora and indexing repository artifacts in agent-emission form are well suited to iterative code-editing workflows. This is consequential because the resulting design is applicable to multiple retrieval engines rather than being tied to a particular underlying retrieval data structure.

2.The evaluation is broad and provides useful explanations for the observed improvements.

The paper evaluates three target models, four benchmarks, two retrieval engines, two batch-size regimes, and several strong baselines. The component analyses also help identify the sources of improvement. Retaining session history substantially increases accept length, workspace indexing provides additional gains, and online draft-length adaptation sharply reduces rejected tokens at high batch sizes. The overhead analysis further suggests that AGSpec itself accounts for only a small fraction of decoding-step time.

Critical weaknesses

1.The current evaluation does not establish end-to-end latency improvements for complete coding-agent tasks.

The primary metric is generation throughput, explicitly excluding prefill. It also excludes tool execution, environment interaction, scheduling, and other agent-framework overheads. Coding-agent sessions can involve long input contexts and expensive tool calls, so a large decode-only speedup may translate into a much smaller reduction in user-visible completion time. Since the paper is primarily motivated as an acceleration method for complete agent pipelines, the absence of end-to-end latency measurements limits the practical significance of its central claim.

2.The comparison may be sensitive to asymmetric workload-specific tuning.

AGSpec uses per-model and per-benchmark corpora, offline agent-specific draft caps, and a hardware-derived break-even threshold. In contrast, several baselines use a single fixed draft length or their published/default vLLM configurations, without reported parameter sweeps on the evaluated hardware and workloads. Because the average improvement over the fastest prior method is 18%, reasonable retuning of baseline draft lengths or speculation factors could materially affect the quantitative ranking. The evaluation also does not show how well AGSpec’s global corpus and offline profiles transfer across benchmarks, repository distributions, or agent pipelines.

Questions for authors

1.Clarification that could affect my assessment: Can the authors report end-to-end wall-clock session latency, including prefill and agent-framework overhead, for at least SWE-bench Verified and TeamBench? A breakdown into prefill, decoding, retrieval/proposal, and tool/environment time would help establish how much of the reported generation-throughput gain is visible to users.

2.Clarification that could affect my assessment: Were baseline draft lengths and speculation factors tuned or swept on the same hardware and workloads? Please provide either an oracle-tuned or swept comparison for the strongest baselines, or sensitivity results demonstrating that the reported configurations are close to optimal.

3.Clarification that could affect my assessment: How does AGSpec perform in a cold-start or transfer setting, for example when using a global corpus and draft-length caps constructed from a different benchmark, repository distribution, or agent scaffold? This would clarify whether representative workload-specific trajectories must be collected before deployment.

4 comments

Thank you for the thoughtful questions. We address each point below.

W1 / Q1. End-to-end latency for complete agent tasks is not established. Can you report end-to-end session latency on SWE-bench Verified and TeamBench, broken into prefill, decoding, retrieval, and tool time?

We now measure end-to-end session latency on both benchmarks at batch size 1 (30 sessions). The tables give the mean time per session in seconds, broken into phases. Speedup is end-to-end over autoregressive decoding, with paired session-level 95% bootstrap CIs for AgSpec. TeamBench times exclude benchmark setup and grading, so that table has no test column.

TeamBench

ModelMethodPrefillDecodingRetrievalTool callsTotalSpeedup
DevstralAR181580.071831.00×
DevstralSAM-Decoding18780.33991.85×
DevstralSuffixDecoding18640.45882.06×
DevstralAgSpec (SAM)18530.63752.42× [2.11, 2.74]
DevstralAgSpec (Suffix)18550.43772.38× [2.05, 2.73]
Gemma3AR122910.053081.00×
Gemma3SAM-Decoding141030.2121282.41×
Gemma3SuffixDecoding131150.351332.32×
Gemma3AgSpec (SAM)11820.6131082.86× [2.21, 3.59]
Gemma3AgSpec (Suffix)12970.361152.67× [2.23, 3.25]
Qwen3.6AR182420.042641.00×
Qwen3.6SAM-Decoding171650.331861.42×
Qwen3.6SuffixDecoding171060.241272.07×
Qwen3.6AgSpec (SAM)17890.551112.37× [2.00, 2.68]
Qwen3.6AgSpec (Suffix)17960.351182.24× [1.90, 2.52]

SWE-bench Verified

ModelMethodPrefillDecodingRetrievalTool callsTestsTotalSpeedup
DevstralAR141160.089282481.00×
DevstralSAM-Decoding14760.486282051.21×
DevstralSuffixDecoding14370.389291701.45×
DevstralAgSpec (SAM)15281.593291661.49× [1.37, 1.65]
DevstralAgSpec (Suffix)14301.287281611.54× [1.40, 1.72]
Gemma3AR154020.082185181.00×
Gemma3SAM-Decoding152000.481143111.67×
Gemma3SuffixDecoding151160.381212332.22×
Gemma3AgSpec (SAM)15922.076212072.50× [2.14, 2.94]
Gemma3AgSpec (Suffix)15971.575142022.56× [2.24, 2.93]
Qwen3.6AR144310.065165271.00×
Qwen3.6SAM-Decoding142760.467163741.41×
Qwen3.6SuffixDecoding141600.367162562.06×
Qwen3.6AgSpec (SAM)141312.069162312.29× [1.96, 2.65]
Qwen3.6AgSpec (Suffix)141431.566142382.21× [1.96, 2.49]

Speculative decoding shortens only the decoding phase. Prefill stays the same, and retrieval costs at most 2 s per session. On TeamBench, decoding takes 86–94% of an autoregressive session, so most of the gain reaches the user. On SWE-bench Verified, tool calls and Docker tests take about 80–120 s per session for every method. This time bounds the end-to-end gain. Devstral decodes fastest, so tools dominate its session and its gain is 1.49–1.54×. AgSpec still cuts a Gemma3 session from 518 s to 207 s and a Qwen3.6 session from 527 s to 231 s.

W2 / Q2. The comparison may be sensitive to asymmetric tuning. Were baseline draft lengths and speculation factors tuned on the same hardware and workloads?

Sweep at batch size 1. We swept the draft length K of every chain baseline, and the speculation factor of SuffixDecoding, on TeamBench with Gemma3 at batch size 1. The table reports end-to-end speedup over autoregressive decoding at the best value (30 sessions).

BaselineValues sweptBest valueSpeedup at best value
EAGLE-3K = 3, 5, 7K = 51.08×
RESTK = 4, 10, 16, 32K = 161.62×
PLDK = 4, 10, 16, 32K = 161.96×
FastCoderK = 4, 10, 16, 32K = 162.15×
SAM-DecodingK = 4, 10, 16, 32, 64K = 162.46×
SuffixDecodingspeculation factor 1, 2, 4factor 42.70×
AgSpec (SAM)——2.86×

Tuning raises some baselines by up to 16%, but none reaches AgSpec. The strongest tuned baseline, SuffixDecoding with a speculation factor of 4, stays 6% behind. With 30 sessions, the paired session-level 95% CI of this gap is [0.81, 1.32].

Sweep at batch size 16. We also swept K at batch size 16 with prefix caching on (TeamBench, Gemma3, 64 sessions).

MethodK = 10K = 16K = 32AgSpec on the same engine
SAM-Decoding2.03×1.80×1.31×2.56×
SuffixDecoding2.60×2.70×2.69×2.83×

Long fixed drafts hurt under batching, and no single K suits both engines. On each engine, AgSpec beats the best-tuned baseline.

All main settings. Across all main settings, we also compare the faster AgSpec variant with the faster of SAM-Decoding and SuffixDecoding in each setting. The table reports end-to-end speedup with paired session-level 95% CIs (batch size 1, 30 sessions, default baseline configurations).

BenchmarkDevstralGemma3Qwen3.6
TeamBench1.17× [1.01, 1.41]1.19× [1.08, 1.33]1.14× [1.07, 1.21]
SWE-bench Verified1.06× [1.03, 1.09]1.15× [1.07, 1.25]1.11× [1.02, 1.22]

AgSpec is 6–19% faster than the fastest prior method in every setting, and every CI excludes 1.

Native trees. We also ran native tree configurations of the baselines on TeamBench at batch size 1. None of them closes the gap.

MethodConfigurationDevstralGemma3
EAGLE-3static tree, 60 nodes1.02×1.48×
EAGLE-3dynamic tree0.92×1.06×
FastCodertree, 64 nodes1.30×2.20×
RESTtree, 64 nodes0.79×1.88×
AgSpec (SAM)chain2.42×2.86×

Q3. How does AgSpec perform in a cold-start or transfer setting, with a global corpus and caps built from a different benchmark?

Cold start. Only the global corpus needs prior data. The session and workspace corpora are built online from the first request.

Transfer. To test transfer, we ran AgSpec on TeamBench with offline material from SWE-bench Verified, a benchmark with different repositories and a different agent pipeline. The global corpus comes from SWE-bench Verified traces. The caps come from SWE-bench agents mapped to the closest TeamBench roles. The workspace and session corpora are built online as usual (batch size 1, 30 sessions).

Configuration (Gemma3)Speedup over AR
SAM-Decoding2.41×
SuffixDecoding2.32×
AgSpec, SWE-bench corpus and caps2.49×
AgSpec, TeamBench corpus and caps2.86×

AgSpec beats both baselines without any trajectory from the target workload. Material from the same workload adds a further 15%, which a deployment gains as its own history accumulates.

We will add these results to the revision.

AuthorsOpenReview ↗

Gentle Reminder to Review Our Responses

Dear Reviewer jxur,

We appreciate the important questions you raised regarding the practical effectiveness and fairness of our evaluation. To address them, we have provided (1) end-to-end latency measurements for complete agent tasks, (2) baseline hyperparameter sweeps showing AgSpec's advantage over tuned baselines, and (3) cross-benchmark transfer experiments for the global corpus and draft-length caps.

We would welcome any further feedback on our responses and hope they have satisfactorily addressed your concerns.

Sincerely, The Authors

Thank you for the detailed response and the additional experiments. The end-to-end latency breakdown, the baseline parameter sweeps at both batch sizes, and the cold-start and cross-benchmark transfer results have addressed my main concerns. These results provide stronger evidence for both the practical benefit and robustness of AGSpec. I encourage the authors to include these results and clarifications in the revised paper.

I will maintain my current positive score.

Reviewer_FnYsOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

AgSpec adapts retrieval-based (model-free) speculative decoding to coding-agent workloads. It drafts from three corpora (session, workspace and a global corpus), indexes workspace files in emission form (copies with diff-style space/minus prefixes and an escaped form), prefers global drafts only when their match exceeds the live match by a margin delta, selects continuations by the most frequent next-3-token group, and sets draft length K = clip(round(alpha * m), 1, Kmax), with Kmax profiled offline per agent and alpha adapted online by a quantile-style rule. Implemented in vLLM on top of SAM and Suffix engines and evaluated with Devstral-24B, Gemma3-27B and Qwen3.6-27B on SWE-bench Verified (custom 5-agent pipeline), TeamBench, LiveCodeBench and Terminal-Bench, AgSpec reports up to 4.37x speedup at batch size 1 and 4.76x at batch size 16, and about 18% average improvement over the best prior method.

Critical strengths
  • Practical relevance. Agent traces are highly repetitive, and exploiting that for lossless speedups is valuable for serving cost.
  • Thoughtful engineering. Emission-form indexing and the live-vs-global preference rule are sensible observations about how agents rewrite code.
  • Broad evaluation. Three models, four benchmarks, two batch sizes and two base engines.
  • Adaptive draft length. The online alpha update with an offline Kmax is a simple, deployable mechanism.
Critical weaknesses
  • Incremental algorithmic novelty. The components adapt existing retrieval-based speculation to agent workflows. This limits the conceptual contribution, but does not by itself negate a well-supported systems improvement. The acceptance case rests on measured decoding gains and implementation value.
  • Scope of acceleration evidence. The reported generation-throughput measurements exclude prefill and do not establish end-to-end agent-task speedup. The paper should distinguish decoding acceleration from full-workflow latency and report task-success checks alongside timing where feasible.
  • Uneven gains. Gains vary widely; Qwen on TeamBench at batch size 16 reaches only 1.08x, and some settings fall below 1x. The headline should reflect typical rather than best-case speedups.
  • Cross-repository generalization remains open. The global corpus uses disjoint tasks; this is not evidence of test-task leakage. However, disjoint tasks within a benchmark do not establish that gains transfer to unseen repositories or workload distributions. A cross-repository or cross-benchmark corpus experiment would address that deployment question.
  • Ablation evidence. The corpus ablation is a trace-driven simulation and the draft-length study uses only Devstral.
Questions for authors
  1. What are end-to-end wall-clock improvements per agent task, including prefill and tool execution?
  2. How do speedups change when the global corpus is built from different repositories or benchmarks than the test set?
  3. What are the memory and indexing costs of the three corpora at batch size 16?
  4. Can you report the corpus ablation with real runs, and the draft-length study for Gemma and Qwen?
  5. Beyond the included EAGLE-3 comparison, can retrieval and model-based drafting be combined, and what is the incremental benefit and overhead?
  6. Why does Qwen on TeamBench gain so little, and what explains the settings below 1x?
4 comments

Thank you for the constructive assessment and acknowledgment of our well-supported systems improvement. We address the remaining concerns below.

W2 / Q1. Generation throughput excludes prefill and does not establish end-to-end speedup. What are the end-to-end wall-clock improvements per task, including prefill and tool execution, and is task success preserved?

End-to-end time. We now measure end-to-end session time at batch size 1 (30 sessions). The table gives the mean time per session in seconds, broken into prefill, decoding with retrieval, and tool calls with tests.

BenchmarkModelMethodPrefillDecoding and retrievalTools and testsTotalSpeedup
TeamBenchDevstralAR1815871831.00×
TeamBenchDevstralAgSpec (SAM)18543752.42×
TeamBenchGemma3AR1229153081.00×
TeamBenchGemma3AgSpec (SAM)1183131082.86×
TeamBenchQwen3.6AR1824242641.00×
TeamBenchQwen3.6AgSpec (SAM)178951112.37×
SWE-bench VerifiedDevstralAR141161172481.00×
SWE-bench VerifiedDevstralAgSpec (SAM)15301221661.49×
SWE-bench VerifiedGemma3AR154021005181.00×
SWE-bench VerifiedGemma3AgSpec (SAM)1594982072.50×
SWE-bench VerifiedQwen3.6AR14431825271.00×
SWE-bench VerifiedQwen3.6AgSpec (SAM)14133842312.29×

Speculative decoding shortens only decoding. Prefill and tool time do not shrink, so the end-to-end gain is smaller than the gain in generation throughput. On TeamBench, decoding dominates each session, so most of the gain reaches the user. On SWE-bench Verified, tool calls and Docker tests take 16–47% of an autoregressive session. The end-to-end gains there remain substantial at 1.49–2.50×. We will report end-to-end latency next to generation throughput in the main results.

Task success. AgSpec is lossless under greedy decoding, so it does not change what the model generates. Small differences can still appear because floating-point nondeterminism makes long trajectories diverge. On SWE-bench Verified, every method resolves the same number of instances for each model. On TeamBench, we compare the mean partial score per session with autoregressive decoding (30 sessions, paired 95% bootstrap CI).

ModelARAgSpec (SAM)Difference [95% CI]
Devstral0.4080.405−0.003 [−0.008, 0.000]
Gemma30.4180.422+0.004 [−0.060, 0.070]
Qwen3.60.4760.441−0.034 [−0.081, 0.000]

Every CI includes zero, so AgSpec shows no significant change in task success.

W3 / Q6. Gains are uneven, such as 1.08× for Qwen on TeamBench at batch size 16. Why does Qwen gain so little, and what explains the settings below 1×?

Typical gains. We agree that the headline should reflect typical gains, and we will report the geometric mean over all settings. At batch size 1, the end-to-end speedup of AgSpec (SAM) ranges from 1.49× to 2.86× across six settings, with a geometric mean of 2.28×.

The gain over prior methods is also consistent. Against the fastest prior method in each setting, with default baseline configurations, AgSpec is 6–19% faster (geometric mean 1.14×), and every paired session-level 95% CI excludes 1.

Low gains at batch size 16. The low batch-16 gains on TeamBench come largely from prefill. With prefix caching off, every call prefills the whole conversation again, and 50–62% of batch-16 generation time is spent in steps that mix prefill. Speculative decoding speeds up these steps little. With prefix caching on, the gains grow substantially (TeamBench, Gemma3, 64 sessions).

TeamBench, batch size 16SAM-DecodingAgSpec (SAM)SuffixDecodingAgSpec (Suffix)
Prefix caching off1.61×1.70×1.76×1.77×
Prefix caching on2.03×2.56×2.60×2.83×

The draft-length caps also matter at batch size 16. The caps in the paper were fit on all verification steps, including those that mixed prefill. We refit them on decode-only steps, which gives differentiated caps on both benchmarks.

Batch size 16Caps per roleSAM-DecodingAgSpec (SAM)SuffixDecodingAgSpec (Suffix)
TeamBench, Gemma34 / 24 / 141.61×1.70×1.76×1.77×
SWE-bench Verified, Gemma3 †24 / 24 / 24 / 4 / 41.78×1.96×2.13×2.35×

Caps are listed as planner / executor / verifier on TeamBench and localizer / coder / executor / feedback / summarizer on SWE-bench Verified. † is generation speedup; other rows are end-to-end speedup.

Qwen3.6 on TeamBench. Two factors limit Qwen3.6, and neither applies to Gemma3 or Devstral.

Memory pressure from the hybrid architecture. Qwen3.6 is a hybrid model. 48 of its 64 layers use Gated DeltaNet, and 16 use full attention. For softmax-attention models, extra draft tokens add only a few KV entries. For Qwen3.6, vLLM keeps a separate recurrent-state copy for each of the K+1 draft positions of every request, so per-request memory grows with the draft budget. The measured growth from K=10 to K=24 (2.3×) matches (24+1)/(10+1).

TeamBench, batch size 16Peak cache use, K=10Peak cache use, K=24Scheduler queue time, K=24
Gemma317%16%14 s
Devstral16.5%14.5%11 s
Qwen3.647%100%649–884 s

At K=24 the cache saturates, and new calls wait in the queue. Throughput stays the same, but per-session latency rises. Prefix caching also helps Qwen3.6 less, since its recurrent state can be cached only at block boundaries (hit rate 72% vs. 86% for Gemma3).

Capping the draft budget at 12 removes the saturation. We choose 12 as the largest budget whose recurrent-state copies fit in the cache, following the (K+1) scaling above. Peak cache use falls from 100% to 56%, queue time falls to 10–28 s, and AgSpec overtakes both same-engine baselines. Without prefix caching, AgSpec rises from 1.28× to 1.35× (SAM) and from 1.39× to 1.57× (Suffix).

TeamBench, Qwen3.6, batch size 16SAM-DecodingAgSpec (SAM), cap 12SuffixDecodingAgSpec (Suffix), cap 12
Prefix caching off (49 sessions)1.24×1.35×1.49×1.57×
Prefix caching on (48 sessions)0.99×1.20×1.50×1.55×

Short, novel outputs from the executor and verifier. Reuse-based drafting predicts text that already appeared in the session. The Qwen3.6 executor writes short tool calls that are mostly new text, so there is little to reuse.

Executor agentGemma3DevstralQwen3.6
Mean output length (words)31829089
6-grams seen earlier in the session48.5%44.3%24.1%
Accepted tokens per step (AgSpec-SAM)2.443.361.29
Steps with no accepted draft token47%59%79%

We measure reuse on autoregressive outputs, so it reflects the model's behavior rather than speculation. The verifier shows the same pattern (1.23 accepted tokens per step vs. 2.34 and 3.58). The Qwen3.6 planner is the opposite case. Its short, repetitive calls make it the model's best-predicted agent (3.14 tokens per step). In Qwen3.6, however, the hard-to-predict agents dominate generation time. We will add this analysis to the revision.

W4 / Q2. Cross-repository generalization remains open. How do speedups change when the global corpus comes from different repositories or benchmarks?

We ran AgSpec on TeamBench with offline material from SWE-bench Verified. The two benchmarks differ in repositories, tasks, and agent pipelines. We replace both the global corpus and the caps. The global corpus comes from SWE-bench Verified traces, and the caps come from SWE-bench agents mapped to the closest TeamBench roles. The workspace and session corpora are built online as before (batch size 1, 30 sessions).

Configuration (Gemma3)Speedup over AR
SAM-Decoding2.41×
SuffixDecoding2.32×
AgSpec, SWE-bench corpus and caps2.49×
AgSpec, TeamBench corpus and caps2.86×

With transferred material, AgSpec still beats both baselines. Material from the same workload adds a further 15%. We will add cross-benchmark results for the other models in the revision.

W5 / Q4. The corpus ablation is trace-driven, and the draft-length study uses only Devstral. Can you report the corpus ablation with real runs and the draft-length study on Gemma and Qwen?

Corpus ablation with real runs. We replace the trace-driven corpus ablation with real end-to-end runs on TeamBench at batch size 1. The table reports speedup over autoregressive decoding (30 sessions).

CorporaDevstralGemma3Qwen3.6
Global + per-request (SAM-Decoding)1.85×2.41×1.42×
Session only1.98×2.35×2.20×
Global + workspace + session (AgSpec)2.42×2.86×2.37×

The session corpus alone beats SAM-Decoding's per-request corpus on Devstral and Qwen3.6. Adding the global and workspace corpora gives the best result on all three models. On Gemma3, we also ran each source alone.

Corpus (Gemma3)Speedup over AR
Global only1.49×
Workspace only1.46×
Session only2.35×
All three (AgSpec)2.86×

No single source comes close to the full set.

Draft-length study on other models. We also repeat the draft-length study on Gemma3 (SWE-bench Verified, batch size 16).

Draft-length controlGeneration speedup over AR
Shared cap of 32, no online scale1.46×
Shared cap of 24, no online scale1.58×
Shared cap of 8, no online scale1.90×
Role-specific caps and online scale (AgSpec)1.96×

As on Devstral, AgSpec's draft-length control is the fastest. Long shared drafts hurt most under batching.

On Qwen3.6, we ran the same comparison at batch size 1 (SWE-bench Verified, 30 instances).

Draft-length controlGeneration speedup over AR
Shared cap of 32, no online scale2.82×
Shared cap of 8, no online scale2.54×
Role-specific caps and online scale (AgSpec)2.89×

Q3. What are the memory and indexing costs of the three corpora at batch size 16?

These costs are small at batch size 16, and the reported throughput already includes them (Appendix G.3). The corpora live in host memory and use no GPU memory.

  • Global corpus. One copy is shared by all sessions. It takes about 3.2–3.6 GB of host memory, the same as SAM-Decoding's static corpus. Loading it at engine start takes 10–17 s.
  • Session and workspace corpora. At batch size 16 with 64 tasks, they add 1.0 GB of host memory over SAM-Decoding (TeamBench, Gemma3). They are released when a session ends.
  • Indexing. Building the per-instance corpora takes 0.67 s per SWE-bench Verified instance, or 1.6% of the run time (Gemma3, batch size 16).

Q5. Can retrieval and model-based drafting be combined, and what are the benefit and overhead?

Yes. The two are orthogonal, and prior work already combines them. SAM-Decoding reports variants that add a model-based drafter such as EAGLE-2. These improve over EAGLE-2 alone by 3.28–11.13% [1]. SuffixDecoding also reports a hybrid that falls back to EAGLE-3. Its authors note that SuffixDecoding alone is best for highly repetitive agentic tasks [2]. MetaSD and HedgeSpec go further and select among several drafters online [3, 4].

AgSpec fits into such frameworks as the retrieval drafter. Its corpora and draft-length control apply unchanged. The main overhead of a combination comes from the model drafter, which must keep its KV cache up to date and occupies GPU memory. We will add this discussion to Section 6.

References

[1] Hu et al., SAM Decoding: Speculative Decoding via Suffix Automaton, ACL 2025.

[2] Oliaro et al., SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications, NeurIPS 2025.

[3] Kim et al., Multi-Drafter Speculative Decoding with Alignment Feedback, ACL 2026 Findings.

[4] Liu et al., Not-a-Bandit: Provably No-Regret Drafter Selection in Speculative Decoding for LLMs, ICLR 2026.

AuthorsOpenReview ↗

Gentle Reminder to Review Our Responses

Dear Reviewer FnYs,

Thank you for your constructive assessment and insightful suggestions. Your feedback motivated us to further investigate (1) end-to-end latency and task success, (2) performance variations, particularly Qwen3.6's model-specific characteristics, (3) cross-benchmark generalization, and (4) online corpus ablations and draft-length control.

We would greatly appreciate your thoughts on whether these additional experiments and analyses sufficiently resolve the issues you raised.

Sincerely, The Authors

Reviewer_vAMiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

AgSpec extends retrieval-based speculative decoding with persistent session context, workspace artifacts indexed in their emitted form, and agent-dependent draft control. It implements these additions over SAM-Decoding and SuffixDecoding.

Critical strengths

Improvements across two engines support the practical value of the integration. The corpus ablations distinguish session persistence, workspace retrieval, and emission-form indexing. Additional evaluations cover non-repository and single-agent workloads. Figure 7(c) directly isolates online scaling, reporting a 66% throughput improvement over fixed K = 32 at batch size 16. Appendix G.3 also quantifies proposer overhead.

Critical weaknesses
  1. The headline comparison does not establish superiority over native competing implementations. Appendix A discloses chain approximations for REST/FastCoder and a depth-3 EAGLE-3 chain. These choices limit comparison with native tree configurations, although the within-engine improvements remain informative.
  2. Deployment generality and uncertainty remain insufficiently characterized. Experiments use eager execution with prefix caching disabled, and throughput excludes prefill. The main SWE-bench and TeamBench evaluations each contain 30 tasks without session-level confidence intervals for headline throughput gains. Appendix significance tests address selected ablations rather than these main comparisons.
  3. The incremental benefit of role-specific caps is not isolated clearly. Table 3 reports differentiated TeamBench caps but uniform ceiling-valued SWE-bench caps at batch size 16; the draft-control ablations use SWE-bench. The demonstrated online-scale benefit should therefore be distinguished from the additional value of role-specific offline caps.
Questions for authors

What do role-specific caps add over a shared cap with identical online scaling on TeamBench? Do the gains persist with native tree baselines and prefix caching, and how much end-to-end latency is saved? Can the authors report paired session-level uncertainty for the main throughput gains?

SuffixDecoding, FastCoder, and SmartSpec already establish several ingredients and are acknowledged by the authors [1–3]. Stronger comparative serving evidence is needed to establish the significance and deployment relevance of the incremental contribution.

References

[1] Oliaro et al. SuffixDecoding: Extreme Speculative Decoding for Emerging AI Applications. NeurIPS, 2025.

[2] Zhao et al. FastCoder: Accelerating Repository-level Code Generation via Efficient Retrieval and Verification. ASE, 2025.

[3] Liu et al. Optimizing Speculative Decoding for Serving Large Language Models Using Goodput. 2024.

7 comments

Thank you for the valuable feedback. We address each concern below.

W1 / Q1. The headline comparison does not establish superiority over native tree implementations of REST, FastCoder, and EAGLE-3. Do the gains persist with native tree baselines?

Native tree baselines. We add native tree baselines on TeamBench at batch size 1. vLLM lacks tree acceptance, so we implemented greedy tree acceptance. On Devstral, every tree configuration reproduces the autoregressive output on all 8 test prompts, while vLLM's original tree path matches on only 1 of 8. On Gemma3, floating-point ties make even chains diverge from autoregressive output, so we instead check the tree attention kernel, which matches reference attention within 3e-4. We exclude Qwen3.6, a hybrid Gated DeltaNet model, since tree-based speculative decoding for such models is still an open proposal in vLLM [1]. The table reports end-to-end speedup over autoregressive decoding and acceptance length (AL) over 30 sessions.

MethodDraft shapeDevstral speedupDevstral ALGemma3 speedupGemma3 AL
EAGLE-3chain, length 101.11×2.201.03×1.29
EAGLE-3static tree, 60 nodes1.02×2.861.48×1.68
EAGLE-3dynamic tree0.92×3.081.06×1.68
RESTtree, 64 nodes0.79×1.351.88×1.71
FastCoderchain, length 101.42×1.992.23×2.06
FastCodertree, 64 nodes1.30×2.242.20×2.39
SuffixDecodingchain2.06×2.992.32×2.84
AgSpec (SAM)chain2.42×4.142.86×3.68
AgSpec (Suffix)chain2.38×3.792.67×3.38

Trees raise AL over chains, but the additional verification cost does not translate this gain into speedup. No tree baseline comes close to AgSpec, which also has the highest AL. The gap holds on SWE-bench Verified (best tree 1.52× vs. 2.50×) and widens at batch size 16, where EAGLE-3's static tree drops to 0.45× on Gemma3. We will add the full results to the revision.

Chains in the main evaluation. AgSpec targets speculative decoding inside a serving system. We evaluated all methods on vLLM, a widely used serving engine. We use chains in the main evaluation because vLLM's tree path uses a separate Triton kernel, lacks tree acceptance, and was removed as "not fully supported" [2]. Serving engines generally favor chains. The EAGLE-3 paper reports its serving results with chains of length 3 in SGLang and 2 in vLLM [3]. Some performance features also do not support trees. SGLang supports its overlap scheduler only for chains [4]. TensorRT-LLM defaults to chains and excludes sliding-window models such as Gemma3 from tree mode [5]. Within each model, all methods in the main evaluation share the same engine and kernel (FlashAttention-2 for Devstral and Qwen3.6, Triton for Gemma3).

References

[1] vLLM RFC #54080, "TreeWY: Tree Speculative Decoding for hybrid (GDN) models": https://github.com/vllm-project/vllm/issues/54080

[2] vLLM PR #42121, "Remove tree attention": https://github.com/vllm-project/vllm/pull/42121

[3] Li et al., EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test, NeurIPS 2025.

[4] SGLang documentation, Speculative Decoding: https://docs.sglang.io/advanced_features/speculative_decoding.html

[5] TensorRT-LLM documentation, Speculative Decoding: https://nvidia.github.io/TensorRT-LLM/features/speculative-decoding.html

W2 / Q1. Deployment generality and uncertainty are insufficiently characterized (eager execution, prefix caching off, prefill excluded, no session-level CIs). Do the gains persist with prefix caching, how much end-to-end latency is saved, and what is the paired session-level uncertainty?

Uncertainty and end-to-end latency. All numbers below are end-to-end session speedups. They include prefill and tool execution. The table gives paired session-level 95% bootstrap CIs at batch size 1 (30 sessions, 10,000 resamples). Each AgSpec variant is compared with autoregressive decoding. The last column compares the faster AgSpec variant with the fastest prior method in each setting.

BenchmarkModelAgSpec (SAM) vs ARAgSpec (Suffix) vs ARBest AgSpec vs best prior
TeamBenchDevstral2.42× [2.11, 2.74]2.38× [2.05, 2.73]1.17× [1.01, 1.41]
TeamBenchGemma32.86× [2.21, 3.59]2.67× [2.23, 3.25]1.19× [1.08, 1.33]
TeamBenchQwen3.62.37× [2.00, 2.68]2.24× [1.90, 2.52]1.14× [1.07, 1.21]
SWE-bench VerifiedDevstral1.49× [1.37, 1.65]1.54× [1.40, 1.72]1.06× [1.03, 1.09]
SWE-bench VerifiedGemma32.50× [2.14, 2.94]2.56× [2.24, 2.93]1.15× [1.07, 1.25]
SWE-bench VerifiedQwen3.62.29× [1.96, 2.65]2.21× [1.96, 2.49]1.11× [1.02, 1.22]

The gains are significant at the session level. Every CI over autoregressive decoding has a lower bound of at least 1.37×, so the speedup holds across sessions rather than coming from a few long ones. Against the fastest prior method in each setting, the faster AgSpec variant is 6–19% faster, and every CI excludes 1 against default baseline configurations. In absolute terms, AgSpec (SAM) shortens a mean session by 108–200 s on TeamBench and 82–311 s on SWE-bench Verified.

Prefix caching. We re-ran TeamBench on Gemma3 with prefix caching on for all methods, at batch size 1 (30 sessions) and batch size 16 (64 sessions).

Batch sizePrefix cachingSuffixDecodingAgSpec (SAM)AgSpec (Suffix)
1Off2.32×2.86×2.67×
1On3.17×3.64×3.19×
16Off1.76×1.70×1.77×
16On2.60×2.56×2.83×

Prefix caching works in AgSpec's favor. It cuts the prefill time of autoregressive decoding by more than half (6.1 to 2.7 minutes over 30 sessions at batch size 1). Speculative decoding does not change prefill, so decoding takes an even larger share of each session. At batch size 1, AgSpec's speedup rises from 2.86× to 3.64×. At batch size 16, the best AgSpec variant rises from 1.77× to 2.83× and leads SuffixDecoding by 9%. AgSpec (SAM) also beats SAM-Decoding on its own engine (2.56× vs. 2.03×).

Eager execution. We re-ran TeamBench on Gemma3 with CUDA graphs enabled for all methods, including autoregressive decoding. Each speedup is over autoregressive decoding in the same mode. The table uses 23 sessions at batch size 1 and 51 at batch size 16, where prefix caching is on.

Batch sizeExecutionSAM-DecodingAgSpec (SAM)SuffixDecodingAgSpec (Suffix)
1Eager2.04×2.30×1.93×2.30×
1CUDA graph1.43×1.75×1.55×1.57×
16Eager2.16×2.54×2.66×2.90×
16CUDA graph1.06×1.19×1.48×1.51×

CUDA graphs make autoregressive decoding 2.6× faster at batch size 1, so every speculative method gains less in relative terms. AgSpec still beats the baseline on its own engine in every setting.

W3 / Q1. The benefit of role-specific caps is not isolated, since the SWE-bench caps at batch size 16 are uniform. What do role-specific caps add over a shared cap with identical online scaling on TeamBench?

Uniform SWE-bench caps. The batch-16 caps in the paper were fit on all verification steps, including those that mixed prefill. We also fit them on decode-only steps. On SWE-bench Verified (Gemma3), this gives differentiated caps: the localizer, coder, and executor get 24, and the feedback and summarizer agents get 4. We will add these caps to Table 3 in the revision.

Role-specific vs. shared caps. To separate the value of role-specific caps from the online scale, we compare them with one shared cap for all agents on TeamBench at batch size 16 (Gemma3, 64 sessions, prefix caching on). All AgSpec rows use the same online scale.

Draft-length controlSpeedup over ARMedian session speedup
SAM-Decoding, fixed K=102.03×1.96×
Shared cap of 24 (engine K)2.41×2.37×
Shared cap of 14 (mean of role caps)2.55×2.39×
Role-specific caps (AgSpec)2.56×2.52×

Every AgSpec configuration beats SAM-Decoding. Role-specific caps match the best shared cap in total time and lead in the median session. That shared cap is the mean of the role caps, so choosing it already needs the per-agent profile. A shared cap at the engine limit is 6% slower.

Q2. SuffixDecoding, FastCoder, and SmartSpec already establish several ingredients. Stronger comparative serving evidence is needed.

Prior work drafts from a single request. AgSpec drafts from the whole agent session.

MethodReuse scopeIndexed formDraft length
SuffixDecodingRequest, past outputsRaw tokensFixed factor
FastCoderSingle call, repositoryRaw filesFixed budget
SmartSpecNo retrieval–One per batch
AgSpecAgent session, workspaceAs agents emit itPer agent, online

In a coding-agent session, much of what an agent writes already exists in earlier calls, tool outputs, and workspace files, often in a different form. AgSpec drafts from this session-wide context, in the form agents re-emit it, with a draft length set per agent from live feedback. As a policy layer above the engine, it improves both SAM-Decoding and SuffixDecoding.

This response adds the following serving evidence.

  • Gains over the best prior method. With default baseline configurations, AgSpec beats the faster of SAM-Decoding and SuffixDecoding in all six settings, and every paired CI excludes 1 (W2).
  • Native tree baselines. The best tree baseline reaches 1.30× on Devstral and 2.20× on Gemma3, against 2.42× and 2.86× for AgSpec (W1).
  • Prefix caching and CUDA graphs. AgSpec stays ahead of the same-engine baseline with prefix caching on and under CUDA graphs (W2).
  • End-to-end latency. All of these gains include prefill and tool execution (W2).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.