acceptodds
Under review as a conference paper at ICLR 2027

LEAP: Learning Efficient Action Proposals For LLM Agents

Abstract

LLM agents are known to be slow in rollouts. An agent completes a task one step at a time. At each step, it reasons and then chooses an action to execute. The next step and action cannot start until the previous one has finished. Speculative decoding accelerates the rollouts at the reason phase by drafting and verifying the inference tokens. Recent works have also started to apply similar ideas at the action phase. These works use off-the-shelf models, usually large, to draft action proposals for target model to verify. Large drafters match the target more often but take longer to propose, while small off-the-shelf models are fast but rarely make the same decision as the target. We ask a more general question: what determines the end-to-end speedup of action speculation? To answer it, we develop a latency framework for the speculative round. The framework compares what a round gains with what it costs. The gain depends on how well the drafter predicts the target and on how many steps the task can take before it ends. The cost comes from drafting, from waiting for target verification and from executing tools. Guided by the framework, we introduce LEAP (Learning Efficient Action Proposals) which keeps the drafter small and makes it accurate by training it on the target actions sequences. With a small 0.6B model, LEAP agrees with the target on most decisions and makes agents up to 60% faster in end-to-end wall clock time, with no systematic change in task success. Across various datasets, target models and draft models, the framework accounts for most of the measured speedups. We also show the draft model can be online trained with no prior trace collection and match the performance of offline training, making LEAP practical to deploy in the real world.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agents, Speculation, Acceleration, Draft, Lossless

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_epVMOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a latency framework to analyze the e2e latency in speculative execution of agent system. It proposes LEAP, a method that can both online and offline trains the draft models to improve the acceptance rate. LEAP shows promising results on different models and benchmarks.

Critical strengths
  1. This paper propose a latency framework. It unifies lots of factors (e.g. draft acceptance rates, drafting costs, verification latency, and tool execution overhead) into a empirical formula to analyze end-to-end speedup in speculative execution of agent planning system.
  2. The paper introduces LEAP. It online or offline trains the draft model to improve the acceptance rate. Improving the acceptance rate is always an important direction in speculative execution field, no matter in LLM decoding or in agent system.
  3. The evaluation is comprehensive. It includes four benchmarks and different draft / target models. I especially appreciate the experiments and analysis about the lossless execution part. It helps distinguish trajectory divergence caused by LLM serving with the verification design.
Critical weaknesses
  1. Parallel verification is a good proposal and is a critical part in LLM's speculative decoding. However, prior work such as ISP overlap the target verification and draft proposal which can hide some of the draft latency [1]. The authors acknowledge this in their limitations and add some analysis, and I guess the authors choose current design probably because of the GPU contention or the throughput of target models? But I still think authors should elaborate and state clearly the reason of choosing current synchronous design in the main paper.
  2. Online training would make the framework more practical, but it will introduce some costs. As this is the core contribution part, it would be really valuable to report the overall wall-clock speedup of the integrated online learning pipeline (starting from the cold start), including training and adapter updates. I'll consider raising my evaluation if this can be fully addressed during rebuttal period.

[1] Wenyue Hua et al. Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interface. ICLR, 2025.

Questions for authors

See weakness.

Reviewer_d5fnOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces LEAP, which accelerates LLM agents by training a small draft model to predict a frozen target’s canonical actions without reproducing its reasoning. The system proposes action sequences, verifies the resulting contexts concurrently, and commits the matching prefix together with a target-generated correction or bonus action when applicable. The paper develops a latency model that accounts for acceptance, task boundaries, drafting, verification, and tool execution. Across four benchmarks and two target families, it reports 1.10–1.63× whole-task inference speedups with a 0.6B draft on a single shared GPU, and also evaluates online draft updates.

Critical strengths

1 The core idea has direct experimental support. Training improves the 0.6B draft’s agreement with the Qwen3 target from roughly 2–25% to 64–75%. The trained-versus-untrained controls show that this improvement translates into meaningful end-to-end speedups within the same execution loop.

2 The ablations provide useful insight into when speculation pays off. The draft-size and tool-delay studies show why higher acceptance does not necessarily mean lower latency, and identify settings where additional drafting or tool costs largely eliminate the benefit.

Critical weaknesses

1 End-to-end comparisons with representative prior methods are insufficient. The untrained-draft and reasoning-disabled-target controls are useful for isolating the value of training a small draft. However, they compare drafter choices within the paper’s execution loop rather than complete existing systems. One comparison with a compatible, representative baseline, retaining its key execution mechanisms under matched target and hardware settings, would substantially strengthen the paper.

2 The inference memory–latency trade-off is not sufficiently quantified. Appendix I reports server memory allocations and enables prefix caching, but does not systematically compare actual peak GPU memory and active KV-cache usage against target-only execution. The small draft’s parameter count does not capture the full overhead, since LEAP also verifies up to k + 1 target contexts. This matters in the reported setup: Gemma uses FP8 because BF16 leaves insufficient KV-cache capacity for concurrent verification alongside the draft. This does not itself make the comparison unfair, but it highlights the need to quantify the resource cost behind the single-GPU speedup. A compact study of memory usage and latency across a few context lengths and speculation depths, with target precision and total memory budget fixed, would clarify the deployment limits.

Questions for authors

1 Can authors evaluate plan correctness rather than only normal termination? Appendix M reports termination counts for OpenAGI and TaskBench, not correctness of the generated plans.

2 How much of the learned agreement is target-specific rather than workload-specific? The gold-label comparison changes both the labels and the training contexts. Could you cross-evaluate the adapters already trained for the two targets on the same benchmark, holding evaluation contexts fixed for each target?

Reviewer_TJ5GOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

LEAP accelerates LLM agents by fine-tuning a small drafter on a frozen target model’s actions. The drafter proposes multiple actions, which the target verifies concurrently before committing the accepted prefix. The paper develops a latency model connecting speedup to action acceptance, drafting cost, verification latency, and tool execution. Experiments cover four benchmarks, two target models, and three drafter sizes, reporting end-to-end speedups of 1.10–1.63× with the 0.6B drafter.

Critical strengths

The paper provides a systematic evaluation across four benchmarks, two target models, and three drafter sizes. Beyond measuring end-to-end latency, it examines training choices, speculation depth, and tool latency. This breadth helps establish when action speculation provides practical benefits and when its overhead limits the gains.

Critical weaknesses
  1. Limited methodological novelty. The target–drafter framework follows existing work such as Speculative Actions, while drafter training relies on standard SFT. Moreover, the answer to the paper’s central question—what determines end-to-end speedup—is relatively intuitive: a drafter should be fast and closely aligned with the target, which the paper achieves by fine-tuning a smaller model. Although the empirical study is useful, the algorithmic contribution beyond these existing components remains limited.
  2. The main contribution - latency model might need to think more. Some evaluations use acceptance rates or drafting costs measured during speculative execution, which helps explain observed speedups but does not fully establish prediction before execution. Even the offline-agreement evaluation uses runtime drafting costs for some configurations (Table 12). Since prediction errors can exceed the differences between configurations, it remains unclear whether the framework can reliably select the best drafter size or speculation depth.
Questions for authors
  1. Could the authors provide a concrete speedup calculation explaining what does beta*t_gen means and when the tau->0 approximation is justified? Table 10 is helpful, but a worked example in the main text would improve clarity.
  2. Train the drafter on target action will definitely have a higher offline agreement than train on the gold label, because that's the evaluation target. But does that correct? If we use different target model, how well does a trained drafter transfer to a different target model, and is retraining required?
  3. For Figure 3, any insight on why the performance for 4B drafter drop?
  4. One of the interesting variant is k (the number of prediction ahead), why choose k=3 for planning and k=2 for tool benchmarks? Can the latency model guide the choice of kk and estimate the maximum achievable speedup?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.