- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a latency framework to analyze the e2e latency in speculative execution of agent system. It proposes LEAP, a method that can both online and offline trains the draft models to improve the acceptance rate. LEAP shows promising results on different models and benchmarks.
Critical strengths
- This paper propose a latency framework. It unifies lots of factors (e.g. draft acceptance rates, drafting costs, verification latency, and tool execution overhead) into a empirical formula to analyze end-to-end speedup in speculative execution of agent planning system.
- The paper introduces LEAP. It online or offline trains the draft model to improve the acceptance rate. Improving the acceptance rate is always an important direction in speculative execution field, no matter in LLM decoding or in agent system.
- The evaluation is comprehensive. It includes four benchmarks and different draft / target models. I especially appreciate the experiments and analysis about the lossless execution part. It helps distinguish trajectory divergence caused by LLM serving with the verification design.
Critical weaknesses
- Parallel verification is a good proposal and is a critical part in LLM's speculative decoding. However, prior work such as ISP overlap the target verification and draft proposal which can hide some of the draft latency [1]. The authors acknowledge this in their limitations and add some analysis, and I guess the authors choose current design probably because of the GPU contention or the throughput of target models? But I still think authors should elaborate and state clearly the reason of choosing current synchronous design in the main paper.
- Online training would make the framework more practical, but it will introduce some costs. As this is the core contribution part, it would be really valuable to report the overall wall-clock speedup of the integrated online learning pipeline (starting from the cold start), including training and adapter updates. I'll consider raising my evaluation if this can be fully addressed during rebuttal period.
[1] Wenyue Hua et al. Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User Interface. ICLR, 2025.
Questions for authors
See weakness.