- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
This paper presents EdgeWise, a framework in which SLMs handles most steps of a long-horizon agentic task and delegates only the hardest steps to LLMs. This reduces inference cost while preserving task performance. The paper groups agent actions into six types and analyzes how each type relates to task success. It finds that writing and editing actions benefit most from LLM delegation. Then the paper proposes two delegation methods. One is an inference-time approach that uses an external judge. The other is a trained self-delegation approach(SFT+RL) whose token-level credit assignment rewards delegating at useful steps, rather than simply minimizing LLM calls. Lastly, experiments on Terminal-Bench 2.1 and SWE-ReBench show lower cloud API costs with competitive performance.
Critical strengths
1.The paper studies how to allocate stronger models across different steps of long-horizon agent execution. The analysis of writing/editing operations also provides useful insights into where cloud-model delegation may be most beneficial. 2.The paper combines inference-time delegation using an external judge with a learned self-delegation mechanism. The token-level credit-assignment design is particularly interesting because it aims to optimize when to delegate rather than merely reducing the overall number of cloud-model calls.
Critical weaknesses
1.The definition of the token-level delegation advantage appears potentially inconsistent between the main text and Appendix H.1. In particular, the main text suggests that no additional advantage is assigned when local execution is sufficient, whereas Equation (3) appears to assign a positive advantage when the agent and evaluator agree on local execution. 2.The evaluation focuses primarily on coding tasks, limiting generalizability to broader long-horizon agents on edge devices. Moreover, the cost analysis excludes local inference costs, leaving the overall deployment efficiency unclear.
Questions for authors
1.Could the authors clarify the intended definition of Equation (3)? 2.Could the authors compare total cost (local inference + cloud API) with an LLM-only baseline under the same settings, together with success rate, number of delegations, and end-to-end runtime? How much would the reported savings change if local inference cost or tokens per delegated call increased? 3.Could the authors clarify the respective roles of GPT-5.5 and Kimi-K3 in generating SFT trajectories and describe the relevant training details, including hyperparameters, data construction, and data splits? These details are important for independently reproducing the self-delegation results.
3 comments
Thank you for the detailed and helpful responses. The authors have adequately addressed my main concerns, particularly regarding the clarification of the step-level advantage definition, the additional runtime and delegation statistics, and the SFT training details. These clarifications improve the transparency and reproducibility of the work. While some practical deployment considerations remain open, I believe the responses sufficiently resolve my major concerns.
Dear Reviewer iatk,
Thank you for your positive feedback and for acknowledging our efforts to address your concerns. We are glad that our responses have adequately resolved your major concerns about local inference cost and training settings. Given that your main concerns have been addressed, we would sincerely appreciate it if you could consider updating your rating accordingly.
Thank you again for your time and constructive feedback.
Reply(1/1)
We thank reviewer's valuable question. We are sorry to the confuse of the equation 3, in the text before the equation 3, we mentioned that 'When both the SLM and the critic determine that the current step should be delegated to the LLM, we assign an advantage of +1'. This statement is same to the context in Section3.2.2 and the equation has some inaccurate, we have fixed it in the revision, you can see through openreview rebuttal revision.
Thank you for your question. As acknowledged in the Limitations section, our evaluation focuses on coding tasks. We leave evaluation on other task domains to future work.
To clarify the runtime implications of local inference, the 3 tables below reports Pass@3, End-to-end runtimes per question and Number of delegations per question on Terminal-Bench 2.1 and SWE-rebench, using DGX Spark as the local device and GPT-5.5 as the cloud model. We compare our method with and without DFlash against the GPT-5.5-only API baseline. DFlash accelerates local inference through lossless speculative decoding. Without DFlash, our method can be slower than the API baseline. With DFlash enabled, it achieves slightly faster runtimes by accelerating local generation and reducing the time spent on cloud API calls. We do not train a new DFlash, it means that we only train the model to find the critical steps to delegate and do not change the model distribution. The delegation numbers always lower than cloud API.
Success rate
Number of delegations
End-to-end time
Thank you for the opportunity to clarify our SFT procedure. We use tasks from SWE-Smith and generate solutions with GPT-5.5. Since the GPT-5.5 outputs do not include CoT traces, we use Kimi-K3 to generate reasoning traces conditioned on these solutions. The resulting data are then used for SFT. We perform 200 SFT steps as a cold-start stage to teach the model the delegation format, using a learning rate of 1e-5 and a batch size of 64 on eight NVIDIA A100 GPUs.