acceptodds
Under review as a conference paper at ICLR 2027

EdgeWise: Bring Long-Horizon Agents to The Edge via Selective Cloud Delegation

28 views

Abstract

Complex and long-horizon agent tasks often rely on frontier models deployed in the cloud to achieve strong performance. However, using these models for every step of a long-horizon trajectory incurs substantial inference cost, even though not all steps may require the same level of model capability. In this paper, we find that allocating the LLM to only a small subset of key steps can achieve performance comparable to full LLM execution while significantly reducing inference cost. Motivated by this observation, we propose \sys, an edge-cloud collaboration framework that deploys an SLM on the local device as the primary solver and selectively delegates critical steps to a cloud LLM when stronger model capacity is needed. To enable autonomous delegation without additional models at inference time, we introduce a step-level RL method that jointly optimizes task solving and delegation, with fine-grained credit assignment for individual delegation decisions. We also introduce an external judge-based delegation strategy and use it to systematically investigate key design choices in edge-cloud collaboration, including when to delegate, how delegation decisions should be made, what context should be shared, and how frequently the cloud LLM should be invoked. Experiments show that \sys matches or outperforms pure-LLM execution while reducing inference cost up to 62% on SWE-Rebench and 34% on Terminal-Bench 2.1. We further deploy \sys on a local workstation, demonstrating comparable cost savings in a practical edge-cloud setting.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Agentic AI, Efficient Agent, Cost Saving, Coding Agent

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_iatkOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3(+1)
Read review
Paper summary

This paper presents EdgeWise, a framework in which SLMs handles most steps of a long-horizon agentic task and delegates only the hardest steps to LLMs. This reduces inference cost while preserving task performance. The paper groups agent actions into six types and analyzes how each type relates to task success. It finds that writing and editing actions benefit most from LLM delegation. Then the paper proposes two delegation methods. One is an inference-time approach that uses an external judge. The other is a trained self-delegation approach(SFT+RL) whose token-level credit assignment rewards delegating at useful steps, rather than simply minimizing LLM calls. Lastly, experiments on Terminal-Bench 2.1 and SWE-ReBench show lower cloud API costs with competitive performance.

Critical strengths

1.The paper studies how to allocate stronger models across different steps of long-horizon agent execution. The analysis of writing/editing operations also provides useful insights into where cloud-model delegation may be most beneficial. 2.The paper combines inference-time delegation using an external judge with a learned self-delegation mechanism. The token-level credit-assignment design is particularly interesting because it aims to optimize when to delegate rather than merely reducing the overall number of cloud-model calls.

Critical weaknesses

1.The definition of the token-level delegation advantage appears potentially inconsistent between the main text and Appendix H.1. In particular, the main text suggests that no additional advantage is assigned when local execution is sufficient, whereas Equation (3) appears to assign a positive advantage when the agent and evaluator agree on local execution. 2.The evaluation focuses primarily on coding tasks, limiting generalizability to broader long-horizon agents on edge devices. Moreover, the cost analysis excludes local inference costs, leaving the overall deployment efficiency unclear.

Questions for authors

1.Could the authors clarify the intended definition of Equation (3)? 2.Could the authors compare total cost (local inference + cloud API) with an LLM-only baseline under the same settings, together with success rate, number of delegations, and end-to-end runtime? How much would the reported savings change if local inference cost or tokens per delegated call increased? 3.Could the authors clarify the respective roles of GPT-5.5 and Kimi-K3 in generating SFT trajectories and describe the relevant training details, including hyperparameters, data construction, and data splits? These details are important for independently reproducing the self-delegation results.

3 comments
AuthorsOpenReview ↗

Reply(1/1)

W1.The definition of the token-level delegation advantage appears potentially inconsistent between the main text and Appendix H.1. In particular, the main text suggests that no additional advantage is assigned when local execution is sufficient, whereas Equation (3) appears to assign a positive advantage when the agent and evaluator agree on local execution. Q1.Could the authors clarify the intended definition of Equation (3)?

We thank reviewer's valuable question. We are sorry to the confuse of the equation 3, in the text before the equation 3, we mentioned that 'When both the SLM and the critic determine that the current step should be delegated to the LLM, we assign an advantage of +1'. This statement is same to the context in Section3.2.2 and the equation has some inaccurate, we have fixed it in the revision, you can see through openreview rebuttal revision.

W2.The evaluation focuses primarily on coding tasks, limiting generalizability to broader long-horizon agents on edge devices. Moreover, the cost analysis excludes local inference costs, leaving the overall deployment efficiency unclear.

Q2.Could the authors compare total cost (local inference + cloud API) with an LLM-only baseline under the same settings, together with success rate, number of delegations, and end-to-end runtime? How much would the reported savings change if local inference cost or tokens per delegated call increased?

Thank you for your question. As acknowledged in the Limitations section, our evaluation focuses on coding tasks. We leave evaluation on other task domains to future work.

To clarify the runtime implications of local inference, the 3 tables below reports Pass@3, End-to-end runtimes per question and Number of delegations per question on Terminal-Bench 2.1 and SWE-rebench, using DGX Spark as the local device and GPT-5.5 as the cloud model. We compare our method with and without DFlash against the GPT-5.5-only API baseline. DFlash accelerates local inference through lossless speculative decoding. Without DFlash, our method can be slower than the API baseline. With DFlash enabled, it achieves slightly faster runtimes by accelerating local generation and reducing the time spent on cloud API calls. We do not train a new DFlash, it means that we only train the model to find the critical steps to delegate and do not change the model distribution. The delegation numbers always lower than cloud API.

Success rate

Self-Delegation w/o DflashSelf-Delegation w/ DflashCloud model using API
Terminal-Bench2.180.9082.0282.02
SWE-Rebench64.5564.5563.64

Number of delegations

Self-Delegation w/o DflashSelf-Delegation w/ DflashCloud model using API
Terminal-Bench2.15.375.6612.88
SWE-Rebench15.5115.4333.86

End-to-end time

Self-Delegation w/o DflashSelf-Delegation w/ DflashCloud model using API
Terminal-Bench2.1703.1s(80.90)588s(82.02)608.4s(82.02)
SWE-Rebench827.4s(64.55)569s(64.55)715.1(63.64)

Q3.Could the authors clarify the respective roles of GPT-5.5 and Kimi-K3 in generating SFT trajectories and describe the relevant training details, including hyperparameters, data construction, and data splits? These details are important for independently reproducing the self-delegation results.

Thank you for the opportunity to clarify our SFT procedure. We use tasks from SWE-Smith and generate solutions with GPT-5.5. Since the GPT-5.5 outputs do not include CoT traces, we use Kimi-K3 to generate reasoning traces conditioned on these solutions. The resulting data are then used for SFT. We perform 200 SFT steps as a cold-start stage to teach the model the delegation format, using a learning rate of 1e-5 and a batch size of 64 on eight NVIDIA A100 GPUs.

Thank you for the detailed and helpful responses. The authors have adequately addressed my main concerns, particularly regarding the clarification of the step-level advantage definition, the additional runtime and delegation statistics, and the SFT training details. These clarifications improve the transparency and reproducibility of the work. While some practical deployment considerations remain open, I believe the responses sufficiently resolve my major concerns.

Dear Reviewer iatk,

Thank you for your positive feedback and for acknowledging our efforts to address your concerns. We are glad that our responses have adequately resolved your major concerns about local inference cost and training settings. Given that your main concerns have been addressed, we would sincerely appreciate it if you could consider updating your rating accordingly.

Thank you again for your time and constructive feedback.

Reviewer_KgZjOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes EdgeWise, a framework in which a small local model handles an agent trajectory and selectively delegates individual steps to a stronger cloud model. It studies external-judge routing and trains the local model for self-delegation using task rewards, a cloud-usage regularizer, and critic-provided supervision on delegation tokens. Experiments on Terminal-Bench and SWE-ReBench show substantial reductions in cloud API cost with generally comparable task performance.

Critical strengths
  1. The writing is clear.
  2. The approach achieves a useful performance-cost trade-off on realistic coding tasks.
  3. The ablations support the proposed training signal.
Critical weaknesses
  1. The step-level advantage is defined inconsistently. Section 3.2.2 specifies no additional advantage when local execution is sufficient, while Equation 3 assigns +1 when both the policy and critic choose local execution. The value of β and the critic prompt are also missing, leaving the exact training procedure unclear.
  2. The claim of matching cloud-model performance needs qualification. In Table 1, self-delegation has lower Pass@1 than the cloud-only baseline in three of the four configurations, although Pass@3 is often comparable. The main comparison also lacks uncertainty estimates, leaving it unclear how reliably the method preserves single-attempt performance.
Questions for authors
  1. Were the tasks in the 2.4k-trajectory analysis disjoint from the final evaluation tasks? Were the delegation criteria fixed before running the final evaluation?
  2. When both the policy and critic choose local execution, is the additional advantage 0 or +1? Please clarify the implemented advantage definition and provide the value of β and the critic prompt.
3 comments
AuthorsOpenReview ↗

Reply(1/1)

W1.The step-level advantage is defined inconsistently. Section 3.2.2 specifies no additional advantage when local execution is sufficient, while Equation 3 assigns +1 when both the policy and critic choose local execution. The value of β and the critic prompt are also missing, leaving the exact training procedure unclear.

2.When both the policy and critic choose local execution, is the additional advantage 0 or +1? Please clarify the implemented advantage definition and provide the value of β and the critic prompt.

We thank the reviewer for this valuable question and apologize for the confusion caused by Equation (3). As stated immediately before the equation, “When both the SLM and the critic determine that the current step should be delegated to the LLM, we assign an advantage of +1.” This description is consistent with Section 3.2.2. When both choose local execution, no additional advantage is assigned (i.e., it is 0). Equation (3) contains an inaccurate, which we have corrected in the revised manuscript on OpenReview.

In our training, we set β = 0.3, which makes the delegation signal strong to influence decisions without overwhelming the task-success signal, so we empirical choose it to 0.3. We will release the critic prompt in the code repo if our paper can be accepted.

W2.The claim of matching cloud-model performance needs qualification. In Table 1, self-delegation has lower Pass@1 than the cloud-only baseline in three of the four configurations, although Pass@3 is often comparable. The main comparison also lacks uncertainty estimates, leaving it unclear how reliably the method preserves single-attempt performance.

  • We report below the individual Pass@1 results from the three runs used to compute Pass@3, together with their means and standard deviations. Across these three runs, self-delegation achieves average Pass@1 accuracy close to that of the cloud-only baseline on both Terminal-Bench 2.1 and SWE-ReBench. These results suggest that the single attempt performance gaps observed in Table 1 are not consistently reproduced across runs.

Terminal-Bench

Vanilla 1Vanilla 2Vanilla 3Self-delegation 1Self-delegation 2Self-delegation 1
GPT5.574.1671.9169.6671.9174.1669.66
Opus4.867.4267.4262.9266.2962.9268.53
  • Avg. +- std
VanillaSelf-delegation
GPT5.571.91 +- 1.8471.91 +- 1.84
Opus4.865.91 +- 2.6065.91 ± 2.31

SWE-Rebench

Vanilla 1Vanilla 2Vanilla 3Self-delegation 1Self-delegation 2Self-delegation 3
GPT5.556.3658.1859.0958.1860.0058.18
Opus4.860.0061.8161.8157.2760.0062.72
  • Avg. +- std
VanillaSelf-delegation
GPT5.557.88 +- 1.3958.79 +- 0.86
Opus4.861.21 ± 0.8560.00 ± 2.22

Q1.Were the tasks in the 2.4k-trajectory analysis disjoint from the final evaluation tasks? Were the delegation criteria fixed before running the final evaluation?

Thank you for raising this question. We would like to clarify that the delegation criteria are fixed during training. During the final evaluation, no explicit external criteria or rules are provided to guide delegation decisions; instead, the model learns to make these decisions autonomously. Furthermore, our final evaluation covers Terminal-Bench 2.1 and SWE-ReBench across different harnesses, including OpenHands, Terminus2, and Pi, which exhibit different distributions from those in the 2.4k trajectories. These results suggest that our method does not overfit to specific data distributions.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer KgZj,

I am writing to gently follow up on our rebuttal. During the rebuttal period, we reported the mean and standard deviation over three runs with different random seeds, and also give the further clarification of our SFT training data.

We hope these additional explanations and results have addressed your concerns. We would greatly appreciate it if you could take a moment to review our responses. If your concerns have been resolved, we would also sincerely appreciate it if you could consider updating your rating accordingly.

Thank you very much for your time and constructive feedback.

Best regards,

Authors

Thank you for the clarifications and additional results. I will maintain my positive rating.

Reviewer_ErcQOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents EdgeWise, a framework in which a local small language model handles long-horizon coding tasks and selectively delegates steps to a cloud LLM. Motivated by the observation that some steps benefit more from stronger models, the authors propose an external judge and a learned self-delegation policy. The latter uses SFT followed by reinforcement learning, combining task success and cloud-usage rewards with feedback on individual delegation decisions, without requiring a separate router at inference time. The paper also studies context sharing and delegation frequency.

Critical strengths

1.The paper identifies a useful pattern: cloud assistance benefits different agent steps unequally, with code writing and editing showing the strongest association with task success. This observation gives selective delegation a concrete basis in long-horizon coding tasks. 2.The self-delegation method combines task-level rewards with feedback on individual delegation decisions. This directly addresses the difficulty of learning when to seek help from final task outcomes alone, while removing the need for a separate router at inference time.

Critical weaknesses

1.The DGX Spark experiment reports task completion times but lacks a cloud-only latency baseline under the same evaluation conditions. Without this comparison, it is unclear whether the API cost savings come at the expense of longer execution time, leaving an important practical trade-off unexamined. 2.Although delegating steps to the SLM can reduce cloud decoding costs, the cloud LLM must process the context accumulated during local execution when it resumes. Prefilling this new context, and potentially reprocessing earlier history after cache misses, may offset some savings.

Questions for authors

The cloud LLM must prefill the text generated by the SLM during local execution, which may offset part of the decoding cost savings. Although the reported API costs include input tokens, the paper does not separately quantify this handoff cost. A breakdown of input and output costs would clarify how much the additional prefill reduces the savings from local execution.

3 comments
AuthorsOpenReview ↗

Reply(1/1)

W1.The DGX Spark experiment reports task completion times but lacks a cloud-only latency baseline under the same evaluation conditions. Without this comparison, it is unclear whether the API cost savings come at the expense of longer execution time, leaving an important practical trade-off unexamined.

Thank you for your question. As acknowledged in the Limitations section, our evaluation focuses on coding tasks. We leave evaluation on other task domains to future work.

To clarify the runtime implications of local inference, the table below reports end-to-end runtimes on Terminal-Bench 2.1 and SWE-rebench, using DGX Spark as the local device and GPT-5.5 as the cloud model. We compare our method with and without DFlash against the GPT-5.5-only API baseline. Values in parentheses indicate accuracy. DFlash accelerates local inference through lossless speculative decoding. Without DFlash, our method can be slower than the API baseline. With DFlash enabled, it achieves slightly faster runtimes by accelerating local generation and reducing the time spent on cloud API calls. We do not train a new DFlash, it means that we only train the model to find the critical steps to delegate and do not change the model distribution.

Self-Delegation w/o DflashSelf-Delegation w/ DflashCloud model using API
Terminal-Bench2.1703.1s(80.90)588s(82.02)608.4s(82.02)
SWE-Rebench827.4s(64.55)569s(64.55)715.1(63.64)

W2.Although delegating steps to the SLM can reduce cloud decoding costs, the cloud LLM must process the context accumulated during local execution when it resumes. Prefilling this new context, and potentially reprocessing earlier history after cache misses, may offset some savings.

Q1.The cloud LLM must prefill the text generated by the SLM during local execution, which may offset part of the decoding cost savings. Although the reported API costs include input tokens, the paper does not separately quantify this handoff cost. A breakdown of input and output costs would clarify how much the additional prefill reduces the savings from local execution.

Thank you for raising this question. We report the input token count, output token count, and prefix cache hit rate across the 89 tasks in Terminal-Bench 2.1. Although our method has a lower prefix cache hit rate than the GPT-5.5-only baseline, edge–cloud collaboration substantially reduces both cloud input and output token counts, resulting in significant API cost savings.

Self-delegationPure GPT5.5
Input tokens3,517,44819,137,467
Output tokens1,409,1352,464,088
Prefix Cache hit tokens2,323,08814,858,496
Prefix Cache Hit rate66.04%77.64%

Dear Author,

Thank you for your reply. I have no further questions at this time and will keep my original positive rating. Best wishes.

Dear Reviewer ErcQ,

Thank you for your positive feedback of our work and for acknowledging our efforts to address your concerns. We are glad that our clarifications on local inference cost and details of cost have resolved your main concerns. In light of this, we would sincerely appreciate it if you could consider raising your rating. Thank you again for your time and constructive feedback.

Thanks,

Authors

Reviewer_q9uEOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper asks whether a long-horizon coding agent needs a frontier model at every step. It argues that a small local model can do most of the work if it hands a few key steps, mainly code edits, to a cloud model. EdgeWise makes this decision either with a prompted external judge or by training the small model to delegate through SFT and RL, with step-level feedback from an LLM critic. On Terminal-Bench and SWE-ReBench, the authors report about the same success rate as the cloud model alone at a much lower API cost.

Critical strengths
  1. The cost savings are large and hold up against the obvious alternative. Self-delegation keeps pass@3 close to the cloud model alone at a third to half less cloud spending, and it beats simply lowering the reasoning effort (figure 1). This is directly useful for running coding agents on a budget.

  2. The learned policy transfers to cloud models and a harness it was not trained with, where the prompted judge falls behind. This supports the paper's argument against routers tied to a fixed model pool.

Critical weaknesses
  1. The paper does not show what RL-trained self-delegation, its main technical contribution, adds over simpler policies. This matters because the idea has close precedents. MTRouter, which the paper compares against, already routes per turn, and Router-R1 trains an LLM with RL and a cost reward to decide when to call other models. Here the simpler policies already do well. Random delegation matches gpt-5.5 alone at lower cost (figure 3), and the training-free judge beats self-delegation at some thresholds (figure 7b). Neither is compared at matched cloud usage, and the judge variant is left out of table 1. The evidence that the policy learns to pick critical steps is also circular, because the step-level reward comes from a critic that applies the judge's own hand-written rule.

  2. The headline claim, similar accuracy at much lower cost, is not measured carefully enough. The accuracies look like single runs, and the gaps between methods are about the size of the run-to-run variation. With opus 4.8, self-delegation is below the cloud model in pass@1 on both benchmarks. The cost also counts only cloud API spending, although the edge deployment takes about 12 minutes per task.

Questions for authors
  1. Can you compare self-delegation, random delegation and the thresholded judge at matched cloud usage or cost, with means and intervals over several runs?

  2. Are the accuracies in table 1 from a single run? Why does gpt-5.5 alone score differently in table 1 and in figures 3 and 5?

  3. How does the self-delegation model do with delegation turned off? Its SFT data are gpt-5.5 trajectories, so part of the gain may come from a stronger small model.

  4. Were prompt-caching discounts applied in the cost numbers? Could you also report wall-clock time per task and the cloud cost of RL training?

5 comments
AuthorsOpenReview ↗

Reply(1/4)

We sincerely thank reviewers for their thoughtful comments and insightful questions.

1.The paper does not show what RL-trained self-delegation, its main technical contribution, adds over simpler policies. This matters because the idea has close precedents. MTRouter, which the paper compares against, already routes per turn, and Router-R1 trains an LLM with RL and a cost reward to decide when to call other models. Here the simpler policies already do well. Random delegation matches gpt-5.5 alone at lower cost (figure 3), and the training-free judge beats self-delegation at some thresholds (figure 7b). Neither is compared at matched cloud usage, and the judge variant is left out of table 1. The evidence that the policy learns to pick critical steps is also circular, because the step-level reward comes from a critic that applies the judge's own hand-written rule.

(1).The paper does not show what RL-trained self-delegation, its main technical contribution, adds over simpler policies. This matters because the idea has close precedents. MTRouter, which the paper compares against, already routes per turn, and Router-R1 trains an LLM with RL and a cost reward to decide when to call other models.

We note that our method and prior works such as MTRouter and Route-R1 differs in both training methodology and the analysis it is grounded upon. Our methods have some key differents:

  • An Explicit Rubric-Guided Delegation Strategy.

    • Our approach is grounded in an empirical analysis of where stronger model capabilities are most valuable. Our random delegation study reveals that delegating only a subset of critical steps can achieve performance comparable to full cloud-model execution. However, MTRouter uses embeddings and Router-R1 learns from trajectory pairs. Table 1 further shows that existing turn-level delegation baselines do not consistently outperform random delegation. This suggests that learning a routing policy alone does not ensure effective allocation of model capabilities across steps, motivating explicit guidance on which steps benefit most from cloud assistance.
    • Building on this analysis, we derive a delegation rubric that distinguishes steps requiring stronger reasoning from those that can be handled locally. We then use this rubric to provide targeted step-level supervision during training, enabling the local model to allocate cloud resources more selectively. Our contribution thus includes both empirical insights into step selection in long-horizon tasks and a training method that translates these insights into an effective delegation policy.
  • No additional router at inference time. Methods such as MTRouter and Router-R1 rely on a separately trained router, which introduces an additional component to deploy and maintain. In contrast, EdgeWise jointly trains the local small language model to solve tasks and decide when to delegate to the cloud. A single local model therefore handles both task execution and delegation, simplifying deployment on edge devices.

  • Reduced dependence on a fixed model pool and internalize delegation capability into model. Current routers trained over a predefined model pool learn routing decisions tied to the capabilities of its candidate models. Updating or replacing these models may therefore require router retraining or adaptation. In contrast, EdgeWise can internailize the delegation ability into the model, the model will know which steps may hard for himself, rather than explicitly selecting among candidate models based on their profiles. This reduces dependence on a specific cloud model and makes the delegation policy more adaptable to changes in the cloud backend.

AuthorsOpenReview ↗

Reply(2/4)

(2).Here the simpler policies already do well. Random delegation matches gpt-5.5 alone at lower cost (figure 3), and the training-free judge beats self-delegation at some thresholds (figure 7b).

  • Random Delegation Motivates More Selective Allocation. We use random delegation as a preliminary experiment to motivate our approach. Even this simple strategy can outperform some existing baselines on long-horizon tasks, suggesting that stronger models are not needed at every step and motivating us to reconsider how model capacity is allocated across the trajectory. However, as shown in Figure 3, random delegation exhibits substantial variability because it cannot identify which steps benefit most from cloud assistance. This motivates learning a selective delegation policy that allocates stronger model capabilities more effectively.

  • Self-Delegation can internalize the delegation capability We also explore a rule-based strategy within EdgeWise, introduced as Binary Delegation in Section 3. This approach relies on an external judge, such as GPT-5.4-mini, which adds inference latency and API costs. In Table1, self-delegation shows lower cost than binary delegation. In contrast, self-delegation internalizes the delegation capability within the local model, enabling it to decide when to invoke the cloud without an external judge. Through RL training, the local model learns to balance local execution and cloud assistance, becoming better suited to edge–cloud collaboration.

(3).Neither is compared at matched cloud usage, and the judge variant is left out of table 1.

Q1.Can you compare self-delegation, random delegation and the thresholded judge at matched cloud usage or cost, with means and intervals over several runs?

  • The judge-based method is referred to as “Binary Delegation” in Section 3 and is already included in the comparisons in Table 1.

  • We evaluate on Terminal-Bench 2.1 using GPT-5.5 with xhigh reasoning effort as the cloud model and a cloud inference budget of $0.70 per task. A task is considered successful only if it is solved correctly within this budget. The results show that self-delegation achieves higher accuracy under the same cost constraint, suggesting that it allocates cloud resources more effectively. This finding helps explain how self-delegation can maintain strong performance while reducing total inference cost.

Pure GPT5.540% random delegationBinary DelegationThreshold(20)Threshold(40)Threshold(60)Threshold(80)Self-Delegation
Acc +- std41.57 +- 1.9549.81 +- 5.5450.19 +- 3.2451.69 +- 2.2553.93 +- 2.3451.69 +- 3.3750.56 +- 3.8954.86 +- 4.25

(4).The evidence that the policy learns to pick critical steps is also circular, because the step-level reward comes from a critic that applies the judge's own hand-written rule.

We would like to clarify that the model used for step analysis in Section 3.1 differs from the training critic: GPT-5.5 assists with step categorization, whereas GPT-5.4-mini serves as the critic. Specifically, the harness records terminal commands with explicit syntax and identifiable functions which are not classified by GPT5.5. GPT-5.5 helps group these commands into semantic action categories but does not directly determine which steps are critical. The results further support the effectiveness of this rule. During evaluation, we do not rely on external rules, yet the method achieves good accuracy at a lower cost. These results support our analysis that such steps are critical.

AuthorsOpenReview ↗

Reply(3/4)

W2.The headline claim, similar accuracy at much lower cost, is not measured carefully enough. The accuracies look like single runs, and the gaps between methods are about the size of the run-to-run variation. With opus4.8, self-delegation is below the cloud model in pass@1 on both benchmarks.

Q2.Are the accuracies in table 1 from a single run? Why does gpt-5.5 alone score differently in table 1 and in figures 3 and 5?

  • We would like to clarify that Table 1 reports Pass@1 and Pass@3, which are commonly used benchmark metrics. Pass@1 measures the proportion of tasks solved in a single attempt, while Pass@3 measures the proportion solved in at least one of three attempts. Figures 3 and 5 report results averaged over three independent runs to reduce the influence of run-to-run variability and make our analysis more robust.

  • We report below the individual Pass@1 results from the three runs used to compute Pass@3, together with their means and standard deviations. Across these three runs, self-delegation achieves average Pass@1 accuracy close to that of the cloud-only baseline on both Terminal-Bench 2.1 and SWE-ReBench. These results suggest that the single attempt performance gaps observed in Table 1 are not consistently reproduced across runs.

Terminal-Bench

Vanilla 1Vanilla 2Vanilla 3Self-delegation 1Self-delegation 2Self-delegation 3
GPT5.574.1671.9169.6671.9174.1669.66
Opus4.867.4267.4262.9266.2962.9268.53
  • Avg. +- std
VanillaSelf-delegation
GPT5.571.91 +- 1.8471.91 +- 1.84
Opus4.865.91 +- 2.6065.91 ± 2.31

SWE-Rebench

Vanilla 1Vanilla 2Vanilla 3Self-delegation 1Self-delegation 2Self-delegation 3
GPT5.556.3658.1859.0958.1860.0058.18
Opus4.860.0061.8161.8157.2760.0062.72
  • Avg. +- std
VanillaSelf-delegation
GPT5.557.88 +- 1.3958.79 +- 0.86
Opus4.861.21 ± 0.8560.00 ± 2.22

Q3.How does the self-delegation model do with delegation turned off? Its SFT data are gpt-5.5 trajectories, so part of the gain may come from a stronger small model.

Thank you for raising this question. To assess whether the gains come from improved local task-solving capability, we evaluate our trained model on Terminal-Bench 2.1 with cloud delegation disabled. Its performance is similar to that of the vanilla local model, suggesting that the gains primarily come from effective delegation rather than a substantial improvement in standalone capability. Our SFT stage is designed primarily to teach the delegation format. Tokens generated by the cloud model, including its chain-of-thought, are masked out of the SFT loss and therefore do not directly supervise the local model. The subsequent RL stage trains the model to decide when to invoke the cloud. Together, these results support effective delegation as the main source of the performance gains.

Vanilla Qwen3.5-9BSelf-Delegation
Terminal-Bench2.119.1020.22

Comments on the paper

1 comment

Dear Reviewers and Area Chair,

Thank you for your valuable questions and feedback. During the rebuttal period, we have addressed the following key concerns:

  1. Stability of Self-Delegation: We report the mean and standard deviation across multiple runs to demonstrate the stability of our self-delegation method. The results show that self-delegation achieves performance close to the pure LLM baseline while substantially reducing costs, even when accounting for standard deviation.

  2. Wall-Clock Time: We provide additional wall-clock time measurements, showing that our method can achieve lower latency than an API-only approach when using DFlash.

  3. Cost Analysis: We provide a more detailed cost breakdown, including input and output token usage, to clarify how our method achieves cost savings.

  4. Motivation: We further clarify the motivation behind our work and highlight the key insights driving our approach.

We sincerely appreciate your time and constructive feedback. We would greatly appreciate it if you could take another look at our responses and let us know whether your concerns have been adequately addressed.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.