acceptodds
Under review as a conference paper at ICLR 2027

LEASH: Uncertainty-Aware Weak-to-Strong Escalation and Oversight for Computer Use

Abstract

Computer-use agents increasingly tackle long-horizon tasks, where running a frontier model at every step is expensive and a small open-weight model alone often fails. Weak-to-strong escalation offers a middle ground: a weak agent acts by default and hands control to a strong one only when needed. The core challenge is deciding when to escalate, which requires an uncertainty signal cheap enough to check at every step. After handoff the risk remains: the strong agent can still go astray, and a black-box API exposes no internals to monitor. To this end, we propose LEASH, a training-free framework that unleashes the strong agent only when the weak agent's latent uncertainty signals it is needed, and keeps it on a leash afterward. LEASH has three layers: initial task uncertainty for early routing, step-level uncertainty for runtime escalation, and, after handoff, a shadow observer for continued oversight of the strong black-box agent. At the core of all three layers is latent uncertainty: we compare the weak model's hidden states with those from successful and failed trajectories, and a state that looks more like the failures signals higher uncertainty. During weak-agent execution, runtime uncertainty reuses activations already produced during action generation, so it needs no extra forward pass or dedicated monitor training. Experiments on OSWorld across four contemporary weak-agent backbones show that LEASH consistently improves over standalone weak agents and outperforms random escalation by 6.55–11.94 points at matched escalation rates. With Qwen3.5-35B-A3B, LEASH achieves 73.66% task success at $0.4117 per task, outperforming always-on Claude Opus 5 (72.90%) while reducing inference cost by 19.2%.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Computer Use Agents, Weak-to-Strong Escalation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_aYg1OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LEASH, a framework that lets a weak computer-use agent hand over tasks to a stronger model when needed. It compares the weak model’s hidden states with successful and failed reference trajectories using a relative Mahalanobis distance score. This score guides both initial routing and runtime handoff. After handoff, the weak model acts as a shadow observer and can ask the strong model to reconsider its actions. The method requires no backbone fine-tuning and reuses existing hidden states during weak-agent execution. Experiments on OSWorld with four weak backbones show improvements over standalone weak agents and random routing. With Qwen3.5-35B-A3B, LEASH reports 73.66% success at an estimated 19.2% lower inference cost than always-on Opus, which achieves 72.90%.

Critical strengths
  1. LEASH offers a practical way to combine initial routing, runtime handoff, and post-handoff monitoring using the weak model’s hidden states (Sections 3.2–3.4). Runtime scoring reuses activations from action generation without an extra forward pass, making it useful for long computer-use tasks where monitoring costs matter.

  2. The results across four weak backbones show promising gains in task success (Table 1). In particular, the Qwen3.5 configuration achieves similar observed success to always-on Opus at 19.2% lower estimated inference cost. The equal-cost comparison in Figure 1 also supports the practical value of selective escalation.

Critical weaknesses
  1. The experiments do not isolate the benefit of Shadow monitoring, which is a main contribution. The Shadow and No-Shadow runs escalate different tasks, as acknowledged in Appendix B.5. Table 4 also shows higher success on tasks where no checkpoint was triggered. Thus, the 5.72-point gain in Table 3 cannot be attributed to Shadow intervention alone. Comparisons from the same handoff states, including random or fixed-interval checkpoints at matched cost, are needed to establish whether the uncertainty signal selects useful intervention times.

  2. The evidence for the proposed escalation signal is limited by the baselines. Table 1 matches escalation rates, but LEASH costs more than random routing for every backbone. Figure 1 partly addresses this with an equal-cost comparison for Qwen3.5, but comparisons with simple stall-detection rules or token-probability routing are missing. Table 2 compares uncertainty scores only through AUROC, not downstream task success. This leaves unclear how much of the gain comes from the proposed signal rather than the overall workflow and additional computation.

Questions for authors
  1. Why do the Shadow and No-Shadow runs escalate different tasks if Shadow only acts after handoff (Appendix B.5)? Were the escalation settings identical, with differences caused by independent runs? Does the gain remain when both methods start from the same handoff states?

  2. Does uncertainty-triggered intervention outperform random or fixed-interval checkpoints with the same intervention budget and reasoning effort (Section 3.4, Tables 3–4)? This would help separate the value of the uncertainty signal from the benefit of additional reflection.

  3. How does LEASH compare with token-probability routing or a simple stall-detection rule at matched inference cost (Tables 1–2)? These comparisons would clarify whether the proposed hidden-state score improves the final cost–success trade-off.

  4. Can you provide paired confidence intervals for the success differences and compare costs on the same task IDs, given the nine missing Opus token logs (Appendix B.1)? This would help assess the reliability of the reported accuracy and cost advantages.

5 comments
AuthorsOpenReview ↗

Responses to Reviewer aYg1 comments (Part 1)

We thank the reviewer for the careful and constructive feedback. We address the concerns below with additional experiments and analyses.

W1: The experiments do not isolate the benefit of Shadow monitoring, which is a main contribution. The Shadow and No-Shadow runs escalate different tasks, as acknowledged in Appendix B.5. Table 4 also shows higher success on tasks where no checkpoint was triggered. Thus, the 5.72-point gain in Table 3 cannot be attributed to Shadow intervention alone. Comparisons from the same handoff states, including random or fixed-interval checkpoints at matched cost, are needed to establish whether the uncertainty signal selects useful intervention times.

Q1: Why do the Shadow and No-Shadow runs escalate different tasks if Shadow only acts after handoff (Appendix B.5)? Were the escalation settings identical, with differences caused by independent runs? Does the gain remain when both methods start from the same handoff states?

Q2: Does uncertainty-triggered intervention outperform random or fixed-interval checkpoints with the same intervention budget and reasoning effort (Section 3.4, Tables 3–4)? This would help separate the value of the uncertainty signal from the benefit of additional reflection.

Response: (a) Explaination of Table 4. The higher success on the 137 untriggered tasks (74.5% vs. 69.3%) may reflect differences between the independently executed Shadow and No-Shadow runs, rather than checkpoint interventions. Importantly, for the task subsets defined by the Shadow run, both No Shadow (48.4% vs. 69.3%) and always-on Opus (51.6% vs. 74.5%) perform substantially worse on triggered than untriggered tasks. This suggests that Shadow's uncertainty signal tends to flag tasks that are difficult even for the strong agent. We will clarify this point in the revised manuscript to make this clear.

(b)Why the escalation sets differ. We used identical escalation settings for the Shadow and No-Shadow runs but executed them independently. We found that variations in the Opus Manager’s subgoal decompositions and screen observations affected escalation decisions by changing the weak agent’s inputs and uncertainty scores. These differences arose before handoff; Shadow operates only after handoff and does not affect pre-handoff routing.

(c)Shadow intervention timing. To better assess Shadow’s contribution, we introduce a random-Shadow baseline with matched intervention counts.

Table 1: Comparison of Shadow intervention strategies.

ConfigurationSuccess (%)
No Shadow67.94
Random Shadow (matched intervention count)70.23
LEASH (uncertainty-triggered Shadow)73.66

LEASH achieves 3.4 percentage points higher success than random Shadow with the same number of interventions. This provides additional evidence consistent with the value of uncertainty-guided checkpoint selection, rather than simply intervening more often. We also revised Figures 8–10 to better illustrate the execution states at handoff and how strong-agent takeover recovers stalled trajectories.

AuthorsOpenReview ↗

Responses to Reviewer aYg1 comments (Part 2)

W2: The evidence for the proposed escalation signal is limited by the baselines. Table 1 matches escalation rates, but LEASH costs more than random routing for every backbone. Figure 1 partly addresses this with an equal-cost comparison for Qwen3.5, but comparisons with simple stall-detection rules or token-probability routing are missing. Table 2 compares uncertainty scores only through AUROC, not downstream task success. This leaves unclear how much of the gain comes from the proposed signal rather than the overall workflow and additional computation.

Q3: How does LEASH compare with token-probability routing or a simple stall-detection rule at matched inference cost (Tables 1–2)? These comparisons would clarify whether the proposed hidden-state score improves the final cost–success trade-off.

Response: (a) Cost performance trade-off. As you pointed out, LEASH costs 9–26% more than random routing at the same escalation rate, because the gate escalates harder and longer tasks and escalated tasks carry the weak agent’s steps before handoff, which is expected outcome by design. We would like to amphasize that, at equal cost, as in Fig. 1, LEASH’s performance gain is +9.1 points on Qwen3.5 compare to that very random routing baseline, implying that LEASH consume the computation budget more effectively to turn the extra computation with the strong model into actual task success rate gain.

(b) Comparison with different baselines. Thanks for suggesting the constructive experiments. In this rebuttal, we compare LEASH with simple handoff signals and different baselines using end-to-end task success and inference cost, rather than AUROC alone.. Each one either holds the workflow fixed and changes only the trigger signal, or holds cost fixed.

(b-1) Comparison with simple token-probability and stall-detection baselines. By reflecting your suggestion, we compare both gates within the same LEASH workflow, changing only the trigger signal. We also increase LEASH’s threshold to match the token-probability gate’s escalation rate.

Table 2: Comparison with token-probability and stall-detection gates within the same LEASH workflow.

GateEsc. (%)Success (%)Cost ($/task)
LEASH (higher threshold)44.665.500.3208
Token probability44.662.790.3057
Stall rule42.661.630.3013

LEASH retains higher observed success than token-probability routing at the same escalation rate, and than stall detection at a similar rate.

(b-2) Comparison with additional advanced baselines. We further compare LEASH against more competative baseline methods, StepWise[1] and Progress Advantage[2], on EvoCUA-8B, as shown in Table 3.

Table 3: Comparison with existing escalation and uncertainty estimation methods on EvoCUA-8B.

MethodEsc. (%)Success (%) ↑Cost ($/task) ↓
EvoCUA-8B (no escalation)0.042.37–
StepWise49.658.400.2824
Progress Advantage46.962.980.3533
LEASH45.467.180.3627

For Progress Advantage, we keep the LEASH workflow unchanged and replace only the trigger signal to the implicit process reward signal proposed in [2]. LEASH achieves higher success, supporting the effectiveness of our hidden-state uncertainty signal.

StepWise, in contrast, is a complete escalation method with its own trained monitor and escalation procedure. Our LEASH outperforms StepWise despite using a training-free approach, and retains its advantage under an equal-cost comparison with a StepWise–Opus mixture.


References

[1]Step-level optimization for efficient computer-use agents. [2]Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents.

AuthorsOpenReview ↗

Responses to Reviewer aYg1 comments (Part 3)

Q4: Can you provide paired confidence intervals for the success differences and compare costs on the same task IDs, given the nine missing Opus token logs (Appendix B.1)? This would help assess the reliability of the reported accuracy and cost advantages.

Response: In this rebuttal, table 4 here reports task-matched success differences and cost savings with 95% paired-bootstrap confidence intervals. LEASH consistently reduces cost, even when all nine missing Opus costs are set to $0.

Table 4. Paired comparisons with always-on Opus.

BackboneΔ Success (pp)Cost saving (%)Conservative saving (%)
Qwen3.5+0.76 [−3.8, +5.3]18.8 [10.8, 26.3]16.3 [7.7, 24.0]
Qwen3.8-27B−3.05 [−8.0, +1.9]29.0 [21.5, 36.3]26.4 [18.1, 34.1]

Notes: Brackets show 95% paired task-bootstrap confidence intervals. Δ Success = LEASH − Opus, in percentage points. We calculate cost savings on the 253 tasks with complete logs for both methods. For the conservative analysis, we include all 262 tasks and assign $0 to the nine missing Opus costs.

We thank the reviewer for pointing this out and will include these experiments above in the refined submission.

Reviewer_rBezOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes LEASH, a training-free framework for weak-to-strong escalation in computer-use agents. It uses hidden-state representations and Mahalanobis distance to estimate failure risk, supporting initial routing, runtime escalation, and post-handoff oversight. Experiments on OSWorld demonstrate improved task success and favorable cost–performance trade-offs across four weak-agent backbones.

Critical strengths
  1. The paper addresses a practical and important problem in efficient computer-use agents.
  2. The experimental results are promising, with consistent improvements across multiple backbones and useful ablation analyses.
Critical weaknesses
  1. Insufficient validation of Shadow oversight. Although Shadow oversight improves task success, it is unclear whether the improvement comes from uncertainty-guided intervention or simply from allowing the strong model to reflect and reconsider its actions. A comparison against random or periodic reflection with matched intervention budgets is necessary to establish the effectiveness of the proposed uncertainty signal.
  2. Missing competitive baselines and insufficient cost-controlled comparisons. The paper lacks comparisons with closely related methods, such as Adaptive Vision-Language Model Routing for Computer Use Agents, making it difficult to assess its advantage over existing approaches. While the training-free nature of LEASH is an appealing advantage, this does not eliminate the need for comparisons with existing approaches. Even if LEASH cannot outperform them in absolute performance, a competitive accuracy–cost trade-off would still be meaningful. Moreover, the random escalation baseline is matched by escalation rate rather than inference cost, despite LEASH incurring higher costs at comparable escalation rates (Table 1). These limitations make it difficult to assess the practical advantages of the proposed method.
Questions for authors

Please refer to the concerns raised in the Weaknesses.

4 comments
AuthorsOpenReview ↗

Responses to Reviewer rBez comments

We thank the reviewer for the careful and constructive feedback. We address the concerns below with additional experiments and analyses.

W1: Insufficient validation of Shadow oversight. Although Shadow oversight improves task success, it is unclear whether the improvement comes from uncertainty-guided intervention or simply from allowing the strong model to reflect and reconsider its actions. A comparison against random or periodic reflection with matched intervention budgets is necessary to establish the effectiveness of the proposed uncertainty signal.

Response: Shadow intervention timing. In this rebuttal, we compare random and uncertainty-triggered Shadow with matched intervention counts in table 1. LEASH outperforms random Shadow with the same intervention count.

Table 1. Shadow intervention strategies on Qwen3.5.

ConfigurationSuccess (%)
No Shadow67.94
Random Shadow (matched intervention count)70.23
LEASH (uncertainty-triggered Shadow)73.66

Table 4 of the submission further shows that always-on Opus performs worse on Shadow-triggered tasks than on unshadowed tasks (51.6% vs. 74.5%), suggesting that Shadow flags cases that remain difficult even for the strong agent.

Together, these results support uncertainty-guided checkpoint selection beyond simply intervening more often.

W2: Missing competitive baselines and insufficient cost-controlled comparisons. The paper lacks comparisons with closely related methods, such as Adaptive Vision-Language Model Routing for Computer Use Agents, making it difficult to assess its advantage over existing approaches. While the training-free nature of LEASH is an appealing advantage, this does not eliminate the need for comparisons with existing approaches. Even if LEASH cannot outperform them in absolute performance, a competitive accuracy–cost trade-off would still be meaningful. Moreover, the random escalation baseline is matched by escalation rate rather than inference cost, despite LEASH incurring higher costs at comparable escalation rates (Table 1). These limitations make it difficult to assess the practical advantages of the proposed method.

Response: Thanks for suggesting the constructive experiments. Comparison with additional advanced baselines. We add StepWise[1] and Progress Advantage[2] on EvoCUA-8B, using Opus as the strong executor in table 2.

Table 2. Comparison with published baselines on EvoCUA-8B.

MethodEsc. (%)Success (%)Cost ($/task)
StepWise49.658.400.2824
Progress Advantage46.962.980.3533
LEASH45.467.180.3627

For Progress Advantage, we keep the LEASH workflow unchanged and replace only the trigger signal. LEASH achieves higher success at comparable inference cost, supporting the effectiveness of its hidden-state uncertainty signal. StepWise uses its own trained monitor and escalation procedure, whereas LEASH requires no dedicated monitor training. LEASH also outperforms StepWise.

As shown in Fig. 1, LEASH outperforms random routing by 9.1 percentage points at matched inference cost on Qwen3.5. Its advantage over random routing therefore cannot be explained solely by a larger inference budget. These comparisons support a competitive accuracy–cost trade-off beyond the training-free property alone.

We thank the reviewer for pointing this out and will include these comparisons in the refined submission.


References

[1]Step-level optimization for efficient computer-use agents. [2]Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents.

AuthorsOpenReview ↗

Follow-up to Reviewer rBez

Dear Reviewer rBez,

Thank you again for your thoughtful review and constructive suggestions. As the discussion period draws to a close, we would like to briefly summarize the additional experiments and clarifications addressing your concerns.

(1) Effectiveness of post-handoff Shadow monitoring. We introduced a random-Shadow baseline with matched intervention counts to evaluate uncertainty-guided checkpoint timing (Appendix B.5). We also clarified the Shadow subset analysis (Table 5), showing that Shadow-triggered tasks remain challenging across independent runs.

(2) Comparison with stronger baselines. We compared against StepWise, which uses a trained monitor and its own escalation procedure, and Progress Advantage, for which we replace only the trigger signal while keeping the LEASH workflow unchanged (Section 4.2).

(3) Cost-controlled comparisons. We clarified the equal-cost comparison with random routing (Figure 1) and added end-to-end comparisons with token-probability routing and stall detection at similar escalation rates (Appendix B.4).

We hope these additional experiments and clarifications have addressed your main concerns. Do you have any remaining questions or concerns? We would be happy to provide further clarification during the remaining discussion period.

Best regards,
The Authors

I appreciate the authors' efforts in addressing most of my questions during the rebuttal. While the engineering execution is solid, the conceptual novelty remains somewhat limited for a higher rating. Therefore, I will maintain my score.

AuthorsOpenReview ↗

Clarification on LEASH's Novelty and Contributions

Dear Reviewer rBez,

Thank you for the follow-up, and for confirming that the rebuttal addressed most of your questions and that the engineering execution is solid. We respond below to the remaining concern about conceptual novelty.

  1. LEASH's contribution is a new method, lightweight by design. We respectfully see LEASH as a methodological contribution rather than an engineering one. Its core idea is that a single uncertainty signal, read from the weak agent's own hidden states, decides which tasks to route up front, when to hand off, and when the strong agent should reconsider after handoff; to our knowledge, no prior escalation method for computer-use agents does this. The closed-form, training-free design is a requirement, not a shortcut: a signal checked at every step must add almost no cost, so during weak-agent execution LEASH reuses activations already produced for action generation. With it, LEASH with Qwen3.5-35B-A3B matches always-on Claude Opus 5 (73.66% vs. 72.90%) at about 19% lower cost. All three reviewers recognized this practical value (4Q8b: "simple and practical"; aYg1: "sufficient evidence for the practical value of the framework"). It is also the kind of contribution that the ICLR reviewer guide asks reviewers to credit: "new, relevant, impactful knowledge", including knowledge for practitioners.

  2. Nevertheless, LEASH is also novel in three respects, to our knowledge.

    1. No monitor training. Step-level methods such as StepWise train a dedicated monitor, which requires annotation and a training phase. LEASH only needs closed-form statistics from a small set of reference trajectories, and it still stays ahead of a cost-matched StepWise-Opus mixture success rate (67.18% vs. 63.5%).
    2. Adopting distance-based uncertainty signal inside a multi-agent escalation loop. Distance-based scores are well established for single-input OOD detection, but not as a step-level handoff signal between agents. The choice of uncertainty signal also matters, i.e., within the same workflow, replacing our score with token-probability confidence or with Progress Advantage reduces the task success (See Table 2 in our initial rebuttal). This highlights the value of our uncertainty signal for agent escalation.
    3. Oversight after handoff. Most existing routing methods, including Adaptive VLM Routing, focus on model selection rather than continued oversight after handoff. However, our LEASH keeps the white-box weak agent as an observer and uses its hidden states to decide when the black-box strong agent should reconsider. Uncertainty-triggered checkpoints outperform random ones at the same intervention count (73.66% vs. 70.23%).

We would be grateful if you could clarify which aspects of our contributions you find insufficiently novel. This would help us better understand your assessment and clarify our contributions. If you think a specific prior work already covers these points, we would be glad to discuss it and add the comparison to the final version. Thank you again for your time and constructive feedback.

Best regards,

Authors

Reviewer_4Q8bOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes a way for a small agent to decide when to hand a computer-use task over to a stronger one, using its own internal signals instead of a trained monitor. It also lets the small model keep watching the strong one after handoff and nudge it when things look off. On OSWorld this beats random handoff at the same budget and gets close to always using the strong model at lower cost

Critical strengths

The idea is simple and practical. It needs no extra training and reuses what the small model already computes. The comparison to random handoff at matched budget is a fair test, and the authors are open about the limitations

Critical weaknesses

The comparison is mainly against random handoff, so it's hard to tell how much the specific signal matters versus simply handing off more tasks. Comparing against other simple signals, such as the model's own confidence, would make the case stronger. The benefit of the post-handoff monitoring is also not fully separated from the other changes, as the authors acknowledge. A few repeated runs would help the claim that LEASH matches always-on opus, since the gap between them is small

Questions for authors
  1. How does LEASH compare with simpler handoff signals at the same escalation rate?
  2. Can you separate the effect of post-handoff monitoring more cleanly, for example by fixing which tasks are handed off?
4 comments
AuthorsOpenReview ↗

Responses to Reviewer 4Q8b comments (Part 1)

We thank the reviewer for the careful and constructive feedback. We address the concerns below with additional experiments and analyses.

W1: The comparison is mainly against random handoff, so it’s hard to tell how much the specific signal matters versus simply handing off more tasks. Comparing against other simple signals, such as the model’s own confidence, would make the case stronger. The benefit of the post-handoff monitoring is also not fully separated from the other changes, as the authors acknowledge. A few repeated runs would help the claim that LEASH matches always-on opus, since the gap between them is small

Q1: How does LEASH compare with simpler handoff signals at the same escalation rate?

Response: (a) Comparison with simple handoff signals. We compare LEASH with token-probability routing and stall detection in table 1. LEASH achieves highest success among the baselines. On Qwen3.5, we raised LEASH’s threshold to match the token-probability gate’s escalation rate.

Table 1. Handoff signals within the same workflow on Qwen3.5.

GateEsc. (%)Success (%)Cost ($/task)
LEASH (higher threshold)44.665.500.3208
Token probability44.662.790.3057
Stall rule42.661.630.3013

The improvement is achieved without escalating more tasks. This supports the value of the hidden-state signal for guiding handoff.

(b) Compare baselines. We further compare LEASH against more competative baseline methods, StepWise[1] and Progress Advantage[2], on EvoCUA-8B, as shown in table 2.

Table 2. Comparison with published baselines on EvoCUA-8B.

MethodEsc. (%)Success (%)Cost ($/task)
StepWise49.658.400.2824
Progress Advantage46.962.980.3533
LEASH45.467.180.3627

For Progress Advantage, we keep the LEASH workflow unchanged and replace only the trigger signal, directly comparing the signals at comparable cost. StepWise uses its own trained monitor and escalation procedure, whereas LEASH requires no dedicated monitor training. LEASH also outperforms a cost-matched StepWise-based mixture (67.18% vs. 63.5%).

(c) Reliability of the Opus comparison. Table 3 reports task-matched success differences and cost savings with 95% paired-bootstrap confidence intervals. LEASH consistently reduces cost. The cost advantage persists even when all nine missing Opus costs are set to $0.

Table 3. Paired comparisons with always-on Opus.

BackboneΔ Success (pp)Cost saving (%)Conservative saving (%)
Qwen3.5+0.76 [−3.8, +5.3]18.8 [10.8, 26.3]16.3 [7.7, 24.0]
Qwen3.8-27B−3.05 [−8.0, +1.9]29.0 [21.5, 36.3]26.4 [18.1, 34.1]

Notes: Brackets show 95% paired task-bootstrap confidence intervals. Δ Success = LEASH − Opus, in percentage points. We calculate cost savings on the 253 tasks with complete logs for both methods. For the conservative analysis, we include all 262 tasks and assign $0 to the nine missing Opus costs.


References

[1]Step-level optimization for efficient computer-use agents. [2]Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents.

AuthorsOpenReview ↗

Responses to Reviewer 4Q8b comments (Part 2)

Q2: Can you separate the effect of post-handoff monitoring more cleanly, for example by fixing which tasks are handed off?

Response: (a) Task-matched comparison. In Table 4 of the submission, we compare the 131 tasks escalated by both configurations, where Shadow achieves 69.5% success versus 67.9% without Shadow. This comparison matches task IDs but not handoff states, as the runs were executed independently.

Table 4 further shows that always-on Opus performs worse on Shadow-triggered tasks than on escalated, untriggered tasks (51.6% vs. 74.5%), suggesting that Shadow identifies cases that remain difficult even for the strong agent.

(b) Shadow intervention timing. We additionally compare against random Shadow, which selects checkpoint times randomly while matching LEASH’s number of interventions in table 4.

Table 4. Shadow intervention strategies on Qwen3.5.

ConfigurationSuccess (%)
No Shadow67.94
Random Shadow (matched intervention count)70.23
LEASH (uncertainty-triggered Shadow)73.66

LEASH achieves higher success with the same intervention count, providing additional evidence consistent with the value of uncertainty-guided checkpoint selection.

We thank the reviewer for pointing this out and will include these experiments above in the refined submission.

Thanks for the extra experiments! They addressed both my questions, though random Shadow control fixes the number of interventions rather than which tasks get handed off. That said, no repeated runs or CIs are reported for these, so the 2.7 point gap over token probability is ~7 tasks, and the shadow gain over random is ~9. Paired CIs like those reported for the opus comparison would make them more convincing in the final version. I'll keep my positive score

AuthorsOpenReview ↗

Thanks for the positive feedback

Dear Reviewer 4Q8b,

Thank you for your constructive suggestions, which have helped us improve our work significantly. We will also consider your suggestion regarding paired confidence intervals in the final revision. Your positive feedback is very encouraging to us.

Best regards, The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Global Response

We sincerely thank all reviewers for their valuable and constructive feedback. We conducted additional experiments and analyses to address the main concerns and revised our manuscript (10-page main text, with major revisions highlighted in blue). Below, we summarize the strengths recognized by the reviewers and our responses to the shared concerns.


Strengths highlighted by reviewers

  1. An important and practical problem in efficient computer-use agents (rBez, 4Q8b).
  2. A simple, training-free framework for uncertainty-guided escalation and oversight (aYg1, 4Q8b).
  3. Promising empirical results across multiple backbones with favorable accuracy–cost trade-offs (aYg1, rBez, 4Q8b).

Summary of our responses to the main concerns

(1) Comparison with stronger baselines and effectiveness of the uncertainty signal (aYg1, rBez, 4Q8b).

We compared against StepWise, which uses a trained monitor and its own escalation procedure, and Progress Advantage, for which we replace only the trigger signal while keeping the LEASH workflow unchanged (Section 4.2). We also added end-to-end comparisons with token-probability routing and stall detection at similar escalation rates (Appendix B.4).

(2) Effectiveness of post-handoff Shadow monitoring (aYg1, rBez, 4Q8b).

We introduced a random-Shadow baseline with matched intervention counts to evaluate uncertainty-guided checkpoint timing (Appendix B.5). We also clarified the interpretation of the Shadow subset analysis (Table 5), showing that Shadow-triggered tasks remain challenging in independent different runs.

(3) Statistical reliability and inference cost (aYg1, 4Q8b).

We added 95% paired task-bootstrap confidence intervals for success differences and cost savings (Appendix B.6), showing consistent cost savings across four backbones. We also conducted a conservative sensitivity analysis in the rebuttal, confirming that the cost advantage persists even when the nine missing Opus costs are set to zero.

(4) Theoretical analysis and presentation.

We added a theoretical justification for uncertainty-based initial escalation under stated assumptions (Appendix A.1–A.2). We also clarified the experimental comparisons and improved the readability of the case studies in the revised manuscript.


Remark

We believe these additional experiments and revisions strengthen the empirical evidence and clarify the scope of our contributions. We appreciate the reviewers' suggestions and welcome further discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.