- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes LEASH, a framework that lets a weak computer-use agent hand over tasks to a stronger model when needed. It compares the weak model’s hidden states with successful and failed reference trajectories using a relative Mahalanobis distance score. This score guides both initial routing and runtime handoff. After handoff, the weak model acts as a shadow observer and can ask the strong model to reconsider its actions. The method requires no backbone fine-tuning and reuses existing hidden states during weak-agent execution. Experiments on OSWorld with four weak backbones show improvements over standalone weak agents and random routing. With Qwen3.5-35B-A3B, LEASH reports 73.66% success at an estimated 19.2% lower inference cost than always-on Opus, which achieves 72.90%.
Critical strengths
-
LEASH offers a practical way to combine initial routing, runtime handoff, and post-handoff monitoring using the weak model’s hidden states (Sections 3.2–3.4). Runtime scoring reuses activations from action generation without an extra forward pass, making it useful for long computer-use tasks where monitoring costs matter.
-
The results across four weak backbones show promising gains in task success (Table 1). In particular, the Qwen3.5 configuration achieves similar observed success to always-on Opus at 19.2% lower estimated inference cost. The equal-cost comparison in Figure 1 also supports the practical value of selective escalation.
Critical weaknesses
-
The experiments do not isolate the benefit of Shadow monitoring, which is a main contribution. The Shadow and No-Shadow runs escalate different tasks, as acknowledged in Appendix B.5. Table 4 also shows higher success on tasks where no checkpoint was triggered. Thus, the 5.72-point gain in Table 3 cannot be attributed to Shadow intervention alone. Comparisons from the same handoff states, including random or fixed-interval checkpoints at matched cost, are needed to establish whether the uncertainty signal selects useful intervention times.
-
The evidence for the proposed escalation signal is limited by the baselines. Table 1 matches escalation rates, but LEASH costs more than random routing for every backbone. Figure 1 partly addresses this with an equal-cost comparison for Qwen3.5, but comparisons with simple stall-detection rules or token-probability routing are missing. Table 2 compares uncertainty scores only through AUROC, not downstream task success. This leaves unclear how much of the gain comes from the proposed signal rather than the overall workflow and additional computation.
Questions for authors
-
Why do the Shadow and No-Shadow runs escalate different tasks if Shadow only acts after handoff (Appendix B.5)? Were the escalation settings identical, with differences caused by independent runs? Does the gain remain when both methods start from the same handoff states?
-
Does uncertainty-triggered intervention outperform random or fixed-interval checkpoints with the same intervention budget and reasoning effort (Section 3.4, Tables 3–4)? This would help separate the value of the uncertainty signal from the benefit of additional reflection.
-
How does LEASH compare with token-probability routing or a simple stall-detection rule at matched inference cost (Tables 1–2)? These comparisons would clarify whether the proposed hidden-state score improves the final cost–success trade-off.
-
Can you provide paired confidence intervals for the success differences and compare costs on the same task IDs, given the nine missing Opus token logs (Appendix B.1)? This would help assess the reliability of the reported accuracy and cost advantages.
5 comments
Responses to Reviewer aYg1 comments (Part 2)
W2: The evidence for the proposed escalation signal is limited by the baselines. Table 1 matches escalation rates, but LEASH costs more than random routing for every backbone. Figure 1 partly addresses this with an equal-cost comparison for Qwen3.5, but comparisons with simple stall-detection rules or token-probability routing are missing. Table 2 compares uncertainty scores only through AUROC, not downstream task success. This leaves unclear how much of the gain comes from the proposed signal rather than the overall workflow and additional computation.
Q3: How does LEASH compare with token-probability routing or a simple stall-detection rule at matched inference cost (Tables 1–2)? These comparisons would clarify whether the proposed hidden-state score improves the final cost–success trade-off.
Response: (a) Cost performance trade-off. As you pointed out, LEASH costs 9–26% more than random routing at the same escalation rate, because the gate escalates harder and longer tasks and escalated tasks carry the weak agent’s steps before handoff, which is expected outcome by design. We would like to amphasize that, at equal cost, as in Fig. 1, LEASH’s performance gain is +9.1 points on Qwen3.5 compare to that very random routing baseline, implying that LEASH consume the computation budget more effectively to turn the extra computation with the strong model into actual task success rate gain.
(b) Comparison with different baselines. Thanks for suggesting the constructive experiments. In this rebuttal, we compare LEASH with simple handoff signals and different baselines using end-to-end task success and inference cost, rather than AUROC alone.. Each one either holds the workflow fixed and changes only the trigger signal, or holds cost fixed.
(b-1) Comparison with simple token-probability and stall-detection baselines. By reflecting your suggestion, we compare both gates within the same LEASH workflow, changing only the trigger signal. We also increase LEASH’s threshold to match the token-probability gate’s escalation rate.
Table 2: Comparison with token-probability and stall-detection gates within the same LEASH workflow.
| Gate | Esc. (%) | Success (%) | Cost ($/task) |
|---|---|---|---|
| LEASH (higher threshold) | 44.6 | 65.50 | 0.3208 |
| Token probability | 44.6 | 62.79 | 0.3057 |
| Stall rule | 42.6 | 61.63 | 0.3013 |
LEASH retains higher observed success than token-probability routing at the same escalation rate, and than stall detection at a similar rate.
(b-2) Comparison with additional advanced baselines. We further compare LEASH against more competative baseline methods, StepWise[1] and Progress Advantage[2], on EvoCUA-8B, as shown in Table 3.
Table 3: Comparison with existing escalation and uncertainty estimation methods on EvoCUA-8B.
| Method | Esc. (%) | Success (%) ↑ | Cost ($/task) ↓ |
|---|---|---|---|
| EvoCUA-8B (no escalation) | 0.0 | 42.37 | – |
| StepWise | 49.6 | 58.40 | 0.2824 |
| Progress Advantage | 46.9 | 62.98 | 0.3533 |
| LEASH | 45.4 | 67.18 | 0.3627 |
For Progress Advantage, we keep the LEASH workflow unchanged and replace only the trigger signal to the implicit process reward signal proposed in [2]. LEASH achieves higher success, supporting the effectiveness of our hidden-state uncertainty signal.
StepWise, in contrast, is a complete escalation method with its own trained monitor and escalation procedure. Our LEASH outperforms StepWise despite using a training-free approach, and retains its advantage under an equal-cost comparison with a StepWise–Opus mixture.
References
[1]Step-level optimization for efficient computer-use agents. [2]Neglected Free Lunch from Post-training: Progress Advantage for LLM Agents.
Responses to Reviewer aYg1 comments (Part 3)
Q4: Can you provide paired confidence intervals for the success differences and compare costs on the same task IDs, given the nine missing Opus token logs (Appendix B.1)? This would help assess the reliability of the reported accuracy and cost advantages.
Response: In this rebuttal, table 4 here reports task-matched success differences and cost savings with 95% paired-bootstrap confidence intervals. LEASH consistently reduces cost, even when all nine missing Opus costs are set to $0.
Table 4. Paired comparisons with always-on Opus.
| Backbone | Δ Success (pp) | Cost saving (%) | Conservative saving (%) |
|---|---|---|---|
| Qwen3.5 | +0.76 [−3.8, +5.3] | 18.8 [10.8, 26.3] | 16.3 [7.7, 24.0] |
| Qwen3.8-27B | −3.05 [−8.0, +1.9] | 29.0 [21.5, 36.3] | 26.4 [18.1, 34.1] |
Notes: Brackets show 95% paired task-bootstrap confidence intervals. Δ Success = LEASH − Opus, in percentage points. We calculate cost savings on the 253 tasks with complete logs for both methods. For the conservative analysis, we include all 262 tasks and assign $0 to the nine missing Opus costs.
We thank the reviewer for pointing this out and will include these experiments above in the refined submission.
Responses to Reviewer aYg1 comments (Part 1)
We thank the reviewer for the careful and constructive feedback. We address the concerns below with additional experiments and analyses.
Response: (a) Explaination of Table 4. The higher success on the 137 untriggered tasks (74.5% vs. 69.3%) may reflect differences between the independently executed Shadow and No-Shadow runs, rather than checkpoint interventions. Importantly, for the task subsets defined by the Shadow run, both No Shadow (48.4% vs. 69.3%) and always-on Opus (51.6% vs. 74.5%) perform substantially worse on triggered than untriggered tasks. This suggests that Shadow's uncertainty signal tends to flag tasks that are difficult even for the strong agent. We will clarify this point in the revised manuscript to make this clear.
(b)Why the escalation sets differ. We used identical escalation settings for the Shadow and No-Shadow runs but executed them independently. We found that variations in the Opus Manager’s subgoal decompositions and screen observations affected escalation decisions by changing the weak agent’s inputs and uncertainty scores. These differences arose before handoff; Shadow operates only after handoff and does not affect pre-handoff routing.
(c)Shadow intervention timing. To better assess Shadow’s contribution, we introduce a random-Shadow baseline with matched intervention counts.
Table 1: Comparison of Shadow intervention strategies.
LEASH achieves 3.4 percentage points higher success than random Shadow with the same number of interventions. This provides additional evidence consistent with the value of uncertainty-guided checkpoint selection, rather than simply intervening more often. We also revised Figures 8–10 to better illustrate the execution states at handoff and how strong-agent takeover recovers stalled trajectories.