- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes OctoNest, a hierarchical stateful control framework for cross-device computer-use agents, aiming to improve task execution and failure recovery across heterogeneous devices. The framework employs two levels of control: a global Orchestrator manages task planning, device assignment, and cross-device replanning, while local Device Agents iteratively refine subtasks, adjust execution instructions, and switch among API, CLI, and GUI modalities based on execution feedback. When local recovery is no longer feasible, failures are escalated to the Orchestrator for further replanning. The authors also introduce CAPEBench, a benchmark comprising 158 instances with modality-level, device-level, and mixed perturbations. Experiments demonstrate improved task success rates and lower token cost per perfect pass compared with existing baselines.
Critical strengths
(1)Well-motivated and effective hierarchical failure recovery architecture.The paper separates intra-device modality switching from inter-device task reassignment into two levels of control. This design allows Device Agents to iteratively adapt their execution strategies based on local feedback, while the Orchestrator handles global task coordination and replanning. The proposed architecture is practically meaningful, and the ablation studies show substantial performance degradation when the Device Agent is removed or global replanning is disabled, supporting the effectiveness of the hierarchical control design. (2) A diagnostically useful benchmark with comprehensive experimental evaluation. The proposed CAPEBench introduces controlled modality-level, device-level, and mixed perturbations, enabling a more fine-grained evaluation of failure recovery capabilities at different control levels. The experimental evaluation includes comparisons with multiple baselines, architectural ablations, recovery behavior analyses, and evaluations across different model backbones. On CAPEBench, OctoNest improves Perfect Pass by 18.35 percentage points over the strongest baseline while reducing the amortized token cost per perfect pass by 39.8%, providing convincing empirical evidence of its practical effectiveness.
Critical weaknesses
(1)Limited methodological novelty beyond existing hierarchical agent architectures. Although the paper highlights the architectural differences between OctoNest and existing approaches in Figure 1 and Appendix C.3, its core mechanisms largely build upon established techniques, including hierarchical task orchestration, local state maintenance, and LLM-based feedback-driven execution. In particular, existing systems such as UFO³ already support device-local state management, iterative execution, and error recovery. OctoNest's main contribution appears to lie in explicitly separating intra-device modality recovery from inter-device task reassignment, rather than introducing a fundamentally new recovery decision algorithm. Furthermore, the UFO³-Hybrid baseline does not incorporate iterative device-local modality control comparable to that of OctoNest. Consequently, the current comparisons do not sufficiently establish the distinctive technical contributions of the proposed design beyond existing local feedback-driven recovery mechanisms. (2) Insufficient specification and empirical isolation of the core recovery decision mechanism.OctoNest primarily relies on LLM-based Device Agents to interpret execution feedback and decide whether to retry, switch modalities, or escalate to the Orchestrator. However, the paper provides limited details on the specific criteria underlying these decisions. Although the architectural ablations and recovery behavior analyses demonstrate the effectiveness of the overall framework, the existing ablations simultaneously modify several factors, including local task refinement, iterative retries, and modality switching, making it difficult to isolate their individual contributions. While Figure 5 reports recovery-boundary matching and recovery success, a systematic evaluation of decision accuracy and inappropriate switching behavior across different failure conditions is still lacking. As a result, the current evidence does not sufficiently establish to what extent the observed performance gains stem from the proposed separation of modality-level and device-level recovery decisions, rather than from additional local LLM reasoning and iterative execution.
Questions for authors
(1)Substantive novelty beyond existing hierarchical agent architectures.Prior work, such as UFO³, already supports device-local state maintenance, LLM-based iterative execution, and error recovery. Although Figure 1 and Appendix C.3 highlight the architectural differences between OctoNest and existing approaches, its core mechanisms still overlap with established hierarchical feedback-driven control frameworks. Could the authors further clarify what distinctive technical mechanisms or capabilities OctoNest introduces beyond explicitly separating intra-device modality recovery from inter-device task reassignment? (2)Recovery decision mechanisms and their independent contributions.OctoNest relies on LLM-based Device Agents to interpret execution feedback and decide whether to retry, switch modalities, or escalate to the Orchestrator. However, the specific criteria guiding these decisions remain insufficiently specified, and the existing ablations do not fully disentangle the contributions of iterative local reasoning, task refinement, and hierarchical recovery decisions. Could the authors clarify how Device Agents assess modality-level and device-level execution feasibility, and provide more direct empirical evidence to establish how much of the observed performance improvement can be attributed to the hierarchical recovery mechanism itself, rather than additional local LLM reasoning and iterative attempts? (3)Token overhead associated with unsuccessful recovery attempts.The paper reports average token consumption per episode and amortized token cost per perfect pass. However, the improvement in the latter primarily stems from higher task success rates rather than reduced token consumption per episode. Given that OctoNest introduces additional local LLM reasoning and iterative recovery attempts, could the authors provide a breakdown of token consumption for successful versus unsuccessful episodes, as well as the costs incurred by the Orchestrator, Device Agents, and Modality Executors during recovery? Such an analysis would help clarify whether local recovery effectively reduces global replanning overhead and whether unsuccessful episodes incur substantial additional costs from unproductive recovery attempts.
4 comments
Response to Q2: Recovery Decisions and the Benefits of Role Separation
Q2. How are recovery decisions made, and what evidence supports architectural benefits beyond additional local reasoning and retries?
(1) Device-local recovery decisions
Based on the unfinished subtask, the Device Agent assesses whether the current modality still offers a feasible execution path and whether other modalities on the same device can continue the task. We provide general heuristic guidance in the prompt, allowing the model to make this judgment using device capabilities, permissions, execution feedback, previous attempts, and the remaining budget, rather than writing rules for individual CAPEBench perturbation variants. For example, a correctable omission in an instruction can be addressed by revising the instruction and retrying; if an API session is invalid but a GUI path remains available, the agent can switch modalities on the same device; if feedback indicates a network outage and the remaining task requires network access, the agent escalates the failure evidence to the Orchestrator, which decides subsequent device assignments.
To examine actual decision behavior, we analyzed the 33 paired instances in Figure 5 of the main text in which both systems encountered the same failure. Table 1 summarizes the actions taken by OctoNest's Device Agent after receiving the first failure feedback, with each instance counted once.
Table 1. Device-local decisions after the first failure feedback (33 instances).
| Perturbation level | Instances | Retry same modality | Intra-device switching | Escalate immediately |
|---|---|---|---|---|
| Modality-level | 13 | 0 | 13 | 0 |
| Device-level | 20 | 0 | 6 | 14 |
All 13 instances with modality-level perturbations first attempted another modality on the original device, and 12/13 recovered the affected step. Of the 20 instances with device-level perturbations, 14 escalated immediately after the first failure. The remaining 6 first tried another modality; all escalated immediately after the second failure and subsequently executed on the corresponding peer, with 5/6 recovering the affected step. In 5 of these 6 instances, the first feedback indicated an invalid service session, which alone was insufficient to rule out other local channels. This analysis is consistent with the mechanism described in the main text for distinguishing modality and device feasibility based on execution evidence. In this set of device-level instances, each instance involved at most one additional local attempt before escalation.
(2) Architectural benefits under a controlled inference budget
To examine whether the benefit of assigning device-local coordination and modality-specific execution to separate roles persists under a constrained inference budget, we use Unified Device Agent as the comparator. This variant retains the same global Orchestrator and allows multi-step execution within a device, but uses the same agent to select the modality and perform operations. We compare the two methods on the paper's fixed 82-instance analysis sample, under both the submitted model configuration and a configuration in which all execution roles use Kimi K2.6. In each comparison, Unified's actual token consumption on each instance serves as the budget cap for OctoNest on the corresponding instance.
Table 2. Per-instance token budget comparison under the submitted and matched-model configurations. OctoNest is truncated and scored using Unified's token consumption on each instance as its budget; Δ denotes OctoNest minus Unified.
| Model configuration | Method | Comp. (%) | Adh. (%) | PP (%) | Total tokens (M) |
|---|---|---|---|---|---|
| Submitted | OctoNest | 58.94 | 66.71 | 29.27 (24/82) | 38.71 |
| Unified Device Agent | 51.30 | 55.09 | 18.29 (15/82) | 53.99 | |
| Δ | +7.64 | +11.63 | +10.98 | −15.27 (−28.3%) | |
| All roles: Kimi K2.6 | OctoNest | 52.40 | 56.92 | 21.95 (18/82) | 51.51 |
| Unified Device Agent | 47.40 | 50.32 | 7.32 (6/82) | 76.23 | |
| Δ | +5.00 | +6.60 | +14.63 | −24.72 (−32.4%) |
Under the submitted model configuration, OctoNest exceeds Unified in Task Completion, Instruction Adherence, and PP by 7.64, 11.63, and 10.98 percentage points, respectively. After setting all execution roles to use Kimi K2.6, the three metrics remain higher by 5.00, 6.60, and 14.63 percentage points. After per-instance budget truncation, the total execution tokens retained by OctoNest are 28.3% and 32.4% lower than Unified's under the two configurations, respectively. These results show that OctoNest's advantage persists under per-instance token budget constraints and after further controlling for differences in the underlying models, consistent with the main text's conclusion that assigning device-local coordination and modality-specific execution to separate roles is beneficial.
Response to Q3 (1/2): Token Costs by Task Outcome
Q3 (Part 1/2). How do token costs compare by task outcome and on instances where both systems achieve PP?
Table 3. Token consumption for PP and non-PP instances.
We break down execution token costs by task outcome and recovery stage. Table 3 uses the same set of 158 instances, each fully run by both systems. Success is defined as achieving Perfect Pass (PP), while non-PP includes partially completed instances. Token counts include execution-model inputs and outputs and exclude evaluation-model costs; the unit is thousands of tokens (k).
| System | PP instances | PP mean / median | Non-PP instances | Non-PP mean / median | Overall mean |
|---|---|---|---|---|---|
| OctoNest | 70 | 684.4 / 618.9 | 88 | 828.3 / 532.9 | 764.6 |
| UFO³-Hybrid | 40 | 638.7 / 533.1 | 118 | 724.6 / 669.6 | 702.9 |
OctoNest's mean cost in the non-PP group is 21.0% higher than in the PP group, but its median is lower, indicating that the non-PP group contains high-cost cases. Across the full set, OctoNest uses an average of 764.6k tokens per instance, compared with 702.9k for UFO³-Hybrid, an increase of 8.8%; the numbers of PP instances are 70 and 40, respectively, corresponding to a success-rate difference of 18.99 percentage points. Dividing all execution tokens by the number of PP instances yields a 37.8% reduction in amortized token cost per PP. This analysis is consistent with the main text's conclusion that a higher success rate improves amortized token cost per PP.
Table 4. Token consumption on the 25 instances where both systems achieve PP. Values are in k tokens.
To compare execution costs on the same instances with the same successful outcome, we further analyze the 25 instances on which both systems achieve PP. OctoNest's mean token consumption is 576.3k, compared with 656.2k for UFO³-Hybrid, a reduction of 12.2%; the medians are 499.9k and 566.9k, respectively.
| System | Shared PP instances | Mean (k) | Median (k) |
|---|---|---|---|
| OctoNest | 25 | 576.3 | 499.9 |
| UFO³-Hybrid | 25 | 656.2 | 566.9 |
This paired comparison also helps interpret the PP-group means in Table 3: OctoNest and UFO³-Hybrid achieve PP on 70 and 40 instances, respectively, so their success-group means reflect different sample compositions. Together, the full-set and paired results show that OctoNest achieves perfect completion on more instances while using fewer tokens on average on the same instances that both systems complete perfectly. These results provide further evidence of OctoNest's token efficiency in achieving Perfect Pass, consistent with the token-efficiency findings in the main text.
Response to Q3 (2/2): Recovery Costs by Component and Stage
Q3 (Part 2/2). What are the component costs during recovery, and how do recovery costs relate to full-task costs?
Token counts include execution-model inputs and outputs and exclude evaluation-model costs; the unit is thousands of tokens (k).
Table 5. Component token costs within the recovery window.
We use the 33 paired instances from Figure 5 of the main text in which both systems encountered the same failure. The recovery window starts when the first feedback for that failure is received and ends when the affected step is recovered; if it is not recovered, the window continues until the task ends. Table 5 reports mean costs per window in k tokens. The control-layer subtotal is the sum of the Orchestrator and Device Agent costs and is already included in the window total. Individual entries and subtotals are calculated using unrounded values and then rounded separately to one decimal place.
| System | Orchestrator | Device Agent | Control subtotal | Modality Executors | Window total | Affected-step recovery |
|---|---|---|---|---|---|---|
| OctoNest | 10.3 | 26.7 | 36.9 | 248.4 | 285.3 | 27/33 |
| UFO³-Hybrid | 153.3 | — | 153.3 | 274.3 | 427.6 | 16/33 |
Even after including all repeated decisions by the Device Agent, OctoNest's control-layer cost per recovery window remains 75.9% lower, with a lower cost in 27 of the 33 pairs. Including the Modality Executors, the total window cost is 33.3% lower, while the numbers of instances with the affected step recovered are 27/33 versus 16/33. This paired analysis is consistent with the main text's conclusion that combining local recovery with cross-device orchestration is beneficial, and provides evidence of lower control costs within the recovery window. UFO³-Hybrid's global graph editing also handles routine execution feedback, so the control-layer cost here measures all controller calls within the window.
Table 6. Token costs by stage for paired instances.
Table 6 uses all 33 paired instances from Table 5 and separately reports the 15 instances in which both systems recovered the affected step. Values are mean k tokens per instance. The pre-window stage includes the execution attempt that first triggered the failure; when recovery does not occur, the window continues until the task ends. Differences are calculated as OctoNest minus UFO³-Hybrid.
| Paired sample | System | Before window | Within window | After recovery | Full task |
|---|---|---|---|---|---|
| All 33 pairs | OctoNest | 166.8 | 285.3 | 326.3 | 778.4 |
| All 33 pairs | UFO³-Hybrid | 137.8 | 427.6 | 190.6 | 756.0 |
| All 33 pairs | Difference | +28.9 | −142.2 | +135.7 | +22.4 |
| Both recovered: 15 pairs | OctoNest | 122.4 | 209.2 | 399.7 | 731.3 |
| Both recovered: 15 pairs | UFO³-Hybrid | 87.7 | 280.2 | 393.3 | 761.3 |
| Both recovered: 15 pairs | Difference | +34.6 | −71.0 | +6.4 | −30.0 |
Across all 33 paired instances, OctoNest uses an average of 142.2k fewer tokens within the recovery window, but 28.9k more before the window and 135.7k more after recovery, resulting in a net increase of 22.4k for the full task. The main offset occurs in the post-recovery execution stage, consistent with OctoNest recovering more affected steps (27/33 versus 16/33) and therefore allowing more instances to proceed with subsequent tasks. Restricting the comparison to the same 15 instances in which both systems recovered, OctoNest still uses an average of 71.0k fewer tokens within the recovery window, while the post-recovery difference shrinks to 6.4k and the average full-task cost is 30.0k lower. This analysis explains how a lower recovery-window cost can coexist with a slightly higher full-task cost, and is consistent with the main text's conclusions of a higher recovery success rate and lower amortized token cost per PP.
Response to Q1: Explicit Device-Local State and Modality Control
Thank you for recognizing the motivation and practical effectiveness of hierarchical recovery, as well as the diagnostic value of CAPEBench. We clarify OctoNest's technical distinctions and recovery decisions, and provide model- and budget-controlled comparisons together with token-cost breakdowns.
Q1. What technical mechanisms and capabilities distinguish OctoNest from existing hierarchical agents beyond separating intra-device and inter-device recovery?
We acknowledge that prior work such as UFO³ supports LLM-based iterative execution and feedback-driven plan revision. UFO³ uses platform-specific execution agents, most of which are bound to a single control modality. OctoNest explicitly maintains device-local state in an independent Device Agent, separating device-local modality control from cross-device planning and modality-specific execution. This state retains prior execution attempts, their outcomes, and interaction context, enabling the Device Agent to progressively refine subtasks, select and combine API, CLI, and GUI modalities, and assess whether execution can continue on the assigned device.
In OctoNest, the Orchestrator assigns device-level subtasks. Using local execution history and intermediate results, the Device Agent progressively refines the remaining goal into instructions suited to the corresponding modality and delegates them to Modality Executors with independent multi-step execution loops. After each executor returns a result, the Device Agent reassesses the subtask's remaining requirements and feasible paths, then decides whether to continue with the current modality, combine it with other modalities, report completion, or escalate evidence of failure. Device assignment and the concrete execution approach can therefore be adjusted separately, and a failure in one modality can be assessed in light of the other capabilities available on that device.
This mechanism also supports progressive subtask refinement and modality composition during normal execution. Modality transitions can be triggered by failures or by the requirements of the next stage after successful execution. For example, in a file version comparison task, the Device Agent first obtains files through the GUI, then invokes the CLI on those files to perform the comparison and generate a report, coordinating different modalities to accomplish the same subtask while keeping the device assignment unchanged. Figure 4 in the main text analyzes modality selection and composition on unperturbed tasks, illustrating this mechanism's role during normal execution.
The architectural ablations in the main text, together with the additional experiments matching models and per-instance token budgets, support the value of assigning device-local coordination and modality-specific execution to separate roles. In particular, when all execution roles use Kimi K2.6 and Unified Device Agent's token consumption on each instance is used as the budget cap, OctoNest's PP remains 14.63 percentage points higher. This result is consistent with the main text's conclusion that separating device-local coordination from modality-specific execution is beneficial, and further shows that this advantage persists with matched models and per-instance budget constraints. In addition, CAPEBench evaluates adaptation to changes in modality and device feasibility through corresponding unperturbed, modality-level, device-level, and mixed conditions, constituting another contribution of the paper.