acceptodds
Under review as a conference paper at ICLR 2027

OctoNest: Adaptive Cross-Device Execution through Stateful Control

Abstract

Computer use agents are expanding from single-device operation toward cross-device systems that coordinate tasks across heterogeneous environments. Execution conditions are often only partially known at planning time and revealed through interaction. Failures may require intra-device modality switching or inter-device reassignment; failing to distinguish these cases can lead to repeated failures or premature termination. However, existing systems primarily scale up single-device agents without sufficiently distinguishing device-level and modality-specific execution conditions. We propose OctoNest, which coordinates stateful cross-device orchestration and iterative device-local modality control. Device Agents refine subtasks and select modalities, while an Orchestrator uses execution feedback to revise plans and device assignments. We also introduce CAPEBench, comprising 158 instances from 23 cross-device seed tasks with controlled perturbations. OctoNest leads all three quality metrics, improving Perfect Pass over the strongest baseline by 18.35 percentage points and reducing token cost per perfect pass by 39.8%. Further analyses support the complementary roles of local refinement and global revision and demonstrate CAPEBench's ability to distinguish control limitations under changing execution conditions.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Cross-Device Coordination, Multi-Agent Collaboration, Computer-Use Agents, LLM-Based Agents, Agent Planning and Execution

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bKrBOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes OctoNest, a hierarchical stateful control framework for cross-device computer-use agents, aiming to improve task execution and failure recovery across heterogeneous devices. The framework employs two levels of control: a global Orchestrator manages task planning, device assignment, and cross-device replanning, while local Device Agents iteratively refine subtasks, adjust execution instructions, and switch among API, CLI, and GUI modalities based on execution feedback. When local recovery is no longer feasible, failures are escalated to the Orchestrator for further replanning. The authors also introduce CAPEBench, a benchmark comprising 158 instances with modality-level, device-level, and mixed perturbations. Experiments demonstrate improved task success rates and lower token cost per perfect pass compared with existing baselines.

Critical strengths

(1)Well-motivated and effective hierarchical failure recovery architecture.The paper separates intra-device modality switching from inter-device task reassignment into two levels of control. This design allows Device Agents to iteratively adapt their execution strategies based on local feedback, while the Orchestrator handles global task coordination and replanning. The proposed architecture is practically meaningful, and the ablation studies show substantial performance degradation when the Device Agent is removed or global replanning is disabled, supporting the effectiveness of the hierarchical control design. (2) A diagnostically useful benchmark with comprehensive experimental evaluation. The proposed CAPEBench introduces controlled modality-level, device-level, and mixed perturbations, enabling a more fine-grained evaluation of failure recovery capabilities at different control levels. The experimental evaluation includes comparisons with multiple baselines, architectural ablations, recovery behavior analyses, and evaluations across different model backbones. On CAPEBench, OctoNest improves Perfect Pass by 18.35 percentage points over the strongest baseline while reducing the amortized token cost per perfect pass by 39.8%, providing convincing empirical evidence of its practical effectiveness.

Critical weaknesses

(1)Limited methodological novelty beyond existing hierarchical agent architectures. Although the paper highlights the architectural differences between OctoNest and existing approaches in Figure 1 and Appendix C.3, its core mechanisms largely build upon established techniques, including hierarchical task orchestration, local state maintenance, and LLM-based feedback-driven execution. In particular, existing systems such as UFO³ already support device-local state management, iterative execution, and error recovery. OctoNest's main contribution appears to lie in explicitly separating intra-device modality recovery from inter-device task reassignment, rather than introducing a fundamentally new recovery decision algorithm. Furthermore, the UFO³-Hybrid baseline does not incorporate iterative device-local modality control comparable to that of OctoNest. Consequently, the current comparisons do not sufficiently establish the distinctive technical contributions of the proposed design beyond existing local feedback-driven recovery mechanisms. (2) Insufficient specification and empirical isolation of the core recovery decision mechanism.OctoNest primarily relies on LLM-based Device Agents to interpret execution feedback and decide whether to retry, switch modalities, or escalate to the Orchestrator. However, the paper provides limited details on the specific criteria underlying these decisions. Although the architectural ablations and recovery behavior analyses demonstrate the effectiveness of the overall framework, the existing ablations simultaneously modify several factors, including local task refinement, iterative retries, and modality switching, making it difficult to isolate their individual contributions. While Figure 5 reports recovery-boundary matching and recovery success, a systematic evaluation of decision accuracy and inappropriate switching behavior across different failure conditions is still lacking. As a result, the current evidence does not sufficiently establish to what extent the observed performance gains stem from the proposed separation of modality-level and device-level recovery decisions, rather than from additional local LLM reasoning and iterative execution.

Questions for authors

(1)Substantive novelty beyond existing hierarchical agent architectures.Prior work, such as UFO³, already supports device-local state maintenance, LLM-based iterative execution, and error recovery. Although Figure 1 and Appendix C.3 highlight the architectural differences between OctoNest and existing approaches, its core mechanisms still overlap with established hierarchical feedback-driven control frameworks. Could the authors further clarify what distinctive technical mechanisms or capabilities OctoNest introduces beyond explicitly separating intra-device modality recovery from inter-device task reassignment? (2)Recovery decision mechanisms and their independent contributions.OctoNest relies on LLM-based Device Agents to interpret execution feedback and decide whether to retry, switch modalities, or escalate to the Orchestrator. However, the specific criteria guiding these decisions remain insufficiently specified, and the existing ablations do not fully disentangle the contributions of iterative local reasoning, task refinement, and hierarchical recovery decisions. Could the authors clarify how Device Agents assess modality-level and device-level execution feasibility, and provide more direct empirical evidence to establish how much of the observed performance improvement can be attributed to the hierarchical recovery mechanism itself, rather than additional local LLM reasoning and iterative attempts? (3)Token overhead associated with unsuccessful recovery attempts.The paper reports average token consumption per episode and amortized token cost per perfect pass. However, the improvement in the latter primarily stems from higher task success rates rather than reduced token consumption per episode. Given that OctoNest introduces additional local LLM reasoning and iterative recovery attempts, could the authors provide a breakdown of token consumption for successful versus unsuccessful episodes, as well as the costs incurred by the Orchestrator, Device Agents, and Modality Executors during recovery? Such an analysis would help clarify whether local recovery effectively reduces global replanning overhead and whether unsuccessful episodes incur substantial additional costs from unproductive recovery attempts.

4 comments
AuthorsOpenReview ↗

Response to Q1: Explicit Device-Local State and Modality Control

Thank you for recognizing the motivation and practical effectiveness of hierarchical recovery, as well as the diagnostic value of CAPEBench. We clarify OctoNest's technical distinctions and recovery decisions, and provide model- and budget-controlled comparisons together with token-cost breakdowns.


Q1. What technical mechanisms and capabilities distinguish OctoNest from existing hierarchical agents beyond separating intra-device and inter-device recovery?

We acknowledge that prior work such as UFO³ supports LLM-based iterative execution and feedback-driven plan revision. UFO³ uses platform-specific execution agents, most of which are bound to a single control modality. OctoNest explicitly maintains device-local state in an independent Device Agent, separating device-local modality control from cross-device planning and modality-specific execution. This state retains prior execution attempts, their outcomes, and interaction context, enabling the Device Agent to progressively refine subtasks, select and combine API, CLI, and GUI modalities, and assess whether execution can continue on the assigned device.

In OctoNest, the Orchestrator assigns device-level subtasks. Using local execution history and intermediate results, the Device Agent progressively refines the remaining goal into instructions suited to the corresponding modality and delegates them to Modality Executors with independent multi-step execution loops. After each executor returns a result, the Device Agent reassesses the subtask's remaining requirements and feasible paths, then decides whether to continue with the current modality, combine it with other modalities, report completion, or escalate evidence of failure. Device assignment and the concrete execution approach can therefore be adjusted separately, and a failure in one modality can be assessed in light of the other capabilities available on that device.

This mechanism also supports progressive subtask refinement and modality composition during normal execution. Modality transitions can be triggered by failures or by the requirements of the next stage after successful execution. For example, in a file version comparison task, the Device Agent first obtains files through the GUI, then invokes the CLI on those files to perform the comparison and generate a report, coordinating different modalities to accomplish the same subtask while keeping the device assignment unchanged. Figure 4 in the main text analyzes modality selection and composition on unperturbed tasks, illustrating this mechanism's role during normal execution.

The architectural ablations in the main text, together with the additional experiments matching models and per-instance token budgets, support the value of assigning device-local coordination and modality-specific execution to separate roles. In particular, when all execution roles use Kimi K2.6 and Unified Device Agent's token consumption on each instance is used as the budget cap, OctoNest's PP remains 14.63 percentage points higher. This result is consistent with the main text's conclusion that separating device-local coordination from modality-specific execution is beneficial, and further shows that this advantage persists with matched models and per-instance budget constraints. In addition, CAPEBench evaluates adaptation to changes in modality and device feasibility through corresponding unperturbed, modality-level, device-level, and mixed conditions, constituting another contribution of the paper.

AuthorsOpenReview ↗

Response to Q2: Recovery Decisions and the Benefits of Role Separation

Q2. How are recovery decisions made, and what evidence supports architectural benefits beyond additional local reasoning and retries?

(1) Device-local recovery decisions

Based on the unfinished subtask, the Device Agent assesses whether the current modality still offers a feasible execution path and whether other modalities on the same device can continue the task. We provide general heuristic guidance in the prompt, allowing the model to make this judgment using device capabilities, permissions, execution feedback, previous attempts, and the remaining budget, rather than writing rules for individual CAPEBench perturbation variants. For example, a correctable omission in an instruction can be addressed by revising the instruction and retrying; if an API session is invalid but a GUI path remains available, the agent can switch modalities on the same device; if feedback indicates a network outage and the remaining task requires network access, the agent escalates the failure evidence to the Orchestrator, which decides subsequent device assignments.

To examine actual decision behavior, we analyzed the 33 paired instances in Figure 5 of the main text in which both systems encountered the same failure. Table 1 summarizes the actions taken by OctoNest's Device Agent after receiving the first failure feedback, with each instance counted once.

Table 1. Device-local decisions after the first failure feedback (33 instances).

Perturbation levelInstancesRetry same modalityIntra-device switchingEscalate immediately
Modality-level130130
Device-level200614

All 13 instances with modality-level perturbations first attempted another modality on the original device, and 12/13 recovered the affected step. Of the 20 instances with device-level perturbations, 14 escalated immediately after the first failure. The remaining 6 first tried another modality; all escalated immediately after the second failure and subsequently executed on the corresponding peer, with 5/6 recovering the affected step. In 5 of these 6 instances, the first feedback indicated an invalid service session, which alone was insufficient to rule out other local channels. This analysis is consistent with the mechanism described in the main text for distinguishing modality and device feasibility based on execution evidence. In this set of device-level instances, each instance involved at most one additional local attempt before escalation.

(2) Architectural benefits under a controlled inference budget

To examine whether the benefit of assigning device-local coordination and modality-specific execution to separate roles persists under a constrained inference budget, we use Unified Device Agent as the comparator. This variant retains the same global Orchestrator and allows multi-step execution within a device, but uses the same agent to select the modality and perform operations. We compare the two methods on the paper's fixed 82-instance analysis sample, under both the submitted model configuration and a configuration in which all execution roles use Kimi K2.6. In each comparison, Unified's actual token consumption on each instance serves as the budget cap for OctoNest on the corresponding instance.

Table 2. Per-instance token budget comparison under the submitted and matched-model configurations. OctoNest is truncated and scored using Unified's token consumption on each instance as its budget; Δ denotes OctoNest minus Unified.

Model configurationMethodComp. (%)Adh. (%)PP (%)Total tokens (M)
SubmittedOctoNest58.9466.7129.27 (24/82)38.71
Unified Device Agent51.3055.0918.29 (15/82)53.99
Δ+7.64+11.63+10.98−15.27 (−28.3%)
All roles: Kimi K2.6OctoNest52.4056.9221.95 (18/82)51.51
Unified Device Agent47.4050.327.32 (6/82)76.23
Δ+5.00+6.60+14.63−24.72 (−32.4%)

Under the submitted model configuration, OctoNest exceeds Unified in Task Completion, Instruction Adherence, and PP by 7.64, 11.63, and 10.98 percentage points, respectively. After setting all execution roles to use Kimi K2.6, the three metrics remain higher by 5.00, 6.60, and 14.63 percentage points. After per-instance budget truncation, the total execution tokens retained by OctoNest are 28.3% and 32.4% lower than Unified's under the two configurations, respectively. These results show that OctoNest's advantage persists under per-instance token budget constraints and after further controlling for differences in the underlying models, consistent with the main text's conclusion that assigning device-local coordination and modality-specific execution to separate roles is beneficial.

AuthorsOpenReview ↗

Response to Q3 (1/2): Token Costs by Task Outcome

Q3 (Part 1/2). How do token costs compare by task outcome and on instances where both systems achieve PP?

Table 3. Token consumption for PP and non-PP instances.

We break down execution token costs by task outcome and recovery stage. Table 3 uses the same set of 158 instances, each fully run by both systems. Success is defined as achieving Perfect Pass (PP), while non-PP includes partially completed instances. Token counts include execution-model inputs and outputs and exclude evaluation-model costs; the unit is thousands of tokens (k).

SystemPP instancesPP mean / medianNon-PP instancesNon-PP mean / medianOverall mean
OctoNest70684.4 / 618.988828.3 / 532.9764.6
UFO³-Hybrid40638.7 / 533.1118724.6 / 669.6702.9

OctoNest's mean cost in the non-PP group is 21.0% higher than in the PP group, but its median is lower, indicating that the non-PP group contains high-cost cases. Across the full set, OctoNest uses an average of 764.6k tokens per instance, compared with 702.9k for UFO³-Hybrid, an increase of 8.8%; the numbers of PP instances are 70 and 40, respectively, corresponding to a success-rate difference of 18.99 percentage points. Dividing all execution tokens by the number of PP instances yields a 37.8% reduction in amortized token cost per PP. This analysis is consistent with the main text's conclusion that a higher success rate improves amortized token cost per PP.

Table 4. Token consumption on the 25 instances where both systems achieve PP. Values are in k tokens.

To compare execution costs on the same instances with the same successful outcome, we further analyze the 25 instances on which both systems achieve PP. OctoNest's mean token consumption is 576.3k, compared with 656.2k for UFO³-Hybrid, a reduction of 12.2%; the medians are 499.9k and 566.9k, respectively.

SystemShared PP instancesMean (k)Median (k)
OctoNest25576.3499.9
UFO³-Hybrid25656.2566.9

This paired comparison also helps interpret the PP-group means in Table 3: OctoNest and UFO³-Hybrid achieve PP on 70 and 40 instances, respectively, so their success-group means reflect different sample compositions. Together, the full-set and paired results show that OctoNest achieves perfect completion on more instances while using fewer tokens on average on the same instances that both systems complete perfectly. These results provide further evidence of OctoNest's token efficiency in achieving Perfect Pass, consistent with the token-efficiency findings in the main text.

AuthorsOpenReview ↗

Response to Q3 (2/2): Recovery Costs by Component and Stage

Q3 (Part 2/2). What are the component costs during recovery, and how do recovery costs relate to full-task costs?

Token counts include execution-model inputs and outputs and exclude evaluation-model costs; the unit is thousands of tokens (k).

Table 5. Component token costs within the recovery window.

We use the 33 paired instances from Figure 5 of the main text in which both systems encountered the same failure. The recovery window starts when the first feedback for that failure is received and ends when the affected step is recovered; if it is not recovered, the window continues until the task ends. Table 5 reports mean costs per window in k tokens. The control-layer subtotal is the sum of the Orchestrator and Device Agent costs and is already included in the window total. Individual entries and subtotals are calculated using unrounded values and then rounded separately to one decimal place.

SystemOrchestratorDevice AgentControl subtotalModality ExecutorsWindow totalAffected-step recovery
OctoNest10.326.736.9248.4285.327/33
UFO³-Hybrid153.3—153.3274.3427.616/33

Even after including all repeated decisions by the Device Agent, OctoNest's control-layer cost per recovery window remains 75.9% lower, with a lower cost in 27 of the 33 pairs. Including the Modality Executors, the total window cost is 33.3% lower, while the numbers of instances with the affected step recovered are 27/33 versus 16/33. This paired analysis is consistent with the main text's conclusion that combining local recovery with cross-device orchestration is beneficial, and provides evidence of lower control costs within the recovery window. UFO³-Hybrid's global graph editing also handles routine execution feedback, so the control-layer cost here measures all controller calls within the window.

Table 6. Token costs by stage for paired instances.

Table 6 uses all 33 paired instances from Table 5 and separately reports the 15 instances in which both systems recovered the affected step. Values are mean k tokens per instance. The pre-window stage includes the execution attempt that first triggered the failure; when recovery does not occur, the window continues until the task ends. Differences are calculated as OctoNest minus UFO³-Hybrid.

Paired sampleSystemBefore windowWithin windowAfter recoveryFull task
All 33 pairsOctoNest166.8285.3326.3778.4
All 33 pairsUFO³-Hybrid137.8427.6190.6756.0
All 33 pairsDifference+28.9−142.2+135.7+22.4
Both recovered: 15 pairsOctoNest122.4209.2399.7731.3
Both recovered: 15 pairsUFO³-Hybrid87.7280.2393.3761.3
Both recovered: 15 pairsDifference+34.6−71.0+6.4−30.0

Across all 33 paired instances, OctoNest uses an average of 142.2k fewer tokens within the recovery window, but 28.9k more before the window and 135.7k more after recovery, resulting in a net increase of 22.4k for the full task. The main offset occurs in the post-recovery execution stage, consistent with OctoNest recovering more affected steps (27/33 versus 16/33) and therefore allowing more instances to proceed with subsequent tasks. Restricting the comparison to the same 15 instances in which both systems recovered, OctoNest still uses an average of 71.0k fewer tokens within the recovery window, while the post-recovery difference shrinks to 6.4k and the average full-task cost is 30.0k lower. This analysis explains how a lower recovery-window cost can coexist with a slightly higher full-task cost, and is consistent with the main text's conclusions of a higher recovery success rate and lower amortized token cost per PP.

Reviewer_Gj2JOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OctoNest combines global planning and device assignment with local agents that use feedback to refine API, CLI, or GUI actions. It also introduces CAPEBench to test recovery from interface-level and device-level failures. OctoNest completes more tasks correctly than UFO3-Hybrid, its strongest baseline.

Critical strengths
  • UFO3-Hybrid shares the corresponding executors and supports global replanning. OctoNest improves Perfect Pass by about 18 percentage points, with a paired seed-task bootstrap interval excluding zero. Ablations and execution examples help explain the value of local recovery.

  • CAPEBench checks required steps and permitted devices. Four-backbone comparisons, cross-judge checks, a human audit, external CRAB tasks, and oracle planning strengthen the evidence. Tokens per perfect pass improve in the main comparison, though not under every backbone.

Critical weaknesses
  • Design choices are not fully isolated. The Unified Device Agent uses Kimi K2.5 for all local execution, whereas the full system uses DeepSeek-V4-Pro for text components and Kimi for GUI execution. This changes both architecture and model choice. The No Device Agent and One-shot variants test repeated local decisions, but also change where additional inference is available. Their executors retain multi-step loops and global replanning remains enabled. These are useful ablations, but comparisons at matched budgets would better separate role organization from additional effort.

  • Broader reliability remains uncertain. CAPEBench derives its variants from 23 seed workflows, with failures designed to leave a feasible recovery route. The seed-task bootstrap accounts for related variants, but one execution per instance cannot measure run-to-run variability. CRAB provides useful external evidence, including better partial completion, yet the success advantage over UFO3 is just one task on a small test set.

Questions for authors
  1. Could you compare unified and separated local agents with the same model wherever possible, and report success at comparable token or time budgets?

  2. Could you repeat representative tasks under each failure condition and report success rates by seed workflow, to show both execution variability and how widely the gains are distributed?

4 comments
AuthorsOpenReview ↗

Response to Q1: Architecture Comparison under Matched Models and Token Budgets

Thank you for recognizing the improvements over UFO³-Hybrid and the evidence supporting local recovery. We add comparisons with matched models and per-instance token budgets, together with repeated runs and results broken down by workflow and runtime condition.


Q1. Does separating device-local coordination from modality-specific execution improve success under matched models and comparable token or time budgets?

We compared OctoNest and Unified Device Agent on the same fixed 82-instance analysis sample. Both methods use Kimi K2.6 for global planning, device-local coordination, and API, CLI, and GUI execution (as Kimi K2.5 is no longer available on Kimi's official platform), with identical perturbation injection and a 30-minute execution time limit per instance. We additionally conducted a per-instance token-budget-matched analysis.

We use Unified's actual execution token consumption on each instance—the sum of model input and output tokens—as the budget cap for OctoNest on the corresponding instance. Full runs on 50 instances fall within the cap, and their complete scores are retained. For the remaining 32 instances, we truncate the saved trajectories and verify completed steps using only operations executed, tool outputs, and screenshots within the budget, then recalculate Task Completion, Instruction Adherence, and Perfect Pass. Calls exceeding the budget, together with subsequent actions and outcomes generated by those calls, receive no credit.

Table 1. Full-run and per-instance budget-matched results under the same model configuration. Each method is evaluated on 82 instances.

Method / settingComp. (%)Adh. (%)PP (%)Execution tokens (M)
Unified Device Agent47.4050.327.3276.23
OctoNest: full run63.7265.0526.8367.02
OctoNest: per-instance budget truncation52.4056.9221.9551.51

After per-instance budget matching, OctoNest still exceeds Unified in Task Completion, Instruction Adherence, and Perfect Pass by 5.00, 6.60, and 14.63 percentage points, respectively. In full runs, OctoNest consumes 67.02M execution tokens in total, 12.09% fewer than Unified's 76.23M, while achieving higher scores on all three quality metrics. This result is consistent with the main text's conclusion that separating device-local coordination from modality-specific execution is beneficial, and further shows that this advantage persists with identical model configurations and per-instance token budget constraints.

Using Unified's actual token consumption on each instance as the budget cap, we truncate the corresponding trajectory of the full system and reassess its scores based only on execution evidence before the cutoff. After truncation, OctoNest retains a total of 51.51M execution tokens. The matching applies to the budget cap for each instance. Budget left unused because of early termination or complete-call boundaries is not filled, so aggregate consumption can be lower than Unified's.

AuthorsOpenReview ↗

Response to Q2: Run-to-Run Variability and Gains across Workflows

Q2. How much do repeated runs vary, and how are the gains distributed across workflows and runtime conditions?

We selected 6 of the 22 candidate workflows that have all four runtime conditions. Each workflow includes four corresponding variants under unperturbed, modality-level, device-level, and mixed conditions, yielding 24 instances in total. Both OctoNest and UFO³-Hybrid use DeepSeek-V4-Pro for text-based agents and API/CLI execution, and Kimi K2.6 for GUI execution. Table 2 labels each method's three runs as R1, R2, and R3, and reports quality and efficiency metrics for each run and the mean ± sample standard deviation.

Table 2. Per-run performance and summary statistics for repeated runs. Each run evaluates the same set of 24 instances.

Tok./Ep. and Tok./PP denote execution tokens per episode and per perfect pass, respectively (k: thousands; M: millions), excluding evaluation-model costs. The mean ± SD rows summarize the three round-level values.

MethodRunComp. (%)Adh. (%)PP (%)Tok./Ep. (k)Tok./PP (M)
OctoNestR156.2556.2533.33709.82.129
R263.5462.2233.33878.02.634
R366.8864.7237.50884.32.358
Mean ± SD62.22 ± 5.4361.06 ± 4.3534.72 ± 2.41824.0 ± 99.02.374 ± 0.253
UFO³-HybridR143.7546.8112.50824.76.598
R245.6349.1716.67919.75.518
R342.5046.8116.67841.25.047
Mean ± SD43.96 ± 1.5747.59 ± 1.3615.28 ± 2.41861.9 ± 50.75.721 ± 0.795

Across the 72 task runs in the three selected rounds, OctoNest achieves PP 25 times, compared with 11 for UFO³-Hybrid. Their mean PP rates are 34.72% and 15.28%, respectively, a difference of 19.44 percentage points. This advantage is present in all three selected rounds: OctoNest's lowest single-round PP rate is 33.33%, still above UFO³-Hybrid's highest rate of 16.67%. Both systems have a round-to-round PP standard deviation of 2.41 percentage points, indicating that, in these three rounds, OctoNest achieves a higher mean success rate with comparable variation in the overall success rate. In addition, OctoNest's mean Task Completion and Instruction Adherence are higher by 18.26 and 13.47 percentage points, respectively, showing that the gains also extend to task completion and instruction adherence. These repeated-run results are consistent with the main text's conclusion that OctoNest performs better overall.

Across these 72 runs, OctoNest consumes 59.33M execution tokens in total, compared with 62.05M for UFO³-Hybrid, a reduction of 4.39%. Dividing each system's total execution tokens by its total number of PP outcomes (25 and 11, respectively) yields pooled Tok./PP values of 2.373M and 5.641M, a 57.93% reduction for OctoNest. These results are consistent with the main text's conclusion that OctoNest improves execution quality and reduces token cost per perfect pass.

Table 3. PP distribution across workflows and runtime conditions. n/3 denotes the number of PP outcomes out of 3 runs. Overall PP is computed over 12 runs across the four conditions; bold indicates the better result.

WorkflowMethodUnperturbedModality-levelDevice-levelMixedOverall PP (%)
S01 Alert handlingOctoNest0/30/30/30/30.00
UFO³-Hybrid0/30/30/30/30.00
S02 PR review coordinationOctoNest1/30/32/31/333.33
UFO³-Hybrid0/30/30/30/30.00
S06 File version comparisonOctoNest2/32/31/33/366.67
UFO³-Hybrid2/31/31/30/333.33
S09 Cross-thread email summaryOctoNest1/33/31/30/341.67
UFO³-Hybrid2/30/31/31/333.33
S13 Change approval notificationOctoNest2/30/33/30/341.67
UFO³-Hybrid0/30/30/30/30.00
S16 Escalated incident triageOctoNest1/32/30/30/325.00
UFO³-Hybrid1/31/30/31/325.00

Aggregated by workflow, OctoNest achieves a higher PP rate on four of the six workflows and ties on the other two. The gains mainly come from change approval notification, PR review coordination, and file version comparison, with net increases of 5, 4, and 4 PP outcomes, respectively. In particular, under device-level perturbations for S13 and mixed perturbations for S06, OctoNest achieves 3/3 PP in both cases, compared with 0/3 for UFO³-Hybrid. These results are consistent with the overall comparison in the main text and further show how the gains are distributed across workflows and runtime conditions.

I thank the authors for the response and additional experiments, which address my main concerns. However, given the limited benchmark scope, I will maintain my current score.

We sincerely thank you for your thoughtful feedback and constructive suggestions throughout the review process. We are glad that our response and additional experiments have addressed your main concerns. We also appreciate your recognition of our work. Please feel free to raise any further questions or concerns, and we would be happy to discuss them.

Reviewer_EkNqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper has certain novelty and contribution to the use of agents in cross-device computers. The authors present a stateful cross-device execution framework OctoNest, focusing on the recovery on intra-device switching failures and inter-device switching failures. The architecture introduces Device Agents capable of iterative operations between the Orchestrator and Modality Executors covering API, GUI, and CLI. The paper also proposes CAPEBench covering unperturbed, modality-level, device-level, and mixed conditions. The experiments include cross-device system comparison, architecture ablation, execution behavior analysis, backbone of different models, Oracle planning, and external CRAB task testing. Overall, the results reported in the paper support the conclusion that OctoNest outperforms the compared implementations. However, they are not sufficient enough to demonstrate that its feasibility assessment mechanism is universally reliable or capable of being widely generalized to real-world deployment environments.

Critical strengths

1.Significant system-level improvements have been achieved in the fairer major comparisons UFO³ only supports single modality, however for comparison with more fairness, the author has endowed it with the same API/CLI/GUI execution capabilities as OctoNest. The acceptance case rests on an observed improvement with comparable modality access and matched models in the primary comparison, not simply on giving OctoNest more kinds of tools. 2: CAPEBench supports targeted diagnosis beyond aggregate success rates The benchmark clearly defines the duration of disturbance and the injection mechanism. Also, it allows instantaneous API errors to be restored through retries instead of forcibly defining each failure as a modality switch. CAPEBench is a valuable diagnostic assessment method which helps to verify their framework works.

Critical weaknesses

The network-perturbation allowed-device rule needs consistency with task-relevant feasibility. Section 2.1 defines reassignment in terms of infeasibility of the unfinished goal on the original device. Appendix B.3 states that device-level perturbations concern task-relevant feasibility and need not disable every device function. This raises a key question: If a step can still be executed entirely locally, is it still necessary to transfer it to another device in order to earn Adherence credit? For instance, input a file generation operation that already exists locally, and theoretically, it may not rely on the network. The existing materials cannot confirm whether such a situation actually exists in the benchmark. Therefore, for now, it should be regarded as an unresolved evaluation ambiguity rather than a proven misrating. The authors should clarify the following questions and showcase a full example: Which specific operations are blocked in case of network failure? On which devices are local-only and network-dependent gold steps allowed to be executed respectively?

Questions for authors
  1. How are locally feasible steps scored after a device network failure? Please reconcile Appendix B.3’s task-relevant failure definition, Appendix D.1’s retention of original constraints for unaffected steps, and Appendix D.2’s network-related peer-device rule. For a local-only gold step with available inputs, which devices receive adherence credit, and why? Could you provide an example to elaborate?
  2. When the local model is completely matched, does the advantage of executor separation still exist? Could you prove that?
4 comments
AuthorsOpenReview ↗

Response to Q1: Device Constraints and Scoring under Network Perturbations

Thank you for recognizing the system-level gains under comparable modality access and the diagnostic value of CAPEBench. We clarify the scoring rules, report the corrected evaluation, and add comparisons with fully matched models and per-instance token budgets.


Q1. How should locally executable gold steps be scored after a device loses network access?

A device-level perturbation does not mean that all operations on the original device become infeasible. A gold step whose inputs are already available and that can be completed locally should receive Instruction Adherence credit when completed on the original device specified in the instruction. The original device always remains in the set of allowed devices; a peer adds an acceptable execution location.

Specifically, for non-network device-level perturbations, a same-type peer is added only for the gold steps affected by the perturbation. For device-wide network perturbations, we add a same-type peer for all gold steps originally assigned to that device, allowing tasks with upstream and downstream dependencies to migrate together. For example, generating a report locally does not itself require network access, but the subsequent upload may need to be performed on a peer. Therefore, when the inputs are already available, the report-generation step can receive credit when completed on either the original device or the corresponding peer.

We have corrected the overly restrictive device constraint in the scoring implementation and re-evaluated all execution trajectories used in the main text under these unified rules. The revised manuscript will also correct the wording in Appendix D.2, clarify the allowed-device rules under network outages, and update the relevant numbers. After re-evaluation, OctoNest's PP advantage over the strongest baseline is 18.99 percentage points, and its amortized token cost per PP is 37.8% lower. All experimental results continue to support the core claims in the main text, and the paper's overall conclusions remain unchanged.

AuthorsOpenReview ↗

Response to Q2: Architectural Benefits with Fully Matched Models

Q2. Does separating device-local coordination from modality-specific execution still help when the models are fully matched?

We have added a comparison with fully matched models. Both OctoNest and Unified Device Agent use Kimi K2.6 for global planning, device-local coordination, and API, CLI, and GUI execution (as Kimi K2.5 is no longer available on Kimi's official platform), while Instruction Adherence is evaluated consistently using DeepSeek-V4-Pro. Each method was run once on the same fixed 82-instance analysis sample, for a total of 164 task runs, with identical perturbation injection and a 30-minute execution time limit per instance. OctoNest retains separate roles for device-local coordination and modality-specific execution. Unified retains the same global Orchestrator and allows multi-step execution within a device, but uses a single local agent for modality selection and operation execution. Execution token counts include model inputs and outputs across all execution roles and exclude evaluation-model costs.

Table 1. Architecture comparison with fully matched models on the same 82 instances. Execution tokens are reported in millions (M). Δ denotes OctoNest minus Unified; quality differences are in percentage points, and all differences are calculated before rounding.

MethodComp. (%)Adh. (%)PP (%)Execution tokens (M)
OctoNest63.7265.0526.83 (22/82)67.02
Unified Device Agent47.4050.327.32 (6/82)76.23
Δ+16.32+14.73+19.51−9.22 (−12.09%)

With fully matched models, OctoNest improves Task Completion, Instruction Adherence, and Perfect Pass over Unified by 16.32, 14.73, and 19.51 percentage points, respectively. OctoNest achieves PP on 22/82 instances, compared with 6/82 for Unified, while reducing total execution tokens from 76.23M to 67.02M, a reduction of 12.09%. These results show higher task execution quality alongside lower total inference costs under identical models, consistent with the main text's conclusion that separating device-local coordination from modality-specific execution is beneficial.

To further control per-instance inference effort, we use Unified's actual token consumption on each instance as the budget cap for OctoNest on the corresponding instance. We perform post-hoc truncation of OctoNest's trajectories and score only completed outcomes supported by execution evidence within the budget.

Table 2. Per-instance token budget comparison with fully matched models on the same 82 instances. OctoNest is truncated and scored under Unified's per-instance token budget cap. Δ denotes OctoNest minus Unified; quality differences are in percentage points.

MethodComp. (%)Adh. (%)PP (%)Execution tokens (M)
OctoNest52.4056.9221.95 (18/82)51.51
Unified Device Agent47.4050.327.32 (6/82)76.23
Δ+5.00+6.60+14.63−24.72 (−32.4%)

Under per-instance budget constraints, OctoNest remains ahead on all three quality metrics, including a 14.63-percentage-point PP advantage. This further shows that the benefit of assigning device-local coordination and modality-specific execution to separate roles persists when the underlying models are fully matched and per-instance inference budgets are constrained. The matching applies to the budget cap for each instance. Budget left unused because of early termination or complete-call boundaries is not filled, so the total execution tokens retained after truncation can be lower than Unified's.

Thank you for your detailed rebuttal & additional experiments.

The clarification for Q1&Q2 resolves my partial concerns. However, they do not change my assessment of robustness & generalizability of the framework. I will keep the original score unchanged.

We greatly appreciate your careful evaluation of our work and the constructive feedback you have provided. We are pleased that our clarifications and additional experiments have helped resolve your concerns regarding Q1 and Q2. Thank you again for your time and valuable insights throughout the discussion. We would be happy to provide further clarification if any additional questions arise.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.