acceptodds
Under review as a conference paper at ICLR 2027

Edge Symphony: A Harness-Aware Benchmark for Hybrid Mobile GUI Agents

Abstract

Mobile GUI agents that run entirely in the cloud raise privacy concerns and incur substantial API cost. Hybrid agents instead distribute inference between on-device and remote models through an agent harness, which decides where computation runs, what information is exchanged, and when control transfers between models. This raises a central question: what capability does a harness realize in hybrid mobile GUI execution, and at what deployment cost? We introduce Edge Symphony, a harness-aware benchmark that evaluates hybrid mobile GUI agents on physical smartphones with native on-device inference. Edge Symphony comprises 258 reproducible tasks across 23 applications, organized into six capability categories and extensible through an automated task curation workflow, and jointly measures Smartness and Cost. We also provide two reference harnesses, PocketAct (fully local) and RelayAct (hybrid). Experiments with Edge Symphony reveal three findings on hybrid agents invisible to model-centric evaluation: efficiency in converting model calls into actions bounds realized capability; harness design relocates cost rather than eliminating it; and model gains does not guarantee a better agent, which is also determined by the harness. These findings provide valuable insights into hybrid mobile GUI agents design.

Primary area:
datasets and benchmarks
Keywords:
Mobile GUI Agents, Device–Cloud Hybrid Agents, Agent Harness, Agent Benchmark, On-device Inference.

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_otWaOpenReview ↗
Rating
2(−1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Edge Symphony is a harness-aware benchmark for hybrid mobile GUI agents: it runs 258 reproducible tasks across 23 apps in six capability categories on physical smartphones with native on-device inference, and jointly measures Smartness (task success, per-capability success, interaction efficiency) and Cost (latency, device energy, API spend). It also ships two reference harnesses, PocketAct (fully local) and RelayAct (hybrid). Its experiments show three effects invisible to model-centric evaluation: harness efficiency in converting model calls into GUI actions bounds realized capability, offloading relocates cost from API bills to device energy and latency rather than removing it, and a stronger remote model can yield a weaker hybrid agent depending on the harness's control-transfer protocol.

Critical strengths
  1. Real on-device execution cost, measured rather than estimated. Prior mobile GUI benchmarks either keep inference off-device (GUI-CEval, MobileBench-OL run on real phones but in the cloud) or measure only isolated inference workloads (PalmBench-style). Edge Symphony runs LiteRT-LM natively on a physical phone and reads ODPM power-rail counters (IPowerStats HAL, cumulative µW·s deltas) to obtain latency, whole-device energy, and API spend from the same execution, under a synchronized window that excludes initialization and watchdog recovery.

  2. A counterintuitive finding only a benchmark could surface. 3.8-Flash beats 3.5-Flash as a standalone agent (72.1% vs. 70.5%) but loses under RelayAct (58.9% → 50.0%), because it follows the handback protocol more faithfully and returns more execution to the weaker on-device model. Realized capability is thus jointly determined by model and harness, and step-level traces show where control actually resides.

Critical weaknesses
  1. Single-vendor remote models. Every run uses only Gemini (3.5-Flash, plus 3.8-Flash in two rows); GPT and Claude appear only in the pricing table, never in an experiment. Since the headline "stronger model, weaker agent" result is explained by protocol-following behavior that likely varies across vendors, its generality is untested.

  2. Inflated novelty attribution. The paper frames "the harness as the object of evaluation" and "model and harness jointly determine capability" as primary contributions, yet ACE-Bench already compares edge-only, cloud-only, and edge–cloud placement, and Harness-Bench already concludes that capability must be reported at the model-harness configuration level. Read against these works, the contribution reduces to the first instantiation of those effects in a mobile GUI setting with on-device inference and measured energy — still valid, but a notch below the framing — and the paper never performs that demotion itself.

Questions for authors

Do the two reference harnesses satisfy system fairness? PocketAct and RelayAct beat every baseline, but both are the authors' own implementations, and Appendix D shows heavily engineered prompts covering failure modes such as loops, stalls, and false completion — yet no prompt-engineering ablation is reported. Reporting one — or running a third-party published harness on the same benchmark — would separate gains from harness design principles from gains from implementation maturity.

3 comments
AuthorsOpenReview ↗

Response to weakness 1.

Weakness 1. Single-vendor remote model. Thank you for raising the concern about remote-model diversity and the generality of the third insight in Section 5.4. Our finding is that upgrading the model does not guarantee a better hybrid agent, not that a stronger model generally yields a weaker one. This finding does not depend on cross-vendor generality: a single counterexample of the Gemini family is sufficient to support it, even if protocol-following behavior varies across vendors. In the revision, we have clarified the corresponding statements in the third insight as ``observed in the tested Gemini configurations'' to make them more precise.

We agree that cross-vendor generality remains untested. What our data do establish is that the effect is a property of the control-transfer protocol rather than of one checkpoint: across three Gemini generations under Default RelayAct, handback rate rises monotonically (40.8% → 65.6% → 72.9%, Gemini3.5 → 3.8 → 4 Argon) while success falls (57.2% → 48.9% → 51.2%), and Sticky handoff reverses it (65.0% → 68.9% → 76.6%). Edge Symphony's LiteLLM-compatible interface runs any OpenAI/Anthropic/other API model under the identical protocol, so the cross-vendor test is directly reproducible by the community.

AuthorsOpenReview ↗

Response to weakness 2.

Weakness 2. Inflated novelty attribution. We thank the reviewer for the careful comparison with ACE-Bench and Harness-Bench.

Major contribution. We agree with the reviewer that the major contribution of Edge Symphony lies in the benchmark itself. We do not regard the general principle that models and harnesses jointly determine capability and related findings as our primary contribution. Instead, Edge Symphony runs on-device models natively on physical smartphones, and executes hybrid harnesses on long-horizon GUI tasks. In addition to a change of setting, it enables analyses that require on-device measurement, such as practical end-to-end latency and whole-device energy of GUI agent task. To our knowledge, no existing benchmark provides such a testbed. The release of Edge Symphony will provide great value to the open-source community and developers in this field.

Related work. Both works have been already discussed in Section 2.2. We agree that their distinctions from Edge Symphony should be made more explicit. ACE-Bench, now EC²-Bench, compares edge–cloud collaboration strategies in terms of task performance, resource usage, and privacy exposure. It does not evaluate execution on physical smartphones: edge-side computation is estimated using FLOPs, and actual device energy and end-to-end latency are not measured. Harness-Bench evaluates model–harness configurations on sandboxed agent tasks, whereas Edge Symphony targets on hybrid harnesses which explicitly coordinate an on-device model and a remote model during GUI execution. We have revised the Abstract and Related Work to clarify these distinctions and position our contributions more precisely.

AuthorsOpenReview ↗

Response to Question

Question. System Fairness and Prompt Engineering Attribution We thank the reviewer for raising the question of system fairness and the role of prompt engineering.

Third-party harness comparison. The main table and figure in Section 5.2 have already included three published third-party harnesses, M3A, SeeAct-V, and EcoAgent, evaluated under the same benchmark protocol with shared base models and GUI action interfaces. We retain their original prompts without modification to preserve their respective harness designs, as role-specific prompts are themselves part of an agent harness.

Scripted control flow and role prompts. We clarify that RelayAct's failure-handling mechanisms are not implemented solely through prompt engineering but mainly from the scripted control flow. In RelayAct, repeated-action loops and subgoal stalls are detected by a scripted progress tracker, which triggers escalation to the remote Unblocker. Similarly, when the local Executor reports subgoal completion, the harness invokes the Verifier and routes corrective feedback if the claim is rejected. The role prompts provide supplementary guidance to help models follow these control-flow rules instead of the primary control. For example, the remote Unblocker's prompt explicitly instructs it to return an updated plan after resolving a stalled subgoal. Importantly, these are task-agnostic harness mechanisms but general control flow: they monitor and correct the agent's own execution behavior using action histories, progress states, and screen observations, without relying on benchmark-specific task categories or labels. We have added this clarification in Appendix D.3 of the revised manuscript.

RelayAct prompt-guardrail ablation. To distinguish the effects of supplementary prompt guidance from scripted control flow, we strip the Appendix D.3 guardrails across all four local and remote roles: the Local Verifier and Executor's anti-retry, non-interactive-element, and date-picker heuristics, the Cloud Planner's one-shot example, and the Cloud Unblocker's handback reminders. Only bare role definitions and coordinate schemas are retained. As shown in Table 6, removing these guardrails does not reduce overall success (57.2% vs.\ 60.7%). The handback rate drops substantially from 40.8% to 18.2%, suggesting that the removed guidance helps the remote Unblocker follow the intended control-transfer protocol. These results indicate that supplementary prompt guidance influences how models comply with harness rules, but RelayAct retains strong performance without these additional prompt-engineering components. We have added this ablation study in Appendix E.7 of the revised manuscript.

Table 6: RelayAct prompt-guardrail ablation. (Gemini 3.5-Flash)

Ablation VariantOverall SR (Δ vs. Default)Handback RateUnblocker ShareLatency (s)Energy (J)API Cost ($)
Default57.2 ± 2.6% (–)40.8%60.2%4181,1840.195
w/o Prompt Guardrails60.7% (+3.6 pts)18.2%63.7%4141,1720.210

w/o Prompt Guardrails: Remove the one-shot example, domain heuristics, and Cloud Unblocker handback warnings as described in Appendix D.3, retaining only bare role and coordinate schemas.

Reviewer_k9WUOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper introduces Edge Symphony, a benchmark for testing hybrid edge-cloud mobile GUI agents on real phones with native on-device execution. It provides 258 tasks across 23 apps, two reference workflows (PocketAct and RelayAct), and built-in battery power tracking. Experiments show that workflow design heavily impacts success rates, edge execution trades API cost for battery drain, and stronger cloud models can sometimes hurt hybrid performance.

Critical strengths
  • It runs edge models natively on real phones and tracks step-by-step battery drain without external power meters.

  • This work has the potential to be a good community asset of testing cloud-edge agent collaboration on real devices for mobile use agents.

  • It reveals that upgrading the cloud model (Gemini 3.5 to 3.8) can lower hybrid success by 8.9% due to early handoffs back to the weak edge model.

Critical weaknesses
  • All tests use only one phone model (Pixel 9 Pro) and one edge model (Gemma-4B), limiting hardware and model diversity.
Questions for authors

na

2 comments
AuthorsOpenReview ↗

Response to weakness.

Weakness. Lack of various phone models and edge models. We thank the reviewer for highlighting the importance of hardware and on-device model diversity. Following the suggestion, we substantially expanded our evaluation from one smartphone and one on-device model to three smartphones from different manufacturers and four on-device VLM variants. Each configuration was evaluated on the complete 280-task suite with repeated trials for the main configurations (Please refer to the response to weakness 1 of reviewer tTDK for the detail of task suite update).

Table 4: Benchmark Generalization across various hardware and on-device models.

Smartphone / Local / Remote ModelSuccess RateHandback RateUnblocker ShareMean Latency
1. Google Pixel 9 Pro / Gemma 4 E4B / 3.557.2 ± 2.6%40.8%60.2%418 s
2. Samsung Galaxy S25+ / Gemma 4 E4B / 3.559.8 ± 4.7%39.6%59.1%285 s
3. Samsung Galaxy S25+ / Gemma 4 E4B / 3.848.6 ± 2.5%67.5%42.9%388 s
4. Xiaomi 14 / Gemma 4 E4B / 3.562.0 ± 1.6%41.8%60.3%357 s
5. Xiaomi 14 / Gemma 4 E4B / 3.852.1 ± 3.5%69.0%45.2%462 s
6. Pixel 9 Pro / LFM3-VL-2B / 3.545.7%43.3%60.8%747 s
7. Pixel 9 Pro / LFM3-VL-2B / 3.830.0%67.1%46.9%872 s
8. Pixel 9 Pro / Gemma 4 E2B / 3.536.8%53.1%46.4%344 s
9. Pixel 9 Pro / LFM2.5-VL-3B / 3.520.7%50.7%59.9%185 s

The additional results in Table.4 demonstrate that the Insights 3 in the original manuscript is consistent across devices (row 2--5) and local models (row 6--7). This generalization result across various smartphone are added in the Appendix E.5 of the revised manuscript. In addition, it also reveal a clear Smartness--Cost trade-off with different local models (row 6,8,9): LFM3-VL-2B achieves the highest success rate (45.7%) but requires substantially longer execution time (747s), compared with Gemma 4 E2B (36.8%, 344s) and LFM2.5-VL-3B (20.7%, 185s). More interestingly, changing the local model also affects the remote's control-transfer behavior, with handback rates ranging from 43.3% to 53.1% and unblocker shares from 46.4% to 60.8%, even under the same remote model and harness. These findings highlight the impact of local model selection on both deployment efficiency and local--remote interaction. Detailed analysis is provided in Section 5.3 of the revised manuscript.

Table 5: Cross-phone comparison of fully local, hybrid, and fully remote execution.

PhonePocketActRelayAct 3.5RelayAct 3.8Remote-E2E 3.5
Google Pixel 9 Pro9.8 ± 0.6%57.2 ± 2.6%48.9 ± 0.6%72.7 ± 0.7%
Samsung Galaxy S25+11.1 ± 2.0%59.8 ± 4.7%48.6 ± 2.5%77.4 ± 0.9%
Xiaomi 149.3 ± 1.0%62.0 ± 1.6%52.1 ± 3.5%78.7 ± 1.0%

Thanks to the suggestion on expanding the device. We have findings of interest that Xiaomi 14 consistently achieves slightly higher success rate once the strong remote model engages (i,e. hybrid and fully remote), as indicated in the Table.5. We infer that it might be related to display resolution, which increases from Pixel 9 Pro (960x2142) to Galaxy S25+ (1080x2340) and Xiaomi 14 (1200x2670). This speculation is motivated by our further trace analysis, which is natively supported by Edge Symphony. The results show that the visual false completion rate decreases from 32-34 tasks per trial down to 22-23 tasks per trial on higher-resolution phones, a relative reduction of approximately 33%. This suggests that for GUI agent task operating on the real hardware devices, the resolutions also matters and should be considered.

Reviewer_k9WUOpenReview ↗

review

Thanks for your response. I will keep my original score of weak acceptance.

Reviewer_tTDKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EdgeSymphony, a benchmark for hybrid mobile GUI agents that evaluates how agent harnesses affect task performance and deployment cost. It comprises 258 tasks across 23 applications and six capability categories, supports native on-device inference on physical smartphones, and jointly measures success rate, model calls, latency, device energy consumption, and API expenditure. The authors also present an automated task-curation pipeline and two reference harnesses: PocketAct for fully local execution and RelayAct for hybrid execution. Experiments compare orchestration strategies and analyze execution traces to examine invocation efficiency, device–cloud cost allocation, and interactions between model behavior and control handoffs.

Critical strengths

The benchmark addresses a practically important evaluation problem. By combining native inference on physical smartphones with end-to-end GUI task execution, the benchmark assesses performance alongside latency, device energy consumption, and API expenditure. This captures deployment costs that success rates or cloud expenditure alone may overlook, providing a more comprehensive basis for comparing hybrid agents.

Critical weaknesses
  1. Some capability categories have too few tasks to support reliable comparisons. Ambiguous Content and Interruption and Recovery contain only 16 and 11 tasks, respectively. On Interruption and Recovery, RelayAct’s success rate ranges from 45.5% to 72.7% across three runs, showing substantial variability. Other configurations are not evaluated repeatedly at the category level, so their rankings may also be unstable. Adding more tasks and reporting averages and uncertainty intervals across multiple runs would make these comparisons more convincing.

  2. More evidence is needed to explain why a model upgrade reduces hybrid-agent performance. The two remote models differ by only about four successful tasks under Remote-E2E, and repeated evaluations are reported only for RelayAct with 3.5-Flash. Although 3.8-Flash returns control more often, this alone does not show that frequent handoffs cause the performance drop; differences in planning and execution could also contribute. Repeating both models’ evaluations and testing them under the same handoff rules would help verify the finding and its explanation.

  3. A fixed number of model calls does not give every harness the same resources. Planning, verification, and execution calls each count as one step, but a single call may produce multiple GUI actions. The results therefore depend partly on how each harness organizes calls and groups actions. Equal call counts also do not mean equal latency, energy use, or API cost. The budget analysis is useful, but comparisons under equal time, monetary, or energy budgets would show whether the main rankings hold in other deployment settings.

Questions for authors

Please refer to the paper weakness.

4 comments
AuthorsOpenReview ↗

Response to weakness 1.

Weakness 1. Variability of category AM and IR. Thank you for this observation. We agree that, with only 16 AM and 11 IR tasks, a single task outcome shifts the category success rate by 6–9 points, which limits the reliability of category-level comparisons. Following the suggestion, we double the task size of both categories AM (16 → 26) and IR (11 → 23). The new tasks also add test types that the original categories lacked. The new 26 AM task set covers target ambiguity (14 tasks), parameter ambiguity (6), and scope ambiguity (6); the new 23 IR task set cover faulty app behavior (7), runtime interruptions (8), conflicting data (4), and missing prerequisites (4). In addition, the added tasks broaden application coverage (from 12 to 14 apps in AM and, more substantially, from 5 to 13 apps in IR). We have updated the description of AM and IR category in Section 3.2.1. All newly added tasks have passed the oracle test as described in Section 3.2.2 as well as human verification.

With these additions, the new task suite is expanded to 280 tasks in total. We repeated the complete evaluation three times for all major configurations under same model-call budget on Pixel 9 Pro. In addition to Gemini 3.5 and 3.8 from the original manuscript, we include Gemini 4 Argon in this extended evaluation. Table.1 reports the mean and standard deviation over the three runs.

Table 1: Repeated success rates of nine model–harness configurations over three trials. Gemini 4 Argon is newly added.

ConfigurationOverall SRAM SRIR SRLatency (s)Energy (J)API Cost ($)
PocketAct, E4B9.8 ± 0.6%5.1 ± 2.3%0.0 ± 0.0%1,329 ± 584,675 ± 690.000
EcoAgent, Gemini 3.5 + E4B16.4 ± 0.7%10.2 ± 2.2%1.4 ± 2.5%746 ± 122,590 ± 460.108
SeeAct-V, Gemini 3.5 + E4B26.0 ± 0.8%24.4 ± 5.9%15.9 ± 2.5%247 ± 5598 ± 260.204
RelayAct, Gemini 3.5 + E4B57.2 ± 2.6%38.5 ± 7.7%40.6 ± 5.0%418 ± 161,184 ± 330.195
RelayAct, Gemini 3.8 + E4B48.9 ± 0.6%39.8 ± 5.9%29.0 ± 5.0%602 ± 131,811 ± 420.130
RelayAct, Gemini 4 Argon + E4B51.2 ± 3.3%46.2 ± 0.0%21.7 ± 4.4%728 ± 512,098 ± 670.237
Remote-E2E, Gemini 3.572.7 ± 0.7%44.9 ± 2.3%62.3 ± 2.5%135 ± 8126 ± 140.219
Remote-E2E, Gemini 3.874.8 ± 2.0%61.5 ± 0.0%66.7 ± 6.6%166 ± 11144 ± 110.213
Remote-E2E, Gemini 4 Argon81.6 ± 0.9%89.8 ± 2.2%79.7 ± 2.5%319 ± 79232 ± 480.485

The results show that the category-level success rate becomes more stable after task suite expansion. For RelayAct with Gemini 3.5, the standard deviation of IR success decreases from 13.9% in the original manuscript to 5.0%. Note that category-level success rates are lower than those in the original manuscript. This is because the added tasks are more difficult with various types as mentioned above. Across the three repeated runs, all major configurations keep the same ranking as in the original manuscript, with the standard deviation of overall success at most 3.3%. We have reported the complete multi-run results in the Section 5.2 and update the capability-wise results in Appendix E.1 of revised manuscript.

AuthorsOpenReview ↗

Response to weakness 2.

Weakness 2. Evidence of the contribution of handoff policy. Thank you for suggesting repeated evaluations and matched handoff rules. As shown in our response to W1, RelayAct with Gemini 3.8 underperforms Gemini 3.5 consistently across repeated runs.

To isolate the effect of the handoff policy from other factors, we designed two controlled experiments. Plan Share examines the effect of planning: RelayAct 3.5 and RelayAct 3.8 both start from the identical initial plan generated by Gemini 3.5. Sticky Execution ablates the handback policy: once invoked, the remote Unblocker keeps control until the task ends instead of returning control to the local Executor. We compare three metrics: success rate (SR); handback rate (HR), the number of handbacks initiated by the Unblocker divided by the number of handovers from the local Executor; and unblocker share (US), the proportion of execution steps performed by the Unblocker. Under the Sticky policy, HR is zero by design. For clarity, the table reports the mean of three runs.

Table 2: RelayAct performance comparison under different handoff policies. 3.5 and 3.8 denote the remote model versions of Gemini. All values are averaged over three trials for clarity.

Handoff PolicyRelayAct 3.5 SRRelayAct 3.5 HRRelayAct 3.5 USRelayAct 3.8 SRRelayAct 3.8 HRRelayAct 3.8 USΔ SRΔ HRΔ US
Default — Unblocker returns a <plan> when the subgoal is completed57.2%40.8%60.2%48.9%65.6%43.8%−8.2+24.8−16.4
Plan Share — Planner uses Gemini 3.5; Unblocker uses the designated model57.2%40.8%60.2%46.8%67.5%42.7%−10.4+26.7−17.5
Sticky Execution — Once invoked, Unblocker keeps control until task completion65.0%0.0%71.2%68.9%0.0%70.0%+3.90.0−1.2

SR: Success Rate; HR: Handback Rate; US: Unblocker Share. Δ = Gemini 3.8 − Gemini 3.5. All Δ values are in percentage points.

We have two major findings from Table.2. First, planning does not account for the lower success of Gemini 3.8. If its plans were the cause, adopting the initial plan of Gemini 3.5 would raise its success rate; instead, the success rate decreases slightly, from 48.9% to 46.8%. Second, under Sticky Execution, the two models have nearly identical unblocker shares (70.0% vs. 71.2%), suggesting that the local Executor hands over the control to remote at similar points. The ranking nonetheless reverses. Gemini 3.8's success rate rises by 20%, from 48.9% to 68.9%. It surpasses Gemini 3.5 by 3.9%, which is consistent with its stronger capability in Table.1. Together, these two comparisons show that the handoff policy is the key factor behind the ranking reversal compared with planning and execution. We have included the complete results in the revised Appendix E.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to weakness 3.

Weakness 3. Rationale of model calls as budget and need of other budget analysis. We agree that results depend partly on how each harness organizes model calls and groups GUI actions. We also acknowledge that equal model-call budgets do not imply equal latency, energy consumption, or API cost. As a benchmark for evaluating how agents perform on physical smartphones, Edge Symphony considers the efficiency with which a harness organizes model calls as an important evaluation dimension. To investigate this, we measured the model-call efficiency of various on-device and hybrid frameworks using step-level traces.

Table 3: Invocation efficiency and per-GUI-action cost.

ConfigurationOverall SRGUI Actions per Model CallSteps w/ 0 GUI ActionsSteps w/ ≥2 GUI ActionsLatency / GUI ActionEnergy / GUI Action
Local-E2E8.2%1.182.7%10.7%57.6 s214.2 J
M3A9.6%0.5049.7%0.0%76.5 s286.9 J
PocketAct9.8%0.8318.9%2.5%44.8 s157.7 J
SeeAct-V26.0%0.3763.4%0.0%19.6 s47.5 J
EcoAgent16.4%0.3062.8%0.0%70.2 s243.7 J
RelayAct 3.557.2%0.957.5%1.9%17.5 s49.6 J
RelayAct 3.848.9%0.9311.0%3.5%23.5 s70.6 J
RelayAct 4 Argon51.2%0.9313.2%4.8%29.5 s85.1 J

The results in Table. 3 show that M3A requires approximately two model calls per GUI action, compared with 1.2 calls for PocketAct. As for the hybrid harness, both EcoAgent and SeeAct-V need approximately 3 model calls to produce 1 GUI action, while RelayAct has very high efficiency as 0.93-0.95 GUI action per modeel call. RelayAct 3.8 and RelayAct 4 Argon have slightly higher efficiency than RelayAct 3.5 as they are more likely to produce >= 2 GUI actions in a single call (4.8% of 4 Argon vs 1.9% of 3.5). This difference illustrates how harnesses allocate model calls across planning, verification, and GUI execution. Such differences cannot be captured by counting GUI actions alone.

In addition, following the suggestion of evaluating the hybrid harness under different types of resource budgets, we conduct resource-budget sweeps across four hybrid configurations: SeeAct-V, EcoAgent, and RelayAct with Gemini 3.5 and 3.8. Specifically, we impose identical per-task time budgets (300--1,500,s) and device-energy budgets (500--2,500,J), and measure task success under each budget cap. The results show consistent performance trends across both resource types. When resource budgets are not severely constrained, RelayAct consistently outperforms the baselines, maintaining the main performance rankings observed under the model-call budget. These findings demonstrate that RelayAct's advantage is not specific to model-call constraints but remains robust under alternative resource budgets. Detailed results and analysis of other budget types have been provided in Appendix E.4 of the revised manuscript.

Dear Reviewer tTDK,

As the November 18 discussion deadline approaches, we would greatly appreciate your feedback on whether our revisions and additional experiments have adequately addressed your concerns, particularly regarding task coverage and evaluation stability, the impact of handoff policies, and comparisons under alternative resource budgets.

If any questions or concerns remain, we would be happy to provide further clarification before the discussion closes.

Thank you again for your time and consideration.

Comments on the paper

1 comment

Appreciation for the Reviewers' Feedback: We sincerely thank the reviewers for recognizing the practical value of Edge Symphony. By capturing real-device deployment costs often overlooked by existing benchmarks, Edge Symphony provides a more comprehensive basis for evaluating hybrid mobile agents. We also appreciate reviewers‘ recognition of the counterintuitive model–harness interactions that highlight the limitations of model-centric evaluation.

Contributions and Key Revisions: We have further strengthened the evidence for the distinct contributions of Edge Symphony relative to prior benchmarks. Edge Symphony reveals and quantifies how harness invocation efficiency, model placement, and control-transfer mechanisms shape realized task capability and deployment costs through end-to-end execution on physical smartphones. Our key revisions, highlighted in blue in the revised manuscript, include:

  1. Expanded benchmark (Section 3.2; Appendix E.1). We expanded the benchmark from 258 to 280 tasks, strengthening the two previously underrepresented capability categories with more diverse scenarios, broader application coverage, and improved evaluation stability.
  2. Comprehensive evaluation (Section 5.3; Appendix E.5). We extended the evaluation from a single smartphone and on-device model to three smartphones and four on-device VLMs, demonstrating native execution across different hardware and models. These experiments further support our finding that a stronger remote model does not necessarily yield better hybrid-agent performance.
  3. Stronger experimental evidence (Section 5.2; Appendix E.6). We reproduced Insight 3 across devices and on-device models, and conducted repeated evaluations and controlled experiments identifying handoff policy as a key factor behind the observed performance reversal.

Responses to Reviewer Concerns: We have also addressed each reviewer's specific concerns, as summarized below:

  • Reviewer tTDK. We improved task coverage and evaluation stability, provided controlled evidence of the impact of handoff policies, and validated the main performance rankings under alternative resource budgets.
  • Reviewer k9WU. We substantially expanded hardware and on-device model diversity, confirming the consistency of our findings across different configurations.
  • Reviewer otWa. We strengthened the distinction from prior benchmarks, refined our generality claims, and provided additional evidence addressing evaluation fairness and prompt-engineering effects.

We are grateful to all reviewers for the time and effort they have devoted to reviewing our work and considering our responses. We sincerely appreciate their careful consideration throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.