- Rating
- 2(−1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Edge Symphony is a harness-aware benchmark for hybrid mobile GUI agents: it runs 258 reproducible tasks across 23 apps in six capability categories on physical smartphones with native on-device inference, and jointly measures Smartness (task success, per-capability success, interaction efficiency) and Cost (latency, device energy, API spend). It also ships two reference harnesses, PocketAct (fully local) and RelayAct (hybrid). Its experiments show three effects invisible to model-centric evaluation: harness efficiency in converting model calls into GUI actions bounds realized capability, offloading relocates cost from API bills to device energy and latency rather than removing it, and a stronger remote model can yield a weaker hybrid agent depending on the harness's control-transfer protocol.
Critical strengths
-
Real on-device execution cost, measured rather than estimated. Prior mobile GUI benchmarks either keep inference off-device (GUI-CEval, MobileBench-OL run on real phones but in the cloud) or measure only isolated inference workloads (PalmBench-style). Edge Symphony runs LiteRT-LM natively on a physical phone and reads ODPM power-rail counters (
IPowerStatsHAL, cumulative µW·s deltas) to obtain latency, whole-device energy, and API spend from the same execution, under a synchronized window that excludes initialization and watchdog recovery. -
A counterintuitive finding only a benchmark could surface. 3.8-Flash beats 3.5-Flash as a standalone agent (72.1% vs. 70.5%) but loses under RelayAct (58.9% → 50.0%), because it follows the handback protocol more faithfully and returns more execution to the weaker on-device model. Realized capability is thus jointly determined by model and harness, and step-level traces show where control actually resides.
Critical weaknesses
-
Single-vendor remote models. Every run uses only Gemini (3.5-Flash, plus 3.8-Flash in two rows); GPT and Claude appear only in the pricing table, never in an experiment. Since the headline "stronger model, weaker agent" result is explained by protocol-following behavior that likely varies across vendors, its generality is untested.
-
Inflated novelty attribution. The paper frames "the harness as the object of evaluation" and "model and harness jointly determine capability" as primary contributions, yet ACE-Bench already compares edge-only, cloud-only, and edge–cloud placement, and Harness-Bench already concludes that capability must be reported at the model-harness configuration level. Read against these works, the contribution reduces to the first instantiation of those effects in a mobile GUI setting with on-device inference and measured energy — still valid, but a notch below the framing — and the paper never performs that demotion itself.
Questions for authors
Do the two reference harnesses satisfy system fairness? PocketAct and RelayAct beat every baseline, but both are the authors' own implementations, and Appendix D shows heavily engineered prompts covering failure modes such as loops, stalls, and false completion — yet no prompt-engineering ablation is reported. Reporting one — or running a third-party published harness on the same benchmark — would separate gains from harness design principles from gains from implementation maturity.
3 comments
Response to weakness 2.
Weakness 2. Inflated novelty attribution. We thank the reviewer for the careful comparison with ACE-Bench and Harness-Bench.
Major contribution. We agree with the reviewer that the major contribution of Edge Symphony lies in the benchmark itself. We do not regard the general principle that models and harnesses jointly determine capability and related findings as our primary contribution. Instead, Edge Symphony runs on-device models natively on physical smartphones, and executes hybrid harnesses on long-horizon GUI tasks. In addition to a change of setting, it enables analyses that require on-device measurement, such as practical end-to-end latency and whole-device energy of GUI agent task. To our knowledge, no existing benchmark provides such a testbed. The release of Edge Symphony will provide great value to the open-source community and developers in this field.
Related work. Both works have been already discussed in Section 2.2. We agree that their distinctions from Edge Symphony should be made more explicit. ACE-Bench, now EC²-Bench, compares edge–cloud collaboration strategies in terms of task performance, resource usage, and privacy exposure. It does not evaluate execution on physical smartphones: edge-side computation is estimated using FLOPs, and actual device energy and end-to-end latency are not measured. Harness-Bench evaluates model–harness configurations on sandboxed agent tasks, whereas Edge Symphony targets on hybrid harnesses which explicitly coordinate an on-device model and a remote model during GUI execution. We have revised the Abstract and Related Work to clarify these distinctions and position our contributions more precisely.
Response to Question
Question. System Fairness and Prompt Engineering Attribution We thank the reviewer for raising the question of system fairness and the role of prompt engineering.
Third-party harness comparison. The main table and figure in Section 5.2 have already included three published third-party harnesses, M3A, SeeAct-V, and EcoAgent, evaluated under the same benchmark protocol with shared base models and GUI action interfaces. We retain their original prompts without modification to preserve their respective harness designs, as role-specific prompts are themselves part of an agent harness.
Scripted control flow and role prompts. We clarify that RelayAct's failure-handling mechanisms are not implemented solely through prompt engineering but mainly from the scripted control flow. In RelayAct, repeated-action loops and subgoal stalls are detected by a scripted progress tracker, which triggers escalation to the remote Unblocker. Similarly, when the local Executor reports subgoal completion, the harness invokes the Verifier and routes corrective feedback if the claim is rejected. The role prompts provide supplementary guidance to help models follow these control-flow rules instead of the primary control. For example, the remote Unblocker's prompt explicitly instructs it to return an updated plan after resolving a stalled subgoal. Importantly, these are task-agnostic harness mechanisms but general control flow: they monitor and correct the agent's own execution behavior using action histories, progress states, and screen observations, without relying on benchmark-specific task categories or labels. We have added this clarification in Appendix D.3 of the revised manuscript.
RelayAct prompt-guardrail ablation. To distinguish the effects of supplementary prompt guidance from scripted control flow, we strip the Appendix D.3 guardrails across all four local and remote roles: the Local Verifier and Executor's anti-retry, non-interactive-element, and date-picker heuristics, the Cloud Planner's one-shot example, and the Cloud Unblocker's handback reminders. Only bare role definitions and coordinate schemas are retained. As shown in Table 6, removing these guardrails does not reduce overall success (57.2% vs.\ 60.7%). The handback rate drops substantially from 40.8% to 18.2%, suggesting that the removed guidance helps the remote Unblocker follow the intended control-transfer protocol. These results indicate that supplementary prompt guidance influences how models comply with harness rules, but RelayAct retains strong performance without these additional prompt-engineering components. We have added this ablation study in Appendix E.7 of the revised manuscript.
Table 6: RelayAct prompt-guardrail ablation. (Gemini 3.5-Flash)
| Ablation Variant | Overall SR (Δ vs. Default) | Handback Rate | Unblocker Share | Latency (s) | Energy (J) | API Cost ($) |
|---|---|---|---|---|---|---|
| Default | 57.2 ± 2.6% (–) | 40.8% | 60.2% | 418 | 1,184 | 0.195 |
| w/o Prompt Guardrails | 60.7% (+3.6 pts) | 18.2% | 63.7% | 414 | 1,172 | 0.210 |
w/o Prompt Guardrails: Remove the one-shot example, domain heuristics, and Cloud Unblocker handback warnings as described in Appendix D.3, retaining only bare role and coordinate schemas.
Response to weakness 1.
Weakness 1. Single-vendor remote model. Thank you for raising the concern about remote-model diversity and the generality of the third insight in Section 5.4. Our finding is that upgrading the model does not guarantee a better hybrid agent, not that a stronger model generally yields a weaker one. This finding does not depend on cross-vendor generality: a single counterexample of the Gemini family is sufficient to support it, even if protocol-following behavior varies across vendors. In the revision, we have clarified the corresponding statements in the third insight as ``observed in the tested Gemini configurations'' to make them more precise.
We agree that cross-vendor generality remains untested. What our data do establish is that the effect is a property of the control-transfer protocol rather than of one checkpoint: across three Gemini generations under Default RelayAct, handback rate rises monotonically (40.8% → 65.6% → 72.9%, Gemini3.5 → 3.8 → 4 Argon) while success falls (57.2% → 48.9% → 51.2%), and Sticky handoff reverses it (65.0% → 68.9% → 76.6%). Edge Symphony's LiteLLM-compatible interface runs any OpenAI/Anthropic/other API model under the identical protocol, so the cross-vendor test is directly reproducible by the community.