acceptodds
Under review as a conference paper at ICLR 2027

PhoneHarness: A Mixed-Action Orchestration Harness and Benchmark for Phone Agents across CLI, GUI, and MCP Tools

13 views

Abstract

Phone agents are increasingly expected to complete real mobile workflows rather than merely predict the next screen action. However, much of the current mobile-agent literature still evaluates agents primarily as GUI controllers that observe a screen, emit taps and swipes, and are scored by target app state. Real phone-use tasks are broader: they require deciding when to use app GUIs, device-side commands, or structured tools, while leaving evidence that the intended side effect actually occurred. We introduce PhoneHarness, a mixed-action benchmark and execution harness for studying phone-use agents on verifiable mobile workflows. PhoneHarness runs a device-side agent loop over GUI, CLI, and host-side tool actions, combining deterministic action routing with bounded GUI delegation and auditable execution traces. Its benchmark, PhoneHarness Bench, evaluates whether agents complete tasks with observable side effects, not only whether they produce plausible final answers. On the annotated evaluation split, PhoneHarness reaches a 75.0% pass rate, outperforming the strongest non-PhoneHarness settings by 12.9 percentage points. PhoneHarness and PhoneHarness Bench therefore play distinct but mutually dependent roles: the harness makes mixed phone workflows executable, while the benchmark measures whether agents can use that harness reliably and safely. Our findings suggest that reliable phone automation depends on action-surface routing and verifiable execution, not only visual GUI control.

Primary area:
datasets and benchmarks
Keywords:
phone-use agents, mixed-action orchestration, agent harness, mobile-agent benchmark, verifiable execution

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_doQAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

PhoneHarness combines CLI, GUI and host-side MCP tools for phone agents. It also introduces PhoneHarness Bench, which evaluates task completion using execution traces and observable changes such as files, messages, and calendar events. On its own 124-task evaluation split, PhoneHarness reports a 75.0% pass rate, compared with 62.1% for the strongest baseline settings. The paper further reports results by task type, model combination, runtime, and safety behavior.

Critical strengths
  1. Combining GUI interaction with CLI and structured tools is a useful direction for phone agents. The examples show practical workflows involving app navigation, data processing, and actions in external services.
  2. The paper recognizes that a plausible final answer is not enough to establish task completion. Recording both the outer agent trace and the delegated GUI trace, together with checks on resulting state, could support useful failure analysis.
Critical weaknesses
  1. I find the experimental validation insufficient. PhoneHarness is evaluated only on its own benchmark, whose tasks were selected to exercise mixed execution. There are no results on established benchmarks such as AndroidWorld or MobileWorld. Connecting the available tools to an existing benchmark, with clearly defined permissions and evaluation rules, would provide stronger evidence that the harness is useful beyond its own task set. The current comparisons also do not isolate the benefits of the harness design from those of additional CLI/MCP access. A baseline with the same model and tools, along with component ablations, is needed to support the routing claim.
  2. The benchmark and evaluation protocol are not described clearly enough. The relationships between the 181 candidate tasks, 14 mock-app tasks, 45 real-app tasks, 30 safety tasks, and 124 scored tasks are unclear. The paper emphasizes verifiable side effects, but provides limited detail about the actual pass conditions and whether they check the full task requirements rather than only the existence of an output. Baseline configurations and step-counting rules also need clarification. These gaps make the reported success and efficiency comparisons difficult to assess.
  3. The writing needs more careful checking. Especially, the related-work section only lists existing papers without explaining their differences or how they relate to the proposed contribution. For example, OS-Copilot is grouped under “Computer-use and execution-based benchmarks” although it is an agent framework rather than a benchmark. More broadly, the paper repeatedly uses terms such as “deterministic routing” without giving enough implementation detail to explain what is new or how these mechanisms work.
Questions for authors
  1. Can the authors evaluate PhoneHarness on an existing benchmark such as AndroidWorld or MobileWorld to show whether its benefits extend beyond its own task set?
  2. Does the performance improvement mainly come from the additional CLI/MCP tools or from PhoneHarness’s orchestration design? Can the authors clarify this through a controlled comparison with the same tool access?
  3. Can the authors explain how the different task subsets relate to the final 124 evaluation tasks and provide one or two task examples with complete success criteria?
Reviewer_LUXGOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents PhoneHarness, a framework for supporting and evaluating realistic and verifiable mobile-device workflows by integrating GUI, CLI, and host-side tools, with deterministic action routing and auditable execution traces.

Critical strengths

1 Beyond GUI-only evaluation: It goes beyond traditional evaluation limited to GUI-based visual control, emphasizing real-world side effects and verifiability.

2 Strong empirical support: It yields significant performance gains over settings without the framework.

Critical weaknesses

1 The paper does not currently state clearly whether the framework generalizes across different mobile operating systems (e.g., iOS and Android).

2 Combining GUI and MCP is already very common, and the contributions claimed in this paper do not hold up.

3 The benchmark does not describe in sufficient detail the breadth of tasks and the types of applications it covers.

Questions for authors

Please provide a detailed breakdown of the tasks in PhoneHarness Bench, including their specific categories, counts, and complexity distribution.

Please further expand the survey of related work and clarify how it relates to this work.

Reviewer_tKBfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PhoneHarness integrates device-side commands, delegated GUI interaction, and host-side tools into a common phone-agent execution loop. It uses deterministic-first routing, progressive loading of tool instructions, and execution traces to support workflows with observable side effects. The accompanying benchmark evaluates task completion through artifacts, application state, and other execution evidence. On the 124-task annotated split, the proposed system achieves a 75.0% pass rate, compared with 62.1% for the strongest reported alternatives. Additional experiments examine task categories, model combinations, execution cost, and exploratory safety behavior.

Critical strengths
  1. The execution setting addresses a practical problem. Many phone workflows naturally combine application interaction, local processing, and external services. Supporting these routes within one auditable system is useful, and the trace examples make the intended workflows concrete.

  2. The task breakdown provides a useful qualification of the headline result. The gains concentrate in device/system operations and tool-assisted workflows. The authors also report that PhoneHarness trails standalone Seed2.0-Pro on single-app GUI tasks, 63.3% versus 76.7%. This helps clarify which capabilities the system actually improves.

Critical weaknesses
  1. The paper describes categories of verification evidence, but does not systematically establish verifier correctness or agreement with independent assessment. Artifact existence, for example, need not establish that the artifact contains the requested content. The relationship between the mock-app, real-app, safety, and 124-task scored subsets remains unclear, as does the auxiliary runtime split. Essential settings—including reset procedures, retry policies, execution budgets, and application versions—are discussed as release requirements without being fully specified. These omissions make the reported pass rates difficult to reproduce and interpret.

  2. The main comparison changes available action routes and orchestration, so the gain cannot be attributed specifically to deterministic-first routing, bounded delegation, or progressive skill disclosure. Using Seed2.0-Pro both inside the harness and as a standalone baseline is helpful, but does not control tool access. MobileClaw’s model configuration and matched capabilities are also insufficiently specified.

  3. Also Table 7 varies models within PhoneHarness rather than testing the harness components. The results establish that the combined system works better on this task mix, but do not explain which design choices are responsible.

Questions for authors
  1. With the same controller, GUI worker, tool inventory, and execution budget, how much improvement remains when routing, delegation, and progressive skill disclosure are individually ablated?

  2. Can you provide the exact task memberships, executable success criteria, verifier-validation results, and environment settings for the headline and auxiliary evaluations?

  3. How is the safety refusal rate computed? In particular, how are “safe complete,” “confirm first,” and “never auto” tasks distinguished, and how are unnecessary refusals penalized?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.