- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
PhoneHarness combines CLI, GUI and host-side MCP tools for phone agents. It also introduces PhoneHarness Bench, which evaluates task completion using execution traces and observable changes such as files, messages, and calendar events. On its own 124-task evaluation split, PhoneHarness reports a 75.0% pass rate, compared with 62.1% for the strongest baseline settings. The paper further reports results by task type, model combination, runtime, and safety behavior.
Critical strengths
- Combining GUI interaction with CLI and structured tools is a useful direction for phone agents. The examples show practical workflows involving app navigation, data processing, and actions in external services.
- The paper recognizes that a plausible final answer is not enough to establish task completion. Recording both the outer agent trace and the delegated GUI trace, together with checks on resulting state, could support useful failure analysis.
Critical weaknesses
- I find the experimental validation insufficient. PhoneHarness is evaluated only on its own benchmark, whose tasks were selected to exercise mixed execution. There are no results on established benchmarks such as AndroidWorld or MobileWorld. Connecting the available tools to an existing benchmark, with clearly defined permissions and evaluation rules, would provide stronger evidence that the harness is useful beyond its own task set. The current comparisons also do not isolate the benefits of the harness design from those of additional CLI/MCP access. A baseline with the same model and tools, along with component ablations, is needed to support the routing claim.
- The benchmark and evaluation protocol are not described clearly enough. The relationships between the 181 candidate tasks, 14 mock-app tasks, 45 real-app tasks, 30 safety tasks, and 124 scored tasks are unclear. The paper emphasizes verifiable side effects, but provides limited detail about the actual pass conditions and whether they check the full task requirements rather than only the existence of an output. Baseline configurations and step-counting rules also need clarification. These gaps make the reported success and efficiency comparisons difficult to assess.
- The writing needs more careful checking. Especially, the related-work section only lists existing papers without explaining their differences or how they relate to the proposed contribution. For example, OS-Copilot is grouped under “Computer-use and execution-based benchmarks” although it is an agent framework rather than a benchmark. More broadly, the paper repeatedly uses terms such as “deterministic routing” without giving enough implementation detail to explain what is new or how these mechanisms work.
Questions for authors
- Can the authors evaluate PhoneHarness on an existing benchmark such as AndroidWorld or MobileWorld to show whether its benefits extend beyond its own task set?
- Does the performance improvement mainly come from the additional CLI/MCP tools or from PhoneHarness’s orchestration design? Can the authors clarify this through a controlled comparison with the same tool access?
- Can the authors explain how the different task subsets relate to the final 124 evaluation tasks and provide one or two task examples with complete success criteria?