- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
PolyAgent is two things. First, data: native translators localize each query culturally, experts co-design and audit the schema annotations, an automated pass filters syntax and parameter-type errors, and a double-blind expert review signs off — yielding queries in six languages over 47 real mobile OS APIs, built by localization rather than machine translation. Second, evaluation: each query runs inside ToolSandbox's stateful sandbox from a pre-populated initial state, the model calls the APIs on its own for up to three turns, and two metrics are reported — Trajectory Precision for whether the first attempt picked the right API with the right arguments, and Goal Success Rate for whether the final sandbox state actually matches the target, checked by database diffs plus an LLM judge for the fuzzy cases.
Critical strengths
The gap is well-posed. It occupies a genuinely empty intersection: multilingual function-calling benchmarks test language-to-tool mapping without persistent execution, while mobile-agent benchmarks (AndroidWorld, SPA-Bench, MobileWorld) are English/Chinese-centric and measure general task completion rather than localized intent-to-argument grounding. Framing the artifact as a diagnostic benchmark rather than a leaderboard is also the right call.
Critical weaknesses
-
It borrows ToolSandbox's shell but removes the parts that make it hard. Sec. 4.1's Pre-conditioned State Guarantee explicitly pre-populates every ground-truth argument value in S₀, which neutralizes ToolSandbox's signature difficulties (state dependency, insufficient information). Combined with single-turn intents throughout, no user simulator, and at most three retries, the "closed loop" amounts to function-call retry — a defensible tradeoff for clean attribution, but not stateful interactive evaluation in ToolSandbox's sense.
-
The core "on-device" selling point is unsupported. The paper repeatedly claims on-device / edge-deployed / resource-constrained, yet reports zero latency, memory, quantization, or inference-backend detail, so "edge" reduces to "small parameter count." The premise motivating the entire paper is never actually validated.
-
The framing oversells the domain. It positions itself within interactive mobile-agent evaluation, yet methodologically its siblings are MASSIVE-Agents and ToolSandbox, not SPA-Bench or MobileWorld — no emulator, no UI, no app ecosystem, no device; "mobile" contributes only the provenance of the API schemas, the content of the queries, and a mocked state. The real contribution is multilingual state-backed function calling, a narrower and more defensible claim than the one being sold.
Questions for authors
What does "integrated with ToolSandbox" concretely mean? Did you register the 47 APIs as @register_as_tool functions inside its Execution Context and reuse its snapshot/milestone machinery, or did you only borrow the name? Please specify the ToolSandbox version or commit. As written, the central environmental claim is not checkable.