acceptodds
Under review as a conference paper at ICLR 2027

PolyAgent: A Multilingual Benchmark for API-Driven Mobile Device Control

Abstract

Small Language Models (SLMs) increasingly enable mobile agents that translate natural-language intent into executable device and application actions. While recent benchmarks have advanced multilingual function calling, stateful tool use, and interactive mobile-agent evaluation, these capabilities remain only partially integrated. In particular, localized multilingual intent grounding for executable mobile APIs under dynamic environment feedback is insufficiently characterized. We introduce PolyAgent, a multilingual diagnostic benchmark for API-driven on-device mobile control. PolyAgent contains 5,400+ expert-validated queries across 6 typologically diverse languages and 47 real-world mobile APIs. A culturally grounded synthesis and native-expert validation pipeline targets localized intent and argument grounding rather than direct translation alone. Integrated with ToolSandbox, PolyAgent evaluates interactive execution through Trajectory Precision and Goal Success Rate (GSR), combining action-level fidelity with environment-state verification. Experiments across lightweight open-source models reveal substantial model- and language-dependent variation under dynamic execution: Qwen models exhibit pronounced degradation for several non-English languages, whereas Gemma models maintain comparatively stable cross-lingual performance while achieving strong overall accuracy. These results highlight the limitations of static or English-centric evaluation for multilingual mobile tool use. PolyAgent provides a unified diagnostic framework for evaluating localized, state-aware, and resource-constrained mobile agents.

Primary area:
datasets and benchmarks
Keywords:
PolyAgent, API-driven, Mobile Autonomous Agents, Multilingual Benchmark, Cross-lingual Performance

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_SHo6OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

PolyAgent is two things. First, data: native translators localize each query culturally, experts co-design and audit the schema annotations, an automated pass filters syntax and parameter-type errors, and a double-blind expert review signs off — yielding queries in six languages over 47 real mobile OS APIs, built by localization rather than machine translation. Second, evaluation: each query runs inside ToolSandbox's stateful sandbox from a pre-populated initial state, the model calls the APIs on its own for up to three turns, and two metrics are reported — Trajectory Precision for whether the first attempt picked the right API with the right arguments, and Goal Success Rate for whether the final sandbox state actually matches the target, checked by database diffs plus an LLM judge for the fuzzy cases.

Critical strengths

The gap is well-posed. It occupies a genuinely empty intersection: multilingual function-calling benchmarks test language-to-tool mapping without persistent execution, while mobile-agent benchmarks (AndroidWorld, SPA-Bench, MobileWorld) are English/Chinese-centric and measure general task completion rather than localized intent-to-argument grounding. Framing the artifact as a diagnostic benchmark rather than a leaderboard is also the right call.

Critical weaknesses
  1. It borrows ToolSandbox's shell but removes the parts that make it hard. Sec. 4.1's Pre-conditioned State Guarantee explicitly pre-populates every ground-truth argument value in S₀, which neutralizes ToolSandbox's signature difficulties (state dependency, insufficient information). Combined with single-turn intents throughout, no user simulator, and at most three retries, the "closed loop" amounts to function-call retry — a defensible tradeoff for clean attribution, but not stateful interactive evaluation in ToolSandbox's sense.

  2. The core "on-device" selling point is unsupported. The paper repeatedly claims on-device / edge-deployed / resource-constrained, yet reports zero latency, memory, quantization, or inference-backend detail, so "edge" reduces to "small parameter count." The premise motivating the entire paper is never actually validated.

  3. The framing oversells the domain. It positions itself within interactive mobile-agent evaluation, yet methodologically its siblings are MASSIVE-Agents and ToolSandbox, not SPA-Bench or MobileWorld — no emulator, no UI, no app ecosystem, no device; "mobile" contributes only the provenance of the API schemas, the content of the queries, and a mocked state. The real contribution is multilingual state-backed function calling, a narrower and more defensible claim than the one being sold.

Questions for authors

What does "integrated with ToolSandbox" concretely mean? Did you register the 47 APIs as @register_as_tool functions inside its Execution Context and reuse its snapshot/milestone machinery, or did you only borrow the name? Please specify the ToolSandbox version or commit. As written, the central environmental claim is not checkable.

Reviewer_k4ddOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PolyAgent, a benchmark comprising over 5,400 API-driven mobile control tasks covering 6 languages and 47 real-world mobile APIs. These tasks were derived from English and have been verified by native experts.The paper evaluated this benchmark on small LLMs from the Qwen and Gemma series and found that the Qwen models exhibited a significant performance drop in non-English scenarios, while the Gemma models remained largely stable. PolyAgent provides a unified diagnostic framework for evaluating localized, state-aware, and resource-constrained mobile agents.

Critical strengths

Localization strategies involve more than just using a translation tool to directly translate English tasks; instead, they involve designing a human-in-the-loop localization pipeline that better aligns with real-world scenarios, effectively enhancing the benchmark’s reliability.

Not only reports the overall accuracy, but also compares the performance of different models and languages, and further analyzes failure modes. Such analysis in the paper has diagnostic value and provides specific directions for improving multilingual tool-calling models.

Critical weaknesses

The paper still does not provide a clear enough description of the details of the four-stage annotation process. For example, it lacks an explanation of the sources of the seed queries, specific case studies (such as what “local ride-hailing/banking apps” correspond to in different languages), and more specific implementation details for Automated QA.

The evaluation includes only five models from two families, Qwen and Gemma.Although this benchmark is primarily limited to on-device scenarios, which significantly narrows the range of applicable models, evaluating it on a broader set of small LMs (such as the MiniCPM series) could effectively enhance the robustness of the results presented in this paper.

Questions for authors

Could you provide some cases to illustrate how a model's performance on a given task is affected by different languages?

Could you provide a more detailed explanation of the four-stage annotation process, including, but not limited to, the source of the seed queries, the instructions provided in the Expert-Audited Schema Annotation, and the specific process for Automated QA?

Reviewer_wcVAOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes PolyAgent, a multilingual diagnostic benchmark for API-driven on-device mobile control. The dataset contains 5,458 queries covering 47 APIs across seven functional domains and six target languages: Arabic, Thai, Latin American Spanish, Indonesian, Tagalog, and Malay. The construction pipeline includes localization by native translators, LLM-generated candidate function calls, automated checks, and expert validation. Each task includes a predefined environment state, a canonical API invocation, and a target goal state, with up to three execution steps allowed in ToolSandbox.

Critical strengths
  • The paper addresses a valuable practical problem. Mobile commands involve localized application names, contacts, time expressions, and fixed enumeration values, which are not adequately covered by English-only function-calling evaluations.

  • The dataset emphasizes localization rather than direct translation. The involvement of native translators and expert validators helps ensure data quality.

Critical weaknesses
  • The benchmark consists of single-turn user intents that allow multiple execution attempts and error recovery, rather than long-horizon mobile workflows involving multiple tasks. This limits task complexity, as also suggested by the relatively high GSR already achieved by some lightweight models.

  • The evaluation does not clearly disentangle language-related performance gaps from differences in task content and difficulty. Moreover, the observed argument errors, cross-lingual degradation, and failures to handle feedback may primarily reflect the limitations of particular model families or lightweight models.

  • The benchmark appears to contain a limited range of challenging tasks. Gemma-4-31B-it already achieves an average GSR of 93.49% across the target languages. Stronger models may perform close to perfectly, potentially limiting the benchmark's ability to distinguish their capabilities and its practical value.

  • While the focus on lightweight, on-device models is reasonable, the evaluation covers only two model families, with Qwen-3.6-27B and Gemma-4-31B-it serving as larger-scale controls. The absence of frontier-model baselines makes it difficult to assess the benchmark's remaining headroom and whether the reported multilingual and feedback-handling failures extend beyond the evaluated families.

Questions for authors
  • Given that the tasks involve single-turn user intents with at most three execution steps, to what extent can the benchmark assess the challenges of longer, multi-task mobile workflows? Are there task subsets that remain particularly challenging even for the strongest evaluated model?

  • How are task content and difficulty controlled across languages? Could you provide an analysis that separates the effects of language from differences in the localized tasks?

  • Could you evaluate stronger frontier models to assess the benchmark's remaining headroom, given that Gemma-4-31B-it already achieves a target-language average GSR of 93.49%?

  • Do the reported argument errors, cross-lingual degradation, and feedback-handling failures persist across additional model families and stronger models, or are they largely specific to the evaluated models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.