acceptodds
Under review as a conference paper at ICLR 2027

OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools

13 views

Abstract

We introduce OpenMobile-2, a near-frontier mobile agent with fully open training environments and recipes. We make three key advances: (1) *Diverse environments with simulated commercial apps*: We build **MobileGym++**, featuring 35 realistic, functionally rich commercial-style apps with cross-app workflows, while preserving full controllability for reset and verification. Together with newly configured apps in Android emulators, we establish a diverse training playground spanning over 110 apps. (2) *Open training data at scale*: Building on this foundation, we curate 12K mobile interaction trajectories for supervised fine-tuning and the largest open collection of verifiable RL training data for mobile agents, comprising over 2K executable tasks with automatic rewards. (3) *Hybrid GUI and app-native tool use*: We explore an experimental mobile-use setting where apps expose selected functionalities as app-native tools alongside their GUIs, allowing agents to interleave GUI actions and tool calls within a task. We implement this setting in MobileGym++ with over 300 carefully scoped tools across 50+ apps. Additionally, we introduce **MobileGym++ Bench** with realistic, long-horizon tasks over commercial-style apps, supporting both GUI-only and hybrid GUI–tool evaluation under a unified task suite. OpenMobile-2 performs competitively across established benchmarks, including AndroidWorld (79.9) and MobileWorld (50.4), while showing promising transfer to real-device mobile use, nearly doubling SPA-Bench performance from 31.9 to 59.6. All resources are available at https://anonymous.4open.science/r/openmobile2.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Mobile Agents, Computer-Use Agents, GUI Agents, Vision-Language Models, Tool-Use Agents

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_DqVaOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper releases an open stack for mobile GUI agents. MobileGym++ adds 35 replicas of commercial apps to the MobileGym simulator, for 63 clients, or over 110 apps with the emulator pool; app state can be reset and checked by program. The authors collect 11,643 SFT trajectories with Gemini-3.1-Pro as the teacher (1,774 of them mix GUI actions with app-native tool calls) and 2,475 RL tasks with verifiers. Over 300 tools write the same state as the GUI, and MobileGym++ Bench (215 tasks) scores GUI-only and hybrid agents on identical tasks. Fine-tuned Qwen3.5-9B and Qwen3.6-27B models reach 79.9 on AndroidWorld, 50.4 on MobileWorld and 59.6 on SPA-Bench L3 (base: 31.9).

Critical strengths

S1. A complete open stack. State can be saved and restored, and checkers read business state, not screenshots (L196-200, L279-282). Data, tools, hyperparameters (Table 5), code (L507) and a reusable app-building recipe (App. B) are provided.

S2. Large SFT gains, also on physical devices. AndroidWorld rises from 54.3 ± 3.7 to 73.3 (9B) and from 67.1 ± 4.1 to 80.8 (27B), far beyond the seed spread (Table 2). On SPA-Bench, hybrid SFT lifts the 9B model from 25.5 to 48.9 and the 27B model from 31.9 to 57.4 (L408-413).

S3. A clean test of tools against the GUI, with a useful finding. Tools write the same store as their GUI paths and are judged on the same tasks and checkers (L253-257). With the same weights, tools add 2.8-4.2 points before hybrid SFT and 5.6-10.3 after it (Fig. 5b); at 27B that is 22 more of 215 tasks. Tool access alone is not enough.

Critical weaknesses

W1. The experiments do not show what the new simulated apps add. Emulator data is most of the SFT mix (7,725 of 11,643 trajectories, Table 6). The emulator pool holds all the AndroidWorld apps (Table 10), and App. D.3 reports AndroidWorld, MobileWorld and MobileGym training subsets (L1024-1025). Both SFT runs include the 2,144 GUI simulator trajectories, and the coverage study uses emulator data only (L430-431), so no experiment removes MobileGym++. The gains on AndroidWorld and MobileWorld could come mostly from emulator data on those benchmarks' own apps. An emulator-only SFT run at 9B would settle this, and a same-base run on the OpenMobile-1 data would isolate the new data.

W2. The generalization and real-device claims go beyond the evidence. Four of the five benchmarks share a runtime with training (L844-845), which weakens the argument at L802-804. The student beats its teacher on AndroidWorld (79.9 vs 70.7) but not on MobileWorld (50.4 vs 58.1) or SPA-Bench (59.6 vs 70.2; Table 2), as expected if it has seen the AndroidWorld apps. That leaves SPA-Bench, which is thin. It is left out of the overlap check (L850-851), "L3" is not defined, and all its values fit a 47-task set (28/47 = 59.6%). There is no spread, only one open-weight baseline, and no GUI-only SFT run. SPA-Bench's Chinese apps also include Taobao, JD, Meituan, Qunar, Cainiao and Alipay (Chen et al., 2025), all of which are in MobileGym++ (Table 9). If L3 uses them, the result is sim-to-real transfer on replicated apps. That is valuable, and it could supply the fidelity check the replicas lack (L206-215), but it is not transfer beyond the training environments (L362-363).

W3. The RL gains are within noise, and RL was validated on a test set. RL changes AndroidWorld by +0.7 and -0.9 points against seed spreads of 1.5-3.5 (Table 2), SPA-Bench by one task (27 to 28 of 47), and MobileGym by 6 and 7 of 256 tasks (L403-405). Those 256 tasks are also the RL validation set, run every 10 steps (L771-772; Table 1 lists 256 MobileGym tasks, and 98/256 = 38.3%). Training stopped at 150 of 300 budgeted steps (Table 5). If that checkpoint was picked on this curve, the MobileGym gain was selected on the test set. The RL tasks are a headline contribution that "largely advance training" (L115-117), and this run is their only evidence. Selection on held-out RL tasks and three evaluation runs would fix this; otherwise, RL should be framed as exploratory.

Questions for authors

Q1. What does SFT on emulator data alone give on AndroidWorld, MobileWorld and SPA-Bench, compared with the GUI-only run?

Q2. Which tasks form SPA-Bench "L3", and which apps do they use? How do results split between apps inside and outside the training pool? Which judge decides success?

Q3. Was the RL checkpoint chosen on the MobileGym validation curve?

4 comments
AuthorsOpenReview ↗

Response to W1&Q1 (Beyond Emulator Data: Gains from Simulated Apps)

W1&Q1 Beyond Emulator Data: Gains from Simulated Apps

Response1:New simulated apps provide modest gains on AndroidWorld and MobileWorld, while substantially improving real-device generalization.

We thank the reviewer for raising this question, which motivates us to further investigate the contribution of simulated apps and the overall contributions of OpenMobile-2 through controlled comparisons, including an emulator-only SFT run and an SFT run using OpenMobile-1 data with the same base model. We conduct all additional SFT experiments using Qwen3.5-9B as the base model to ensure a consistent comparison across training data configurations. The results are reported below, where MG++ denotes MobileGym++.

SettingAndroidWorldMobileWorld (GUI)MobileGymMG++ (GUI)MG++ (Hybrid)SPA-Bench
Base54.3 ± 3.712.89.49.312.125.5
SFT on OpenMobile-166.4 ± 1.720.520.320.920.025.5
SFT on Emulator71.3 ± 3.236.830.533.533.029.8
GUI-only SFT72.8 ± 2.2 (+1.5)38.5 (+1.7)34.8 (+4.3)40.9 (+7.2)41.9 (+8.9)48.9 (+19.1)

As expected, emulator data accounts for a substantial portion of the gains on AndroidWorld and MobileWorld. However, emulator-only training shows limited transfer to real-world mobile applications, partly because commercial apps are unavailable or their functionality is insufficiently covered due to constraints such as authentication. Simulated apps provide complementary training signals that improve cross-app generalization beyond emulator data, while also yielding substantial gains on SPA-Bench, which evaluates agents on real commercial applications. The comparison with OpenMobile-1 further demonstrates the benefits of the expanded training data in OpenMobile-2.

AuthorsOpenReview ↗

Response to W2&Q2 (Does the improvement on SPA-Bench reflect sim-to-real transfer or generalization from simulated environments to real devices?)

W2&Q2 Does the improvement on SPA-Bench reflect sim-to-real transfer or generalization from simulated environments to real devices?

We thank the reviewer for pointing out this issue. We will provide details of the SPA-Bench evaluation setup and further investigate the factors underlying our performance on SPA-Bench.

Response1: SPA-Bench L3 covers both seen and held-out apps

We follow the official SPA-Bench repository (ai-agents-2030/SPA-Bench, commit 4c292b6) and its data/single-app-CHN.csv. Among the 50 official tasks with difficulty level 3 (the highest), we evaluate 47, excluding three (huaweiliulanqi_2, huaweiliulanqi_5, and settings_2) due to environment incompatibilities: Xiaomi Browser requires phone-number authentication for web access, while Xiaomi Settings does not support Huawei-specific tasks.

Among the 47 evaluated tasks, 29 involve 15 apps included in our training pool, while 18 involve 10 held-out apps: Chinese Perpetual Calendar, China Telecom, Douyin, Umetrip, Haodf, QQ Music, Tencent Docs, Peking Union Medical College Hospital, Youdao Dictionary, and Zhihu. This split enables evaluation on real devices across both seen and unseen apps.

Response2: Our models improve performance on both the held-in and held-out splits of SPA-Bench under the official evaluation protocol.

We use SPA-Bench's official fine-grained evaluation prompt with gpt-5.6-sol as the judge model, providing all screenshots from each task in chronological order.

ModelTasks on Seen Apps (29)Tasks on Held-out Apps (18)All Tasks (47)
Qwen3.5-9B27.6 (8)22.2 (4)25.5 (12)
OpenMobile-2-9B51.7 (15)44.4 (8)48.9 (23)
Qwen3.6-27B34.5 (10)27.8 (5)31.9 (15)
OpenMobile-2-27B62.1 (18)55.6 (10)59.6 (28)

As shown in the table, our models achieve substantial gains on both subsets. These results provide evidence of sim-to-real transfer on apps represented in our training environments, while also supporting generalization to unseen apps.

AuthorsOpenReview ↗

Response to W3&Q3 (Unstable RL gains)

W3&Q3 Unstable RL gains

We thank the reviewer for pointing out these concerns. We clarify the intended scope of our claims about RL in the paper and provide additional experiments to better characterize the actual contributions of RL.

Response1: On the Role and Effectiveness of RL in OpenMobile-2

First, we would like to clarify that our claim that OpenMobile-2 “largely advances training” (L115–117) primarily refers to the released training data, executable environments, and training recipes, rather than solely to the RL gains reported in our experiments.

We primarily use MobileGym to monitor RL training because its task distribution closely matches that of our RL training tasks, and its tasks and evaluation protocol have been validated to be reliable. Meanwhile, our paper also reports results on other established benchmarks beyond MobileGym. We appreciate the reviewer’s point and will revise the relevant discussion of MobileGym in the paper to better clarify its role.

We further report results from three evaluation runs on MobileGym++ and SPA-Bench. Overall, RL yields modest improvements over Hybrid SFT on MobileGym++ for both the 9B and 27B models, although the gains vary across evaluation runs, reflecting the inherent instability of RL training.

ModelMobileGym++ (GUI-only)MobileGym++ (Hybrid)SPA-Bench
9B-Hybrid SFT38.6 ± 1.443.8 ± 1.047.3 ± 3.3
9B-RL40.0 ± 1.2 (+1.4)46.7 ± 2.2 (+2.9)49.7 ± 3.3 (+2.4)
27B-Hybrid SFT54.9 ± 1.765.1 ± 1.258.2 ± 3.3
27B-RL56.4 ± 1.8 (+1.5)66.7 ± 1.0 (+1.6)60.3 ± 1.2 (+2.1)

Response2: RL Exploration from the Base Model

We acknowledge that the current RL gains over the fully trained SFT baseline remain limited, and are continuing to investigate the potential of RL to further improve agent performance.

Specifically, we sample 128 additional MobileGym++ RL tasks from the 2K+ executable RL tasks released in our paper, excluding the 512 tasks used for RL training in the original experiment. We use these 128 tasks as a validation set for the new RL experiment, following the same training configuration as in the paper.

Due to time and computational resource constraints, we run RL training for only 60 steps. We report the validation performance every 10 steps, along with the corresponding evaluation results on MobileGym.

BaseStep 10Step 20Step 30Step 40Step 50Step 60
128-task Val.6.310.27.013.315.614.818.0
MobileGym9.412.111.316.019.519.120.3

Notably, the MobileGym performance improves from an initial score of 9.4 to 20.3, indicating that RL training can yield substantial gains from a relatively low starting point. This further demonstrates the potential of our environments to support RL training.

AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer DqVa,

Thank you again for your detailed and constructive feedback.

We would like to kindly follow up on our responses to your concerns regarding the contributions of simulated apps, sim-to-real generalization and generalization beyond the training environments, and the effectiveness of RL.

We have conducted additional controlled SFT experiments to distinguish the contributions of emulator data and simulated apps, showing that simulated apps provide complementary gains beyond emulator training. We have also clarified the SPA-Bench evaluation setup and reported results separately on seen and held-out apps.

Regarding RL, we have provided repeated evaluations to characterize its variability and conducted an additional 60-step RL experiment starting from the base model. The MobileGym score improves from 9.4 to 20.3, indicating that RL training can yield substantial gains from a relatively low starting point. This further demonstrates the capability of our environments to support RL training.

We would be grateful if you could share whether our responses and additional experiments have addressed your main concerns. We would be happy to clarify any remaining issues.

Thank you very much for your time and consideration.

Best regards,

The Authors

Reviewer_otE1OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
3(+1)
Read review
Paper summary

This paper introduces a series of recipes for training and evaluating mobile agents. First, it presents a controllable environment with 35 simulated, realistic commercial apps. Using these apps, the authors collect and open-source 12k mobile interaction SFT trajectories and 2k tasks with automatic rewards. Finally, they introduce a hybrid method that exposes app-native tool calls alongside GUI actions. The authors further train models using the SFT data and RL tasks through multiple stages: SFT, Hybrid SFT, and then RL. They compare these models against popular models on their own benchmark and other major mobile benchmarks and report the results.

Overall, the motivation and contributions are clear: to open-source SFT data, tasks, and environments for training and benchmarking mobile agents. However, concerns remain regarding the details and validity of the approaches, which require further clarification and improvement.

Critical strengths
  • The motivation is clear: to contribute SFT data, RL tasks, and benchmarks to facilitate mobile agent research. The value to the open-source community would be undeniable if further clarification and verification demonstrate that the work provides sufficient new contributions addressing gaps in existing work.
  • The infrastructure and architectural choices address a real pain point for mobile agents: real-world apps on physical phones generally lack controllability and state examinability, while existing simulated environments are scarce and lack fidelity. The proposed environment is claimed to provide high-fidelity commercial apps and data that address this problem, which is promising.
Critical weaknesses
  • A foundational question affecting the quality of this work is whether the environment meets the claimed high fidelity and commercial quality. It would be helpful to examine the actual environment, data, and tasks to assess this, but the provided link does not work. Its ecological validity and alignment with real-world mobile phones and apps also remain unclear, particularly regarding dynamic content and errors, which are often absent from simulated environments.
  • It is unclear how these data and tasks differ from existing work and whether they are significantly better. Could the same performance be achieved by training on previously open-sourced data? Ablations and discussion are needed to justify the need for another such contribution.
Questions for authors

There are a couple of key questions that need to be addressed, and I'm willing to raise my score if further clarification/info are provided for them.

  • Most importantly, the provided link does not work; read access to the original GitHub repository appears to be unavailable. Please check and fix this. For a dataset/benchmark/environment paper, examining real examples is important for assessing its quality and understanding the overall work.
  • Prior work has open-sourced data and environments. How do you justify that your data, environment, and tasks are better and provide significant value for training an effective mobile agent? For example, could SFT on android-in-the-wild/androidcontrol data achieve the same performance? Alternatively, can you provide a comparison or justification for why this SFT data is better?
  • Following up on the previous question, more recent work has open-sourced SFT data, tasks, and environments covering both real-world and simulated apps, such as DigiData[1] (mobile data and tasks), OpenApps[2] (simulated environment), and DigiWorld[3] (mobile environment with full controllability). How do these differ from this work, and what are their respective pros and cons? Please consider comparing and discussing these works and justifying the additional value this work provides.
  • In Table 2, why are SPA-Bench results missing for the GUI-only SFT OpenMobile-2 models?

Reference

[1] Y. Sun et al., “DigiData: Training and evaluating general-purpose mobile control agents,” arXiv [cs.AI], Nov. 10, 2025. doi: 10.48550/arXiv.2511.07413.

[2] K. Ullrich et al., “OpenApps: Simulating environment variations to measure UI-agent reliability,” arXiv [cs.AI], Nov. 25, 2025. doi: 10.48550/arXiv.2511.20766.

[3] P. D’Oro et al., “Computer use at the edge of the statistical precipice,” arXiv [cs.SE], May 07, 2026. doi: 10.48550/arXiv.2605.08261.

8 comments
AuthorsOpenReview ↗

Response to W1&Q1 (The provided link does not work)

W1&Q1 The provided link does not work

Response:

We apologize that the anonymous repository link became inaccessible due to an issue with our version management. We have now restored public access. We sincerely invite you to try out our environment, which we believe is one of the best simulated mobile-agent environments available, offering a strong combination of high fidelity, functional diversity, visual quality, and efficiency. Following the provided guidelines, the environment can be configured and launched with a single command. We hope you find it useful and welcome any feedback. https://anonymous.4open.science/r/openmobile2/README.md

AuthorsOpenReview ↗

Response to W2&Q2&Q3 (Necessity and Added Value of OpenMobile-2 over Existing Data and Environments)

W2&Q2&Q3 Necessity and Added Value of OpenMobile-2 over Existing Data and Environments

Response1 to Q2: The advantages of our data over existing datasets

We acknowledge the important contributions of AitW and AndroidControl, which established foundational resources for GUI-agent research and enabled early progress through large-scale human interaction trajectories, particularly on static tasks. Building upon these efforts, OpenMobile-2 advances mobile-agent data construction in several aspects.

  1. Longer-horizon demonstrations. Our 9,869 GUI-only demonstrations average 10.8 steps per trajectory, substantially longer than AitW (6.5) and AndroidControl (5.5).
  2. Additional reasoning supervision. We incorporate explicit reasoning supervision to support long-horizon planning and execution.
  3. Hybrid SFT data with app-native tools. Beyond conventional GUI interaction trajectories, we construct 1,774 hybrid SFT demonstrations in the MobileGym++ environment, integrating app-native tools into GUI interaction workflows.

We fine-tuned the Qwen3.5-9B base model on AndroidControl and evaluated it on AndroidWorld, but observed no substantial improvement (51.7), within the variance range reported in our paper. SoLS[1] also notes that SFT on AndroidControl can transfer to AndroidWorld, but the gains are limited.

[1]Papoudakis, Georgios, et al. "Succeed or Learn Slowly: Sample Efficient Off-Policy Reinforcement Learning for Mobile App Control." Advances in Neural Information Processing Systems 38 (2026): 125031-125054.

Response2 to Q3: Comprehensive training data, rich and controllable environments, and diverse interaction modes for mobile-agent training.

DigiData provides large-scale human demonstrations collected through systematic app-feature exploration, yet scaling human-driven data collection remains costly, with coverage currently limited to 26 apps. OpenApps primarily targets app reliability evaluation across versions and configurations and currently covers only 6 apps, limiting its app coverage for large-scale agent training. DigiWorld provides 15 mock apps running in Android emulators for reproducible and controllable mobile-agent research. Compared with lightweight simulators, emulator-based environments incur greater computational overhead, while its limited app coverage restricts the diversity of available environments and app functionalities.

At the data level, OpenMobile-2 provides nearly 12K demonstrations spanning 110+ apps, substantially expanding app coverage beyond DigiData (26 apps), while maintaining comparable trajectory diversity (0.43 vs. 0.45) and longer GUI-only trajectories (10.8 vs. 9.2 steps). At the environment level, OpenMobile-2 provides 35 ready-to-use, fully controllable apps, substantially broadening app coverage and functional diversity compared with OpenApps (6 apps) and DigiWorld (15 apps). Our environments offer rich app functionality and support complex workflows, enabling efficient, low-cost agent-driven data expansion. Furthermore, 300+ app-native tools across 50 apps enable diverse interaction modes beyond GUI-only execution, including hybrid GUI–tool use. Leveraging these capabilities, we train agents that achieve competitive performance on multiple widely recognized benchmarks (AndroidWorld, MobileWorld, etc), whereas only a limited number of comparable studies have evaluated their methods on these benchmarks.

AuthorsOpenReview ↗

Response to Q4 (SPA-Bench results missing for the GUI-only SFT OpenMobile-2 models)

Q4 SPA-Bench results missing for the GUI-only SFT OpenMobile-2 models

Response:

We report additional GUI-only results on SPA-Bench, which are comparable to those of the Hybrid setting.

ModelSPA-Bench
9B GUI-only SFT48.9
9B Hybrid SFT48.9
27B GUI-only SFT55.3
27B Hybrid SFT57.4
Reviewer_otA3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

OpenMobile-2 presents an open training stack for mobile agents, including environments covering more than 110 applications, 11.6K demonstration trajectories, and over 2K executable reinforcement-learning tasks. MobileGym++ expands simulated mobile use with commercial-style applications, cross-application workflows, and more than 300 app-native tools. The authors train 9B and 27B agents using supervised fine-tuning and reinforcement learning and evaluate them across simulated environments, Android emulators, and physical devices. The results demonstrate substantial improvements over the corresponding base models and benefits from hybrid GUI–tool training.

Critical strengths
  • The contribution is a substantial expansion of reusable training infrastructure. Building on OpenMobile and MobileGym, the paper adds commercial application functionality, cross-application workflows, demonstrations, executable tasks, and trained models. This integrated resource can support both replication and further training research.
  • Evaluation extends beyond the authors’ own simulator to established benchmarks and physical devices. The fixed-data-budget coverage experiment and transfer to unseen MobileWorld MCP tools provide useful evidence that the benefits are not confined to a single evaluation setting.
Critical weaknesses
  • The hybrid SFT comparison changes both the supervision format and the training-data mixture. Hybrid SFT adds hybrid trajectories to the GUI data, so the results do not fully isolate the benefit of hybrid interaction supervision from additional examples and exposure to their task distribution.
  • The overlap analysis relies primarily on instruction embedding similarity. Low textual similarity does not exclude shared task templates, related initial states, or equivalent business-object operations. This is particularly relevant when training and evaluation share application implementations. A structural account of the split would better support the generalization claims.
Questions for authors
  • Under matched example or token budgets, how much of the hybrid SFT advantage remains? Can hybrid demonstrations be converted into GUI-only demonstrations for a closer comparison?
  • How are training and evaluation tasks separated in terms of templates, initial states, and business-object operations, beyond instruction similarity?
  • What is the evaluation uncertainty on SPA-Bench and MobileGym++? Are the smaller RL improvements consistent across repeated evaluations?
4 comments
AuthorsOpenReview ↗

Response to W1&Q1 (Is the Hybrid SFT advantage attributable to hybrid interaction supervision or additional training data?)

W1&Q1 Is the Hybrid SFT advantage attributable to hybrid interaction supervision or additional training data?

Response: The gains from Hybrid SFT cannot be replicated by simply augmenting GUI-only training with additional training examples.

Thank you for this valuable suggestion. Following your recommendation, we converted the 1,774 hybrid trajectories into GUI-only demonstrations and incorporated them into the training data for Qwen3.5-9B, retraining the model under the same configuration:

Training dataAndroidWorldMobileWorld (Hybrid)MobileGymMobileGym++ (GUI-only)MobileGym++ (Hybrid)
GUI-only SFT (Original)72.8 ± 2.212.534.840.941.9
GUI-only SFT (+1,774)71.1 (-1.7)12.535.5 (+0.7)41.4 (+0.5)41.4 (-0.5)
Hybrid SFT (Original)73.3 ± 1.527.535.939.144.7

Overall, the results are close to the original GUI-only results reported in the paper. This is likely because scaling up GUI-only trajectories has already reached its performance ceiling, further validating the effectiveness of hybrid SFT. This supports our central claim: app-native tools provide a complementary action channel, and hybrid training enables generalizable tool-use behavior.

AuthorsOpenReview ↗

Response to W2&Q2 (Beyond instruction embedding similarity, is there additional evidence to more convincingly establish the independence of the training and evaluation data?)

W2&Q2 Beyond instruction embedding similarity, is there additional evidence to more convincingly establish the independence of the training and evaluation data?

Response1: Structural Analysis of Train–Test Overlap

We agree that this is a reasonable concern. Our tasks are constructed through systematic app exploration rather than predefined templates (Section 4.1). We further analyze potential train–test overlap at following levels:

  1. Initial states. We diversify initial environments during data construction. For emulator-based tasks, we construct diverse initial states and user contexts through a combination of agent-as-a-user interactions and manual interactions. Although this process is not explicitly described in the paper, we will release the corresponding Android emulator images along with our work. For simulators, programmatic initialization enables us to instantiate diverse initial states.
  2. Template & Business-object operations. We argue that some degree of atomic-operation overlap is inevitable in CUA data synthesis. A good synthesis method should cover important atomic operations and functionalities in the environment while avoiding overfitting to specific test-task instructions. This enables agents to acquire diverse atomic capabilities and flexibly compose them to generalize to unseen long-horizon tasks.
    • Template overlap. We use Gemini-3.1-pro-preview to extract entities and normalize task instructions, reducing the influence of entity variations and surface-level phrasing. We then compute embedding similarity between normalized task templates to assess potential template-level overlap.
    • Atomic-operation coverage. Beyond task- and template-level overlap, we further analyze overlap in distinct atomic operation types between the training and evaluation sets. Specifically, we report (i) Test cov., the fraction of evaluation operation types present in training, ∣Otest∩Otrain∣/∣Otest∣|\mathcal{O}{\mathrm{test}}\cap\mathcal{O}{\mathrm{train}}|/|\mathcal{O}{\mathrm{test}}|, and (ii) Train share, the fraction of training operation types present in evaluation, ∣Otrain∩Otest∣/∣Otrain∣|\mathcal{O}{\mathrm{train}}\cap\mathcal{O}{\mathrm{test}}|/|\mathcal{O}{\mathrm{train}}|.
Task SimilarityTemplate SimilarityTest Cov.Train Share
Emulator-AndroidWorld0.4340.48978.0%4.7%
Emulator-MobileWorld0.3910.45273.4%7.0%
Simulator-MobileGym0.4580.50569.8%9.0%
Simulator-MobileGym++0.5040.54877.7%17.7%

After normalizing entity variations and surface-level phrasing, Template embedding similarity increases slightly, but remains low, with no evidence of meaningful train–test contamination. While Test cov. shows that our training data covers a substantial fraction of atomic operations in evaluation tasks, the relatively low Train share suggests that this overlap stems from our goal of broad functional coverage rather than deliberate targeting of test-specific operations.

Response2: Additional Generalization and Overlap Analysis

To further examine potential overlap arising from shared apps and its impact on evaluation, we conduct two additional experiments using the emulator training pool.

  1. Held-out app generalization. We exclude a subset of apps from training and evaluate model performance on these held-out apps, while also reporting results on seen apps and the full benchmark.
ModelSystem (13)Markor (14)SimpleSms (6)Tasks (6)OsmAnd (3)OOD (42)InD (74)All (116)
9B-base53.850.050.033.30.0045.258.153.4
9B-full84.671.483.366.733.373.871.672.4
9B-OOD84.664.366.750.033.366.771.669.8
  1. Similarity-based data removal. We remove 86 training tasks with instruction similarity above 0.7 to AndroidWorld evaluation tasks, and compare against randomly removing the same number of training tasks. We evaluate all resulting models on AndroidWorld.
SettingAndroidWorld
Base54.3 ± 3.7
Full71.3 ± 3.2
Random removal (86)71.8 ± 3.5
Similarity-based removal (86)71.0 ± 2.6

These results suggest that the performance gains are not solely attributable to highly task-similar training examples, as removing them yields negligible performance degradation.

AuthorsOpenReview ↗

Response to Q3 (The evaluation uncertainty on SPA-Bench and MobileGym++)

Q3 The evaluation uncertainty on SPA-Bench and MobileGym++

Response: Modest RL improvements consistently observed across repeated MobileGym++ evaluations

We evaluate each setting three times and report the mean and standard deviation. RL generally yields modest improvements over Hybrid SFT on MobileGym++ for both 9B and 27B models, although the gains can vary due to the inherent instability of RL training. On SPA-Bench, the 9B model also shows a positive mean improvement, but the relatively small number of tasks leads to higher evaluation variance.

ModelMobileGym++ (GUI-only)MobileGym++ (Hybrid)SPA-Bench
9B-Hybrid SFT38.6 ± 1.443.8 ± 1.047.3 ± 3.3
9B-RL40.0 ± 1.2 (+1.4)46.7 ± 2.2 (+2.9)49.7 ± 3.3 (+2.4)
27B-Hybrid SFT54.9 ± 1.765.1 ± 1.258.2 ± 3.3
27B-RL56.4 ± 1.8 (+1.5)66.7 ± 1.0 (+1.6)60.3 ± 1.2 (+2.1)
AuthorsOpenReview ↗

Gentle Follow-up on Our Rebuttal

Dear Reviewer otA3,

Thank you again for your constructive feedback. As the discussion period is approaching its end, we would like to briefly follow up on our rebuttal.

We conducted additional experiments showing that simply adding 1,774 GUI-only demonstrations cannot reproduce the gains of Hybrid SFT. We also provided further train–test overlap and generalization analyses, including held-out app evaluations and similarity-based data removal. Finally, we reported three-run evaluations on MobileGym++ and SPA-Bench to better characterize performance variability.

We hope these additional results address your concerns and would appreciate any further feedback.

Thank you for your time and consideration.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to All Reviewers and Rebuttal Summary

We sincerely thank all reviewers for their careful reading, constructive feedback, and positive assessments. We are particularly encouraged by their recognition of OpenMobile-2's contributions to open mobile agent research. Reviewers (otA3, otE1, DqVa) all acknowledged the value of our infrastructure—including environments, data, and models—to the open-source community. Reviewers otA3 and DqVa also recognized the performance gains from transferring our models to real-world settings, while DqVa further appreciated our exploration of hybrid tool use on mobile devices. Below, we address the main concerns and further clarify our contributions.

  1. A showcase of our environment: https://anonymous.4open.science/r/openmobile2/README.md

  2. Comprehensive contributions to the open mobile-agent community. OpenMobile-2 fully open-sources an end-to-end mobile-agent training stack, including environments, SFT data, data-construction and training recipes, evaluation code, and trained models. Models trained on data from diverse environments achieve competitive results across established benchmarks with gains transferring to real-device evaluation on SPA-Bench. To our knowledge, OpenMobile-2 is among the strongest-performing mobile-agent efforts with publicly released training data and recipes.

  3. Contributions of simulated apps. Simulated apps provide a practical means of reproducing real-world commercial applications while reducing the cost of agent-driven exploration and enabling controllable environments. As reviewer DqVa noted, training on our simulated apps supports sim-to-real transfer beyond simulated environments to real devices. These results demonstrate the value of simulated apps not only as scalable data-construction infrastructure but also as an effective training foundation for real-world mobile agents.

  4. Generalization capability of our models. Reviewers raised concerns about the generalization of our training data. We clarify that functional overlap with benchmark tasks is inevitable when scaling data construction, but our data and experiments show that we do not deliberately fit our training distribution to test benchmarks. We further conduct strict app-level OOD evaluations and exclude training data with similarity exceeding 70% (reviewer otA3, Q2), further demonstrating OpenMobile-2's effectiveness on unseen apps.

Finally, we would like to thank all reviewers once again for their time and efforts.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.