- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper releases an open stack for mobile GUI agents. MobileGym++ adds 35 replicas of commercial apps to the MobileGym simulator, for 63 clients, or over 110 apps with the emulator pool; app state can be reset and checked by program. The authors collect 11,643 SFT trajectories with Gemini-3.1-Pro as the teacher (1,774 of them mix GUI actions with app-native tool calls) and 2,475 RL tasks with verifiers. Over 300 tools write the same state as the GUI, and MobileGym++ Bench (215 tasks) scores GUI-only and hybrid agents on identical tasks. Fine-tuned Qwen3.5-9B and Qwen3.6-27B models reach 79.9 on AndroidWorld, 50.4 on MobileWorld and 59.6 on SPA-Bench L3 (base: 31.9).
Critical strengths
S1. A complete open stack. State can be saved and restored, and checkers read business state, not screenshots (L196-200, L279-282). Data, tools, hyperparameters (Table 5), code (L507) and a reusable app-building recipe (App. B) are provided.
S2. Large SFT gains, also on physical devices. AndroidWorld rises from 54.3 ± 3.7 to 73.3 (9B) and from 67.1 ± 4.1 to 80.8 (27B), far beyond the seed spread (Table 2). On SPA-Bench, hybrid SFT lifts the 9B model from 25.5 to 48.9 and the 27B model from 31.9 to 57.4 (L408-413).
S3. A clean test of tools against the GUI, with a useful finding. Tools write the same store as their GUI paths and are judged on the same tasks and checkers (L253-257). With the same weights, tools add 2.8-4.2 points before hybrid SFT and 5.6-10.3 after it (Fig. 5b); at 27B that is 22 more of 215 tasks. Tool access alone is not enough.
Critical weaknesses
W1. The experiments do not show what the new simulated apps add. Emulator data is most of the SFT mix (7,725 of 11,643 trajectories, Table 6). The emulator pool holds all the AndroidWorld apps (Table 10), and App. D.3 reports AndroidWorld, MobileWorld and MobileGym training subsets (L1024-1025). Both SFT runs include the 2,144 GUI simulator trajectories, and the coverage study uses emulator data only (L430-431), so no experiment removes MobileGym++. The gains on AndroidWorld and MobileWorld could come mostly from emulator data on those benchmarks' own apps. An emulator-only SFT run at 9B would settle this, and a same-base run on the OpenMobile-1 data would isolate the new data.
W2. The generalization and real-device claims go beyond the evidence. Four of the five benchmarks share a runtime with training (L844-845), which weakens the argument at L802-804. The student beats its teacher on AndroidWorld (79.9 vs 70.7) but not on MobileWorld (50.4 vs 58.1) or SPA-Bench (59.6 vs 70.2; Table 2), as expected if it has seen the AndroidWorld apps. That leaves SPA-Bench, which is thin. It is left out of the overlap check (L850-851), "L3" is not defined, and all its values fit a 47-task set (28/47 = 59.6%). There is no spread, only one open-weight baseline, and no GUI-only SFT run. SPA-Bench's Chinese apps also include Taobao, JD, Meituan, Qunar, Cainiao and Alipay (Chen et al., 2025), all of which are in MobileGym++ (Table 9). If L3 uses them, the result is sim-to-real transfer on replicated apps. That is valuable, and it could supply the fidelity check the replicas lack (L206-215), but it is not transfer beyond the training environments (L362-363).
W3. The RL gains are within noise, and RL was validated on a test set. RL changes AndroidWorld by +0.7 and -0.9 points against seed spreads of 1.5-3.5 (Table 2), SPA-Bench by one task (27 to 28 of 47), and MobileGym by 6 and 7 of 256 tasks (L403-405). Those 256 tasks are also the RL validation set, run every 10 steps (L771-772; Table 1 lists 256 MobileGym tasks, and 98/256 = 38.3%). Training stopped at 150 of 300 budgeted steps (Table 5). If that checkpoint was picked on this curve, the MobileGym gain was selected on the test set. The RL tasks are a headline contribution that "largely advance training" (L115-117), and this run is their only evidence. Selection on held-out RL tasks and three evaluation runs would fix this; otherwise, RL should be framed as exploratory.
Questions for authors
Q1. What does SFT on emulator data alone give on AndroidWorld, MobileWorld and SPA-Bench, compared with the GUI-only run?
Q2. Which tasks form SPA-Bench "L3", and which apps do they use? How do results split between apps inside and outside the training pool? Which judge decides success?
Q3. Was the RL checkpoint chosen on the MobileGym validation curve?
4 comments
Response to W2&Q2 (Does the improvement on SPA-Bench reflect sim-to-real transfer or generalization from simulated environments to real devices?)
W2&Q2 Does the improvement on SPA-Bench reflect sim-to-real transfer or generalization from simulated environments to real devices?
We thank the reviewer for pointing out this issue. We will provide details of the SPA-Bench evaluation setup and further investigate the factors underlying our performance on SPA-Bench.
Response1: SPA-Bench L3 covers both seen and held-out apps
We follow the official SPA-Bench repository (ai-agents-2030/SPA-Bench, commit 4c292b6) and its data/single-app-CHN.csv. Among the 50 official tasks with difficulty level 3 (the highest), we evaluate 47, excluding three (huaweiliulanqi_2, huaweiliulanqi_5, and settings_2) due to environment incompatibilities: Xiaomi Browser requires phone-number authentication for web access, while Xiaomi Settings does not support Huawei-specific tasks.
Among the 47 evaluated tasks, 29 involve 15 apps included in our training pool, while 18 involve 10 held-out apps: Chinese Perpetual Calendar, China Telecom, Douyin, Umetrip, Haodf, QQ Music, Tencent Docs, Peking Union Medical College Hospital, Youdao Dictionary, and Zhihu. This split enables evaluation on real devices across both seen and unseen apps.
Response2: Our models improve performance on both the held-in and held-out splits of SPA-Bench under the official evaluation protocol.
We use SPA-Bench's official fine-grained evaluation prompt with gpt-5.6-sol as the judge model, providing all screenshots from each task in chronological order.
| Model | Tasks on Seen Apps (29) | Tasks on Held-out Apps (18) | All Tasks (47) |
|---|---|---|---|
| Qwen3.5-9B | 27.6 (8) | 22.2 (4) | 25.5 (12) |
| OpenMobile-2-9B | 51.7 (15) | 44.4 (8) | 48.9 (23) |
| Qwen3.6-27B | 34.5 (10) | 27.8 (5) | 31.9 (15) |
| OpenMobile-2-27B | 62.1 (18) | 55.6 (10) | 59.6 (28) |
As shown in the table, our models achieve substantial gains on both subsets. These results provide evidence of sim-to-real transfer on apps represented in our training environments, while also supporting generalization to unseen apps.
Response to W3&Q3 (Unstable RL gains)
W3&Q3 Unstable RL gains
We thank the reviewer for pointing out these concerns. We clarify the intended scope of our claims about RL in the paper and provide additional experiments to better characterize the actual contributions of RL.
Response1: On the Role and Effectiveness of RL in OpenMobile-2
First, we would like to clarify that our claim that OpenMobile-2 “largely advances training” (L115–117) primarily refers to the released training data, executable environments, and training recipes, rather than solely to the RL gains reported in our experiments.
We primarily use MobileGym to monitor RL training because its task distribution closely matches that of our RL training tasks, and its tasks and evaluation protocol have been validated to be reliable. Meanwhile, our paper also reports results on other established benchmarks beyond MobileGym. We appreciate the reviewer’s point and will revise the relevant discussion of MobileGym in the paper to better clarify its role.
We further report results from three evaluation runs on MobileGym++ and SPA-Bench. Overall, RL yields modest improvements over Hybrid SFT on MobileGym++ for both the 9B and 27B models, although the gains vary across evaluation runs, reflecting the inherent instability of RL training.
| Model | MobileGym++ (GUI-only) | MobileGym++ (Hybrid) | SPA-Bench |
|---|---|---|---|
| 9B-Hybrid SFT | 38.6 ± 1.4 | 43.8 ± 1.0 | 47.3 ± 3.3 |
| 9B-RL | 40.0 ± 1.2 (+1.4) | 46.7 ± 2.2 (+2.9) | 49.7 ± 3.3 (+2.4) |
| 27B-Hybrid SFT | 54.9 ± 1.7 | 65.1 ± 1.2 | 58.2 ± 3.3 |
| 27B-RL | 56.4 ± 1.8 (+1.5) | 66.7 ± 1.0 (+1.6) | 60.3 ± 1.2 (+2.1) |
Response2: RL Exploration from the Base Model
We acknowledge that the current RL gains over the fully trained SFT baseline remain limited, and are continuing to investigate the potential of RL to further improve agent performance.
Specifically, we sample 128 additional MobileGym++ RL tasks from the 2K+ executable RL tasks released in our paper, excluding the 512 tasks used for RL training in the original experiment. We use these 128 tasks as a validation set for the new RL experiment, following the same training configuration as in the paper.
Due to time and computational resource constraints, we run RL training for only 60 steps. We report the validation performance every 10 steps, along with the corresponding evaluation results on MobileGym.
| Base | Step 10 | Step 20 | Step 30 | Step 40 | Step 50 | Step 60 | |
|---|---|---|---|---|---|---|---|
| 128-task Val. | 6.3 | 10.2 | 7.0 | 13.3 | 15.6 | 14.8 | 18.0 |
| MobileGym | 9.4 | 12.1 | 11.3 | 16.0 | 19.5 | 19.1 | 20.3 |
Notably, the MobileGym performance improves from an initial score of 9.4 to 20.3, indicating that RL training can yield substantial gains from a relatively low starting point. This further demonstrates the potential of our environments to support RL training.
Gentle Follow-up on Our Rebuttal
Dear Reviewer DqVa,
Thank you again for your detailed and constructive feedback.
We would like to kindly follow up on our responses to your concerns regarding the contributions of simulated apps, sim-to-real generalization and generalization beyond the training environments, and the effectiveness of RL.
We have conducted additional controlled SFT experiments to distinguish the contributions of emulator data and simulated apps, showing that simulated apps provide complementary gains beyond emulator training. We have also clarified the SPA-Bench evaluation setup and reported results separately on seen and held-out apps.
Regarding RL, we have provided repeated evaluations to characterize its variability and conducted an additional 60-step RL experiment starting from the base model. The MobileGym score improves from 9.4 to 20.3, indicating that RL training can yield substantial gains from a relatively low starting point. This further demonstrates the capability of our environments to support RL training.
We would be grateful if you could share whether our responses and additional experiments have addressed your main concerns. We would be happy to clarify any remaining issues.
Thank you very much for your time and consideration.
Best regards,
The Authors
Response to W1&Q1 (Beyond Emulator Data: Gains from Simulated Apps)
W1&Q1 Beyond Emulator Data: Gains from Simulated Apps
Response1:New simulated apps provide modest gains on AndroidWorld and MobileWorld, while substantially improving real-device generalization.
We thank the reviewer for raising this question, which motivates us to further investigate the contribution of simulated apps and the overall contributions of OpenMobile-2 through controlled comparisons, including an emulator-only SFT run and an SFT run using OpenMobile-1 data with the same base model. We conduct all additional SFT experiments using Qwen3.5-9B as the base model to ensure a consistent comparison across training data configurations. The results are reported below, where MG++ denotes MobileGym++.
As expected, emulator data accounts for a substantial portion of the gains on AndroidWorld and MobileWorld. However, emulator-only training shows limited transfer to real-world mobile applications, partly because commercial apps are unavailable or their functionality is insufficiently covered due to constraints such as authentication. Simulated apps provide complementary training signals that improve cross-app generalization beyond emulator data, while also yielding substantial gains on SPA-Bench, which evaluates agents on real commercial applications. The comparison with OpenMobile-1 further demonstrates the benefits of the expanded training data in OpenMobile-2.