acceptodds
Under review as a conference paper at ICLR 2027

PhoneWorld: From Real-App Trajectories to Dynamic and Verifiable Environments for Phone-Use Agents

25 views

Abstract

Real applications provide the training setting closest to phone-agent deployment, but are difficult to reset, scale safely, and verify programmatically. Static screenshots and interaction trajectories preserve realistic evidence but cannot generate new experience. We introduce PhoneWorld, a trace-grounded framework that converts such evidence into runnable, resettable, and verifiable Android environments. PhoneWorld induces a usage-weighted interaction skeleton from observed pages, transitions, and state-changing operations; translates it into a behavior-grounded app specification; realizes the specification through an autonomous build–inspect–repair loop; and synthesizes executable tasks with programmatic verifiers. The resulting suite spans 34 consumer-facing apps across 16 domains and supports an audited online benchmark, verified trajectory generation, and online RL through common reset and verification interfaces. Evaluations with diverse general and open-source GUI agents show that PhoneWorld supports reliable end-to-end online interaction and exposes capabilities complementary to AndroidWorld. Controlled SFT experiments further show that PhoneWorld trajectories complement AndroidWorld supervision, transfer across online and offline benchmarks, and become more effective as data volume and app coverage increase. Under a matched RL budget, combining PhoneWorld mock-app rollouts with real-app rollouts improves performance over real-app RL alone on both real-phone tasks and AndroidWorld. Together, these results demonstrate that trace-grounded executable abstraction can bridge realistic mobile behavior and scalable agent learning, turning limited real-app evidence into a growing supply of controllable and verifiable environments for training and evaluation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
GUI Agent, Environment Synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_V8xhOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces PhoneWorld, a trace-grounded framework that converts such evidence into runnable, resettable, and verifiable Android environments. PhoneWorld leverages mocked-app environments as a reset-friendly proxy for real apps. Combining PhoneWorld's mock-app rollouts with real-app rollouts yields performance gains over real-app RL alone on both real-phone tasks and the AndroidWorld benchmark.

Critical strengths

a. PhoneWorld successfully leverages mocked-app environments as a reset-friendly proxy for real apps. b. PhoneWorld's mock-app rollouts yields performance gains over real-app RL alone.

Critical weaknesses
  1. Environments are built bottom-up with hand-crafted mocks, which the authors admit is hard to scale. Without a top-down approach that covers the real task distribution, the benchmark's validity is unestablished, and agent results inside this space may not transfer to the settings AndroidWorld targets. This attacks the paper's foundation, not a peripheral detail.
  2. There is no direct, fine-grained comparison with AndroidWorld, and the Figure 2 justification is unconvincing. The verifier is also not positioned against prior work (e.g., WorldCoder-bench), leaving unclear whether gains come from the environment, verifier, or test-set construction. The paper claims to be both new and better but demonstrates neither.
Questions for authors
  1. Experimentally, more direct and fine-grained comparisons with AndroidWorld are needed to demonstrate the value of this benchmark. The experimental justification in Figure 2 also appears less convincing.
  2. I recognize the value of mocks, but I would like to ask about the generalizability of the constructed environment. I understand that scaling the current approach is difficult; it feels like a bottom-up method for building environments, which may not be efficient enough. I believe a more efficient approach would be top-down. Am I misunderstanding something?
  3. There are many ways to write the skeleton as an app specification. How can this step be made efficient? If a weaker model were used for this step, would it affect both the results and the efficiency? Are there alternative approaches?
  4. How does the specific implementation of your programming verifier differ from prior methods? For instance, compared to WorldCoder-bench and similar approaches.
5 comments
AuthorsOpenReview ↗

Response to Reviewer V8xh (1/2)

We sincerely thank you for recognizing the value of mock environments and for the constructive comments. We address each point below.


W1/Q2: How the environments are built and how the approach scales.

We are concerned that there may be some misunderstanding of our method, and we would like to clarify it further below.

  1. The environments are built automatically by an agent, not by hand. Starting from automatically generated specifications, a coding agent autonomously implements, compiles, tests, and repairs the app following the build–inspect–repair procedure. As stated in Section 3.4, the app is built "without human intervention during construction"; humans are involved only in the final acceptance testing. For this reason, PhoneWorld could scale easily: internally, we have expanded from 34 to about 100 apps. In addition, the three statements in the paper about difficulty in scaling (Abstract, Section 1, and Section 2) all refer to real production apps, not to the construction of mock apps, which is scalable.
  2. The construction is a top-down process that starts from real tasks. The real trajectories we use reflect the distribution of users' real tasks. We first induce a specification (PRD) from these trajectories, which covers the app's core pages and their priorities, then build the mock app accordingly and synthesize tasks on it. We agree that a top-down approach is more efficient. At the same time, a mock app should be as close as possible to the real app in its functions and pages, so the specification must be induced from real trajectories rather than written directly from a description of requirements; otherwise, both page fidelity and functional coverage suffer. Therefore PhoneWorld build the environments top-down from real task trajectories, so that both the environments and the tasks synthesized from them largely follow the real task distribution.
  3. "Whether the environments reflect capabilities needed in real apps" and "whether the learned capabilities transfer" are answered by two sets of experiments. The former is answered by the RL experiments: on 150 real-phone tasks, RL that mixes mock-app and real-phone rollouts outperforms real-phone-only RL in all three seeded runs (41.33 ± 0.67 → 45.56 ± 0.38, solving 7, 5, and 7 more tasks, respectively), showing that capabilities learned in mock environments indeed carry over to real apps. The latter is answered by the data-composition experiments: on top of 26K AndroidWorld (AW) steps, we added 10K AW, 10K MobileWorld (MW), or 10K PhoneWorld (PW) steps, and adding PW data yields the largest gain on AndroidWorld, showing that capabilities learned in PhoneWorld transfer to AndroidWorld:
Training dataAndroidWorld
26K AW55.2
36K AW56.9
26K AW + 10K MW59.5
26K AW + 10K PW71.6

W2/Q1: Comparison with AndroidWorld and Figure 2.

The distinctive feature of PhoneWorld is that it provides controllable environments for the consumer apps that users actually use, addressing the problem that closed-source consumer apps cannot be reliably reset and verified. The comparison with AndroidWorld is as follows:

AndroidWorldPhoneWorld
Number of apps2034 (16 domains)
Number of tasks116 (parameterized task templates)120 (benchmark), plus a pool of 7,936 tasks
LanguageEnglishChinese
App sourceOpen-source/system appsClosed-source consumer apps

Figure 2 examines how scaling along two dimensions, data volume and the number of apps, affects performance on each benchmark:

  • Scaling data volume (left panel): in-domain PhoneWorld and Internal-MobileBench, which also evaluates Chinese consumer apps, both improve (+59.1 and +5.4, respectively), while the out-of-domain English benchmarks AndroidWorld and AndroidControl remain unchanged or improve slightly (+0.0 and +1.4, respectively). This shows that scaling PhoneWorld data clearly improves in-domain performance while maintaining or slightly improving out-of-domain performance.
  • Scaling the number of apps (right panel, with the PW budget fixed at 10K steps): increasing from 5 to 34 apps improves all four benchmarks: PhoneWorld +18.3, Internal-MobileBench +18.3, AndroidWorld +10.4, and AndroidControl +3.0. This shows that as the diversity of training environments from PhoneWorld increases, mobile agents improve both in domain and out of domain.

We hope these additions make the experimental results more convincing.

(Continued in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer V8xh (2/2)

W2/Q4: Relation of the verifier to prior work, and the source of the gains.

Thank you for pointing out that the positioning of our verifier was missing. We respond in two parts.

  1. Relation to WorldCoder-Bench: different objects of verification. WorldCoder-Bench verifies the generated program itself: it uses StateProbe to execute generated 3D-world programs in a controlled browser and checks, against expert-written behavioral contracts hardened by mutation testing, whether the program's state transitions conform to the specification. It thus answers the question "is this code implemented correctly?" The PhoneWorld verifier instead verifies the task outcome of an agent in a fixed environment: once built and accepted, the environment is taken as given, and the verifier checks whether the final SQLite state at the end of a rollout, or the returned answer, satisfies the task's success condition (71 of the 120 tasks are state-based and 49 are answer-based). The former verifies the environment, whereas the latter verifies the agent; the two sit at different stages of the pipeline and are not competing solutions to the same problem. We will add this distinction in the revision.
  2. The gains come from the environment itself. To identify the source of the gains, we collected the same amount (10K steps) of data in MobileWorld, mixed it with the same 26K AW steps, and compared the results on AndroidWorld (see the table in point 3 of our response to W1/Q2). On top of the same 26K AW steps, adding 10K PW steps yields a gain of 16.4 points, whereas adding 10K MW steps yields 4.3 points, a difference of 12.1 points. The two settings use exactly the same data volume, mixing ratio, and AW portion; the only difference is the environment from which the 10K steps are collected. The gains therefore come from the PhoneWorld environment itself, not from simply adding more data. Moreover, reducing AW data from 36K to 26K costs only 1.7 points (56.9 → 55.2), so the 71.6 cannot be attributed to the change in AW data volume either.

Q3: Efficiency of specification generation, the effect of a weaker model, and alternatives.

Efficiency. Generating the specification from the input screenshots and trajectories takes about 6–10 minutes, whereas the full mock-app construction pipeline typically takes 5–6 hours. Specification generation thus accounts for only about 3% of the total time and is not the bottleneck of the pipeline.

Using a weaker model. We rebuilt App2 with the weaker Sonnet 4.6, with which specification generation takes 6–8 minutes. The fidelity results are as follows (metrics as in Table 5):

ModelRPCNPCFPCDERTERVF
Opus 4.697.794.976.777.182.570.0
Sonnet 4.694.689.268.374.280.363.4

The weaker model still recovers most pages and page transitions (RPC, DER, and TER drop by only 2.2–3.1 points), but functional coverage (FPC, −8.4) and visual fidelity (VF, −6.6) degrade noticeably. This shows that stronger models reproduce functional and visual details more faithfully, which we believe is the key factor whether a mock app can transfer to the real app.

Alternatives. Reverse Engineering has recently attracted considerable attention; for example, open-source projects such as Reverse Engineering Anything can already decompile APKs directly. However, we see three issues with this approach: (1) security concerns; (2) the app structure obtained by reverse engineering does not reflect users' real usage needs; and (3) reverse engineering cannot fully recover the app's resource assets, so the reconstructed app has poor visual fidelity. We agree that more efficient ways of constructing the specification are worth pursuing. Considering the trade-off between efficiency and quality, we think our current approach is a reasonable choice, and we will continue to explore this direction in future work.


Once again, we sincerely thank you for the time and effort devoted to reviewing our paper and for the valuable suggestions. If our responses have resolved your concerns, we would be very grateful if you could consider raising your score. If you have any further questions, we would be more than happy to engage in further discussion.

AuthorsOpenReview ↗

Summary of rebuttal to Reviewer V8xh

Dear Reviewer V8xh,

Thanks again for your time and effort. As the discussion period is nearing its end, we would be grateful if you could review our response.

We would like to clarify that the environments are not hand-crafted: a coding agent builds each app automatically from a specification induced top-down from real user trajectories, and this framework is scalable that our internal suite has grown from 34 to about 100 apps. On transfer, adding mock-app rollouts improves all three seeded RL runs on 150 real-phone tasks (41.33 ± 0.67 → 45.56 ± 0.38), and PhoneWorld data generalize better to AndroidWorld than an equal amount of MobileWorld data. We also distinguish our verifier from WorldCoder-Bench: the latter verifies generated programs against behavioural contracts, whereas ours checks the final state of an agent's rollout.

If our responses have addressed your concerns, we would be grateful if you could continuously support our work and kindly consider raising your score; if you have any other questions, we would be glad to discuss further. Thank you again for your time.

Reviewer_9V7WOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PhoneWorld, which converts real-app screenshots and trajectories into runnable mock Android apps with reset and automatic task verification. The authors construct 34 apps and show that data and RL experience from these environments can improve performance beyond the mock apps.

Critical strengths
  • Resettable and verifiable environments are useful for mobile agent evaluation and training.

  • The 16 domains covered appears diverse and well represented.

  • The SFT and RL results show promising transfer to other environments and real-phone tasks.

Critical weaknesses
  • The main contribution over existing mobile agent environments and benchmarks is unclear. Datasets like AndroidWorld, MobileWorld, and AndroidLab already support interactive benchmarking, and MobileWorld also provides backends and verification. Although PhoneWorld introduces automatic construction from real-app traces, the experiments mainly compare against AndroidWorld, making it difficult to judge what it adds over the broader set of existing environments.

  • I believe the authors want to make the point that this environment allows scalable construction of mock apps to help the performance on real apps. If this is the case, I think this needs stronger validation. Things like costs, time, or the amount of human acceptance testing and correction needed per app for example.

  • The motivation for another evaluation benchmark is also unclear given the results. Claude and Gemini already achieve up to 94.2% success. The authors should clarify which additional capabilities or failure modes the benchmark evaluates. (Or, as my previous point mentions, maybe the paper should be framed differently.)

Questions for authors
  1. What advantages does PhoneWorld provide over using or extending existing environments such as MobileWorld and AndroidLab?

  2. What are the costs and human effort per app, and how do these compare with existing approaches?

  3. Could we see similar improvements by adding data from another existing mobile environment?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 9V7W (1/2)

We sincerely thank you for the review and the constructive comments. Your central concern is the added value of PhoneWorld over existing environments; to address it, we added an evaluation on MobileWorld and a controlled comparison with the same amount of MobileWorld data. We address each point below.


W1/Q1: Advantages over existing environments.

The core contribution of PhoneWorld is a method that automatically reconstructs mock apps from real trajectories and screenshots. It produces Android environments with resettable state and programmatically verifiable outcomes at scale, and a single environment interface supports data generation, benchmark construction, and RL training.

Existing Android environments obtain controllability by choosing apps whose state they can directly access: AndroidWorld uses open-source and system apps whose state is stored locally, AndroidLab runs apps in a pre-built offline AVD image, and MobileWorld self-hosts the backends of open-source alternatives. These environments are therefore limited to apps whose state is accessible, and they are designed manually. PhoneWorld differs in two respects:

  1. It targets the closed-source consumer apps that users actually use. These apps are protected by risk-control mechanisms, their state cannot be reset, and their outcomes cannot be verified deterministically, so existing approaches cannot turn them into controllable environments.
  2. It is built automatically and can keep growing. The pipeline is not tied to any particular app and can continuously convert new consumer apps into mock apps; our internal version already covers about 100 apps.
AndroidWorldAndroidLabMobileWorldPhoneWorld
App source20 open-source/system apps9 apps20 apps, mostly open-source alternatives34 closed-source consumer apps (16 domains)
How app state is controlledApp state is stored locally and directly readablePre-built offline AVD imageSelf-hosted backends; source code modified for some appsRebuilt as mock apps; no source code or backend of the original app required
Environment constructionManual designManual designManual design + adaptation of open-source projectsAutomatically synthesized from screenshots and trajectories

To show what PhoneWorld adds beyond other environments, we further evaluated on the GUI-only subset of MobileWorld (117 tasks) (task success rate, %):

ModelAndroidWorld (116)PhoneWorld (120)MobileWorld (117)
UI-TARS-7B-SFT25.045.81.7
GUI-Owl-7B46.638.36.8
UI-Venus-Navi-7B44.850.09.4
UI-Venus-1.5-8B47.456.78.5
SFT (26K AW only)55.210.86.0
PWAgent (26K AW + 10K PW)71.665.013.7

After adding 10K PhoneWorld (PW) steps to 26K AndroidWorld (AW) steps, the model improves on all three benchmarks (AndroidWorld +16.4, PhoneWorld +54.2, MobileWorld +7.7) and outperforms all open-source GUI agents on each of them, which shows that PhoneWorld data benefit multiple Android benchmarks.

(Continued in Part 2/2.)

AuthorsOpenReview ↗

Response to Reviewer 9V7W (2/2)

W2/Q2: Construction cost and human effort.

We build the mock apps with Claude Code (with Claude Opus 4.6). Across the 34 apps, each app costs $149.92 and takes 5–6 hours on average. Human involvement is minimal: each app undergoes a single round of human acceptance testing after construction, which takes about 10 minutes, during which a tester walks through the pages of the app and checks whether its functions are fully implemented.

Comparison with existing approaches: to the best of our knowledge, PhoneWorld is the first work to build controllable mobile environments for closed-source consumer apps, and the resulting APKs run directly on Android devices. Among the mobile-app environment construction works we surveyed, none other than GUI-Genesis reports its construction cost or releases its construction pipeline, so we cannot reproduce their pipelines to estimate costs. GUI-Genesis builds one executable environment per task, at an average cost of $0.27 per environment. The advantage of PhoneWorld is that once a mock app is built, a large number of tasks can be generated without rebuilding the environment: with the 7,936 tasks in the paper, the amortized environment cost is $0.63 per task; internally, we have generated 982 tasks per app on average with PhoneWorld, which reduces the cost to $0.15 per task environment. This cost will continue to decrease as more tasks are generated for each app. In short, a mock app built once supports large-scale task synthesis, data generation, model evaluation, and RL training.


W3: Positioning of the benchmark.

Thank you for the suggestion. We agree that the benchmark is an auxiliary contribution; the main contribution of this paper is the environment construction pipeline: a method that automatically converts static trajectories into dynamic mobile environments, which is resettable, verifiable, and controllable, simultaneously supports data generation, evaluation, and RL training. We will clarify this positioning in the revision. Under this positioning, the benchmark serves two functions, which is also where its discriminative power lies:

  1. Demonstrating that PhoneWorld provides an environment that supports evaluation. All 34 environments run end to end, covering reset, multi-step interaction, persistent state writes, and automatic judgment, and all 13 heterogeneous agents can be evaluated on them. This is evidence of the environments' usability, independent of the score levels.
  2. Measuring training effects. PhoneWorld success rates span 12.5–73.3% across training recipes (Table 3) and 38.3–56.7% across open-source GUI agents (Table 2).

Compared with existing mobile benchmarks, the PhoneWorld benchmark is Android-native, approximates the ability of mobile agents to complete tasks in consumer apps, and supports state reset and reliable verification for such apps, rather than relying on rubric-based judgments with uncertain outcomes.


Q3: Would data from another mobile environment yield similar improvements?

We collected the same amount (10K steps) of data in the MobileWorld (MW) environment, mixed it with the same 26K AW steps, and compared the results on AndroidWorld:

Training dataAndroidWorld
26K AW55.2
36K AW56.9
26K AW + 10K MW59.5
26K AW + 10K PW71.6

On top of the same 26K AW steps, adding 10K PW steps yields a gain of 16.4 points, whereas adding 10K MW steps yields 4.3 points. This shows that data from other mobile environments also help, but PhoneWorld data generalize better across environments. We attribute this to the diversity of training environments, which helps the model generalize; this is consistent with the app-coverage scaling experiment in the right panel of Figure 2.


We hope that the added evaluation on MobileWorld and the controlled comparison with MobileWorld data address your concern about the contribution of PhoneWorld. Once again, we sincerely thank you for the time and effort devoted to reviewing our paper and for the valuable suggestions. If our responses have resolved your concerns, we would be very grateful if you could consider raising your score. If you have any further questions, we would be more than happy to engage in further discussion.

AuthorsOpenReview ↗

Summary of rebuttal to Reviewer 9V7W

Dear Reviewer 9V7W,

Thanks again for your time and effort. As the discussion period is nearing its end, we would be grateful if you could review our response.

We added an evaluation on the GUI-only subset of MobileWorld, where PhoneWorld data raise the success rate from 6.0% to 13.7%, above all evaluated open-source GUI agents. With the same 26K AW steps, 10K PhoneWorld steps improve AndroidWorld by 16.4 points, whereas 10K MobileWorld steps improve it by 4.3 points. Building an app costs $149.92 and 5–6 hours, plus about 10 minutes of human acceptance testing, and the amortized environment cost drops to $0.15 per task. We also clarified that the environment construction method is our main contribution and the benchmark an auxiliary one.

If our responses have addressed your concerns, we would be grateful if you could kindly consider raising your score; if you have any other questions, we would be glad to discuss further. Thank you again for your time.

Reviewer_ZuLLOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes PhoneWorld, a pipeline that turns screenshots and real-device trajectories of production apps into resettable mock Android apps with programmatic verifiers. A VLM sorts screenshots into a page taxonomy, trajectory statistics weight the pages and transitions, a VLM writes prioritized page specifications, and a coding agent implements the app in Kotlin through a build-inspect-repair loop with final human acceptance testing. Tasks are synthesized jointly with answer-based or SQLite-based verifiers. The suite covers 34 consumer apps in 16 domains with a 120-task benchmark, a pool of about 8K tasks, and about 3.4K verified trajectories. Experiments evaluate thirteen agents on the benchmark, show that replacing part of an AndroidWorld SFT mixture with PhoneWorld trajectories improves both benchmarks under a matched budget, show that mixing mock-app RL rollouts with real-app rollouts improves a real-phone evaluation and AndroidWorld, and audit the fidelity of four mock apps.

Critical strengths
  • An artifact that fills a real gap. Online training of phone agents lacks resettable, verifiable environments for realistic consumer apps. The paper delivers 34 of them behind one reset and verifier interface, with a planned release and an honest fidelity audit. The artifact stays useful even where the numbers need firming up.

  • Evidence that environment diversity transfers. Widening coverage from 5 to 34 apps at a fixed PhoneWorld budget improves AndroidWorld and the internal offline benchmark. Mixing mock-app rollouts into real-app RL at a matched budget improves both the real-phone evaluation and AndroidWorld. If these results survive the controls requested below, they support the central thesis that mock environments complement real-app experience.

Critical weaknesses
  • The central transfer claim rests on a confounded comparison and a single small RL run. The matched-budget SFT comparison removes AndroidWorld data and adds PhoneWorld data at the same time, and there is no AndroidWorld-only run at the reduced volume. The paper's own sweeps show the effect is not additive: PhoneWorld-only training never moves AndroidWorld, AndroidWorld-only training reaches 56.9, and the mixture reaches 71.6. This is never discussed. The RL comparison reports a single run on 50 human-judged tasks per category, and the base 4B model, the algorithm, and the reward for real-phone rollouts are not stated. The single-app and mini-app gains amount to four tasks each, and cross-app goes down. These two experiments carry the main contribution, so the claim is plausible but not established.

  • The benchmark side is weaker than the framing suggests. Every general agent scores higher on PhoneWorld than on AndroidWorld, and the best agents are near ceiling even on hard tasks. Each of the three rank reversals used in Section 4.1 to argue complementarity rests on a margin of one or two tasks on one of the two benchmarks. The example verifier in Figure 3 accepts any message to the target user that contains the digit 3, which is hard to square with the strict success criterion. The paper does not say how the 120 tasks were audited or whether they are excluded from the training trajectories. Since PhoneWorld is also offered as an audited benchmark, these gaps affect how much the in-domain numbers mean.

Questions for authors
  1. Please report a 26K-AndroidWorld-only run. Which agents generated the AndroidWorld and PhoneWorld training trajectories? Do the 120 benchmark tasks or their verifier predicates overlap with the task pool and the training trajectories?

  2. Which 4B model and RL algorithm were used, and how many rollouts per step? How are the real-phone rollouts rewarded, given that production apps lack programmatic verifiers? What is the seed-to-seed variance of the Table 4 recipes? With 50 tasks per category, even three seeds would show whether the gains exceed noise.

  3. Which VLM and coding agent build the apps? How many build-inspect-repair iterations and how much human acceptance testing does a typical app need, and how many attempted apps were discarded? What fraction of benchmark verifiers were manually checked for false positives like the one in Figure 3?

  4. GUI-Genesis is cited only inside a citation cluster. A direct statement of what PhoneWorld adds beyond the Android target would sharpen the novelty claim.

6 comments
AuthorsOpenReview ↗

Response to Reviewer ZuLL (1/4)

We sincerely thank you for the careful review and for recognizing the value of PhoneWorld as a suite of resettable and verifiable environments. We address each point below.


W1/Q1: Roles of AndroidWorld and PhoneWorld data.

Following your suggestion, we trained a control on 26K AndroidWorld (AW) steps only. Its results are listed together with those in Table 3 (task success rate, %):

Training dataAndroidWorldPhoneWorld
26K AW55.210.8
36K AW56.912.5
26K AW + 10K PW71.665.0
36K PW46.673.3
  • With AW data only, the model performs poorly on PhoneWorld, and increasing AW data from 26K to 36K improves each benchmark by only 1.7 points.
  • Adding 10K PhoneWorld (PW) steps to the same 26K AW steps raises PhoneWorld from 10.8 to 65.0, and AndroidWorld becomes 16.4 and 14.7 points higher than with 26K AW and 36K AW, respectively. Hence, the 71.6 cannot be attributed to the change in AW data volume, and PhoneWorld data also improve AndroidWorld.
  • Regarding the non-additivity you pointed out: PhoneWorld and AndroidWorld are two very different environments, the former consisting of Chinese consumer apps and the latter of English open-source/system apps. A model trained on only one type of data has seen only one type of environment, so its performance on the two benchmarks is clearly imbalanced. Adding PhoneWorld data increases the diversity of training environments, and it is this added diversity that brings the improvements on both benchmarks. The right panel of Figure 2 provides consistent evidence: with the PW budget fixed at 10K steps, increasing the number of PW apps from 5 to 34 raises AndroidWorld from 61.2 to 71.6.

Both the AW and PW training trajectories were collected with Seed 2.0 Pro.


W1/Q2: RL settings.

The RL settings are as follows:

ItemSetting
Backbone modelQwen3.5-4B
AlgorithmGRPO
Tasks per step32
Rollouts per task8
Reward for mock-app rolloutsPhoneWorld programmatic verifier, binary (0/1)
Reward for real-phone rolloutsVLM-as-a-Judge, rubric-based continuous score in [0, 1]

Since production apps lack programmatic verifiers, real-phone rollouts are scored with VLM-as-a-Judge: we first generate a rubric for each task; a VLM then analyzes the trajectory step by step and judges whether each rubric item is satisfied, and the fraction of satisfied items is used as the reward.

In Table 4, mixing mock-app and real-phone rollouts improves Single-App, Mini-App, and AndroidWorld. The 2-point drop on Cross-App corresponds to a single task and does not indicate a loss of capability. Cross-App tasks are longer and involve more complex intermediate dependencies, so effective rewards are hard to obtain during RL, and the policy on these tasks stays close to its initialization. On average, Real-app + mock RL (54.8) outperforms both SFT (42.6) and Real-app RL (49.8).

(Continued in Part 2/4.)

AuthorsOpenReview ↗

Response to Reviewer ZuLL (2/4)

W1/Q2: Seed-to-seed variance of the Table 4 recipes.

To examine the stability of these results, we repeated both RL recipes with three different random seeds:

SeedRecipeSingle-AppCross-AppMini-AppReal-phone avg.
1Real-app RL54.020.048.040.67
1Real-app + mock RL62.018.056.045.33
1Δ+8.0−2.0+8.0+4.67
2Real-app RL58.018.050.042.00
2Real-app + mock RL62.020.054.045.33
2Δ+4.0+2.0+4.0+3.33
3Real-app RL54.018.052.041.33
3Real-app + mock RL64.016.058.046.00
3Δ+10.0−2.0+6.0+4.67
Mean ± stdReal-app RL55.3 ± 2.318.7 ± 1.250.0 ± 2.041.33 ± 0.67
Mean ± stdReal-app + mock RL62.7 ± 1.218.0 ± 2.056.0 ± 2.045.56 ± 0.38
MeanΔ+7.3−0.7+6.0+4.22

Each category contains 50 tasks, so one task corresponds to 2.0 points; the real-phone average is the success rate over all 150 tasks; Δ is Real-app + mock RL minus Real-app RL under the same seed. Table 4 reports seed 1, and the means over the three runs (41.33 and 45.56) are close to those of that run (40.67 and 45.33). The repeated runs support three conclusions:

  1. The overall gain consistently exceeds the noise. The real-phone average rises from 41.33 ± 0.67 to 45.56 ± 0.38. The difference of 4.22 points is about 6–11 times the standard deviation of either recipe, and Real-app + mock RL is higher in every run (by 7, 5, and 7 tasks). A Welch t-test gives t = 9.5 (df ≈ 3, two-sided p < 0.01); with only three runs per recipe, we regard this as an order-of-magnitude reference rather than a rigorous significance claim.
  2. The gains on Single-App and Mini-App are consistent in direction across all three runs (4, 2, and 5 tasks, and 4, 2, and 3 tasks, respectively) and do not depend on any single run.
  3. Cross-App is essentially unchanged. The per-run differences are −1, +1, and −1 tasks: inconsistent in sign and all within one task. Their mean of −0.33 tasks (−0.67 points) is smaller than the standard deviation of this category itself (2.0 points). The −2-point difference on Cross-App in Table 4 is therefore within run-to-run variation.

W2/Q1: Overlap between benchmark tasks and training data.

We agree that benchmark tasks must not overlap with the training data. When constructing the tasks, we separated the training data from the test tasks to ensure valid evaluation. To directly answer your question, we measured the overlap at both the task level and the verifier level:

  • Task level. We extracted a template from each test task and instantiated it 10 times; different instances of the same template have a mean Jaccard similarity of 0.699, whereas the Jaccard similarity between training and test tasks is 0.368. No test-task template matches any training task.
  • Verifier level. Identical rules or duplicated predicates occur in 2 of the 120 tasks. We manually reviewed these two tasks and confirmed that they do not overlap with any training task at the task level; their verifiers are similar because they test similar capabilities.

Therefore, the benchmark tasks are independent of the training data; the benchmark is valid for evaluation, and its results are free from training-data contamination.

(Continued in Part 3/4.)

AuthorsOpenReview ↗

Response to Reviewer ZuLL (3/4)

W2: Difficulty and discriminative power of the benchmark.

This benchmark was built to verify that PhoneWorld provides a reliable evaluation environment. Its tasks are synthesized automatically, and the difficulty levels are defined as follows:

DifficultyTask typeTypical steps
EasySingle-goal browsing or lookup3–8
MediumSimple write operations6–15
HardCompositions of multiple operations12–25

Hard tasks are mainly composed of operations from easy and medium tasks. For general-purpose models with strong history management, this composition adds limited difficulty, so their hard-task scores are close to their easy and medium scores. Nevertheless, the benchmark remains discriminative:

  1. Open-source GUI agents reach only 31.4–51.4% on hard tasks, leaving substantial room for improvement.
  2. One of the main uses of the benchmark is to measure training effects, and success rates across training recipes span 12.5–73.3% (Table 3).

We position this benchmark as an evaluation of agents' ability to perform daily tasks in Chinese consumer apps, rather than as a set of problems designed to exceed the capabilities of frontier models. High scores of frontier models are therefore expected; what the benchmark measures is the gap between open-source agents and frontier models on such daily tasks, and how much training can close this gap.


W2: Changes in model rankings.

What we intend to show is the complementarity between PhoneWorld and AndroidWorld: on one benchmark, model A is on par with or even slightly better than model B, whereas on the other benchmark, A falls clearly behind B. The examples from Section 4.1 are as follows (task success rate, %):

Model A vs. Model BPhoneWorldAndroidWorld
UI-Venus-1.5-8B vs. GUI-Owl-1.5-8B56.7 vs. 55.8 (A +1 task)47.4 vs. 57.8 (A −12 tasks)
GUI-Owl-7B vs. UI-Venus-Navi-7B38.3 vs. 50.0 (A −14 tasks)46.6 vs. 44.8 (A +2 tasks)

As you pointed out, A leads by only 1–2 tasks on one benchmark, where the two models are essentially on par; on the other benchmark, however, A trails by 12–14 tasks. In other words, looking at only one benchmark suggests that the two models are comparable, or even that A is slightly stronger, whereas the other benchmark reveals that A is clearly weaker in a certain type of scenario. This is because PhoneWorld focuses on Chinese consumer apps while AndroidWorld focuses on English open-source/system apps, making the two benchmarks complementary in both scope and evaluation results.


W2: The verifier in Figure 3.

The verifier in Figure 3 is a simplified illustration due to space constraints, and this task does not appear in our training or test data. Below, we show one verifier actually used in the training set and one in the test set.

Training task: "On Douyin, search for '室友的神操作' ("my roommate's epic move"), open the video detail page, and post the comment '哈哈太真实了' ("haha, so true")." The verifier constrains both the comment target and the comment content:

{
  "type": "sqlite",
  "database_path": "/data/data/com.phoneuse.mdouyin/databases/douyin.db",
  "table": "user_comments",
  "rules": [
    {"field": "target_id", "operator": "==", "value": 3},
    {"field": "content", "operator": "LIKE", "value": "%哈哈太真实了%"}
  ],
  "count_min": 1
}

Test task: "Complete three operations in the 12306 app: (1) search for trains from Beijing West (北京西) to Guangzhou South (广州南), select train G87, and buy a first-class ticket for passenger 代晓静; (2) open the order list, find the G87 order just placed, and refund it; (3) add the Beijing West–Guangzhou South route to the saved routes." The verifier checks both the orders table and the saved-routes table (已退票 = "refunded"):

{
  "type": "sqlite",
  "database_path": "/data/data/com.phoneuse.m12306/databases/railway.db",
  "checks": [
    {
      "table": "orders",
      "rules": [
        {"field": "train_no", "operator": "contains", "value": "G87"},
        {"field": "passenger_name", "operator": "contains", "value": "代晓静"},
        {"field": "status", "operator": "contains", "value": "已退票"}
      ],
      "count_min": 1
    },
    {
      "table": "favorite_routes",
      "rules": [
        {"field": "from_station", "operator": "contains", "value": "北京西"},
        {"field": "to_station", "operator": "contains", "value": "广州南"}
      ],
      "count_min": 1
    }
  ]
}

(Continued in Part 4/4.)

Comments on the paper

1 comment
~John_Constantine1OpenReview ↗

I struggle to identify a substantial research contribution in this paper, whether in problem formulation, methodology, conceptual insights, or empirical findings. MobileGym [1] has already explored constructing verifiable mobile environments for GUI agent training, including the use of coding agents for app implementation and iterative refinement. PhoneWorld largely repackages familiar practices—screenshot classification, frequency-based prioritization, PRD generation, and agentic coding—without establishing what fundamentally new problem it solves or what new scientific insight it offers. Simply constructing another collection of mock apps is not, by itself, a compelling research contribution.

Furthermore, the app construction process is described vaguely, lacking sufficient technical details, meaningful ablations, or comparisons against existing coding-agent-based approaches. The reported training gains merely reinforce the already established utility of synthetic environments, rather than demonstrating a distinctive contribution of this work. Overall, I find the paper lacking in research novelty and insufficiently motivated beyond engineering implementation. Separately, the public arXiv version prematurely states "Published as a conference paper at ICLR 2027".

[1] Dingbang Wu, Rui Hao, Haiyang Wang, Shuzhe Wu, Han Xiao, Zhenghong Li, Bojiang Zhou, Zheng Ju, Zichen Liu, Lue Fan, and Zhaoxiang Zhang. MobileGym: A Verifiable and Highly Parallel Simulation Platform for Mobile GUI Agent Research. In Proceedings of EMNLP 2026, pp. 23223–23247, 2026. https://aclanthology.org/2026.emnlp-main.1009/

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.