- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces PhoneWorld, a trace-grounded framework that converts such evidence into runnable, resettable, and verifiable Android environments. PhoneWorld leverages mocked-app environments as a reset-friendly proxy for real apps. Combining PhoneWorld's mock-app rollouts with real-app rollouts yields performance gains over real-app RL alone on both real-phone tasks and the AndroidWorld benchmark.
Critical strengths
a. PhoneWorld successfully leverages mocked-app environments as a reset-friendly proxy for real apps. b. PhoneWorld's mock-app rollouts yields performance gains over real-app RL alone.
Critical weaknesses
- Environments are built bottom-up with hand-crafted mocks, which the authors admit is hard to scale. Without a top-down approach that covers the real task distribution, the benchmark's validity is unestablished, and agent results inside this space may not transfer to the settings AndroidWorld targets. This attacks the paper's foundation, not a peripheral detail.
- There is no direct, fine-grained comparison with AndroidWorld, and the Figure 2 justification is unconvincing. The verifier is also not positioned against prior work (e.g., WorldCoder-bench), leaving unclear whether gains come from the environment, verifier, or test-set construction. The paper claims to be both new and better but demonstrates neither.
Questions for authors
- Experimentally, more direct and fine-grained comparisons with AndroidWorld are needed to demonstrate the value of this benchmark. The experimental justification in Figure 2 also appears less convincing.
- I recognize the value of mocks, but I would like to ask about the generalizability of the constructed environment. I understand that scaling the current approach is difficult; it feels like a bottom-up method for building environments, which may not be efficient enough. I believe a more efficient approach would be top-down. Am I misunderstanding something?
- There are many ways to write the skeleton as an app specification. How can this step be made efficient? If a weaker model were used for this step, would it affect both the results and the efficiency? Are there alternative approaches?
- How does the specific implementation of your programming verifier differ from prior methods? For instance, compared to WorldCoder-bench and similar approaches.
5 comments
Response to Reviewer V8xh (2/2)
W2/Q4: Relation of the verifier to prior work, and the source of the gains.
Thank you for pointing out that the positioning of our verifier was missing. We respond in two parts.
- Relation to WorldCoder-Bench: different objects of verification. WorldCoder-Bench verifies the generated program itself: it uses StateProbe to execute generated 3D-world programs in a controlled browser and checks, against expert-written behavioral contracts hardened by mutation testing, whether the program's state transitions conform to the specification. It thus answers the question "is this code implemented correctly?" The PhoneWorld verifier instead verifies the task outcome of an agent in a fixed environment: once built and accepted, the environment is taken as given, and the verifier checks whether the final SQLite state at the end of a rollout, or the returned answer, satisfies the task's success condition (71 of the 120 tasks are state-based and 49 are answer-based). The former verifies the environment, whereas the latter verifies the agent; the two sit at different stages of the pipeline and are not competing solutions to the same problem. We will add this distinction in the revision.
- The gains come from the environment itself. To identify the source of the gains, we collected the same amount (10K steps) of data in MobileWorld, mixed it with the same 26K AW steps, and compared the results on AndroidWorld (see the table in point 3 of our response to W1/Q2). On top of the same 26K AW steps, adding 10K PW steps yields a gain of 16.4 points, whereas adding 10K MW steps yields 4.3 points, a difference of 12.1 points. The two settings use exactly the same data volume, mixing ratio, and AW portion; the only difference is the environment from which the 10K steps are collected. The gains therefore come from the PhoneWorld environment itself, not from simply adding more data. Moreover, reducing AW data from 36K to 26K costs only 1.7 points (56.9 → 55.2), so the 71.6 cannot be attributed to the change in AW data volume either.
Q3: Efficiency of specification generation, the effect of a weaker model, and alternatives.
Efficiency. Generating the specification from the input screenshots and trajectories takes about 6–10 minutes, whereas the full mock-app construction pipeline typically takes 5–6 hours. Specification generation thus accounts for only about 3% of the total time and is not the bottleneck of the pipeline.
Using a weaker model. We rebuilt App2 with the weaker Sonnet 4.6, with which specification generation takes 6–8 minutes. The fidelity results are as follows (metrics as in Table 5):
| Model | RPC | NPC | FPC | DER | TER | VF |
|---|---|---|---|---|---|---|
| Opus 4.6 | 97.7 | 94.9 | 76.7 | 77.1 | 82.5 | 70.0 |
| Sonnet 4.6 | 94.6 | 89.2 | 68.3 | 74.2 | 80.3 | 63.4 |
The weaker model still recovers most pages and page transitions (RPC, DER, and TER drop by only 2.2–3.1 points), but functional coverage (FPC, −8.4) and visual fidelity (VF, −6.6) degrade noticeably. This shows that stronger models reproduce functional and visual details more faithfully, which we believe is the key factor whether a mock app can transfer to the real app.
Alternatives. Reverse Engineering has recently attracted considerable attention; for example, open-source projects such as Reverse Engineering Anything can already decompile APKs directly. However, we see three issues with this approach: (1) security concerns; (2) the app structure obtained by reverse engineering does not reflect users' real usage needs; and (3) reverse engineering cannot fully recover the app's resource assets, so the reconstructed app has poor visual fidelity. We agree that more efficient ways of constructing the specification are worth pursuing. Considering the trade-off between efficiency and quality, we think our current approach is a reasonable choice, and we will continue to explore this direction in future work.
Once again, we sincerely thank you for the time and effort devoted to reviewing our paper and for the valuable suggestions. If our responses have resolved your concerns, we would be very grateful if you could consider raising your score. If you have any further questions, we would be more than happy to engage in further discussion.
Summary of rebuttal to Reviewer V8xh
Dear Reviewer V8xh,
Thanks again for your time and effort. As the discussion period is nearing its end, we would be grateful if you could review our response.
We would like to clarify that the environments are not hand-crafted: a coding agent builds each app automatically from a specification induced top-down from real user trajectories, and this framework is scalable that our internal suite has grown from 34 to about 100 apps. On transfer, adding mock-app rollouts improves all three seeded RL runs on 150 real-phone tasks (41.33 ± 0.67 → 45.56 ± 0.38), and PhoneWorld data generalize better to AndroidWorld than an equal amount of MobileWorld data. We also distinguish our verifier from WorldCoder-Bench: the latter verifies generated programs against behavioural contracts, whereas ours checks the final state of an agent's rollout.
If our responses have addressed your concerns, we would be grateful if you could continuously support our work and kindly consider raising your score; if you have any other questions, we would be glad to discuss further. Thank you again for your time.
Response to Reviewer V8xh (1/2)
We sincerely thank you for recognizing the value of mock environments and for the constructive comments. We address each point below.
We are concerned that there may be some misunderstanding of our method, and we would like to clarify it further below.
The distinctive feature of PhoneWorld is that it provides controllable environments for the consumer apps that users actually use, addressing the problem that closed-source consumer apps cannot be reliably reset and verified. The comparison with AndroidWorld is as follows:
Figure 2 examines how scaling along two dimensions, data volume and the number of apps, affects performance on each benchmark:
We hope these additions make the experimental results more convincing.
(Continued in Part 2/2.)