- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces, WildAWM, a scalable pipeline for synthesizing persona-driven and cross-application environments. Empirical results demonstrate the introduced pipeline could synthesize environments that (1) facilitate agent training and (2) are diverse and solvable.
Critical strengths
1. The manuscript has a timely good motivation. Realistic computer use span across softwares and interfaces; existing environment synthesizing methods do not cover this.
2. The synthesizing pipeline is comprehensive. The pipeline covers different aspects of environment synthetic: diversity (different persona), cross-agent interface, solvability, difficulty and verifier reliability.
Critical weaknesses
1. The main claim is not fully supported by the experiments. The Introduction motivates WildAWM by arguing that realistic computer use spans multiple softwares and interaction interfaces, whereas many existing environment-synthesis approaches focus on a single application or interface. However, the experiments do not directly establish the benefit of this design. Table 1 provides a comparison of method capabilities, while Table 2 demonstrates improvements over the underlying base models after training on WildAWM. Neither provides a direct comparison with agents trained on single environments or agents trained using Agent World Model.
2. The method is not fully ablated. The proposed synthesis pipeline has two major steps: 1) persona-driven diversification and 2) cross-interface task synthesis. However, the existing ablations primarily examine the quality gates used during task synthesis (Table 4) and components of the training procedure (Table 5). They do not isolate the contributions of persona introduced diversity and cross-interface diversity.
3. None of synthesized environment's property is validated by human. The proposed generation pipeline generate diverse and solvable tasks. Human judgement even on a small subset would provides more guarantees on the validity of the tasks and the properties above.
Questions for authors
-
Would you please provide a direct comparison with agents trained on single environments or agents trained using Agent World Model?
-
Would you please provide an ablation on persona-driven diversification's contribution to synthesized environments?
6 comments
Response to Reviewer XKSb (2/3)
Weakness 2 & Question 2
The method is not fully ablated. The proposed synthesis pipeline has two major steps: 1) persona-driven diversification and 2) cross-interface task synthesis. However, the existing ablations primarily examine the quality gates used during task synthesis (Table 4) and components of the training procedure (Table 5). They do not isolate the contributions of persona introduced diversity and cross-interface diversity.
Would you please provide an ablation on persona-driven diversification's contribution to synthesized environments?
Thank you for this valuable suggestion. Our existing analyses provide initial evidence of the diversity achieved by the synthesis pipeline: WildAWM states are larger (median 52.4K vs. 1.1K tokens) and more diverse than CUA-Gym's (higher Vendi score at every sample size, nearest-neighbor cosine distance 0.25 vs. 0.20, Figure 3a), and task coverage keeps growing as synthesis scales from 6 to 36 personas (134 domains and 91 task types without saturation, Figure 4c).
To directly evaluate its contribution to downstream training, we fine-tuned Qwen3.6-35B-A3B using the same SFT recipe and teacher (GLM-5.2), varying persona coverage from 4 to 16. To control the number of samples, each set uses only 32 tasks, matching the number of tasks available with 4 personas, rather than the full training data.
| Training tasks from | O2W | WA | QCB | CLE | CUG | OSV | OS-2 | ALE | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| 4 personas | 63.67 | 54.87 | 53.69 | 51.17 | 47.56 | 27.12 | 5.42 | 18.23 | 40.22 |
| 16 personas | 67.28 | 61.50 | 54.23 | 50.67 | 56.06 | 39.20 | 6.74 | 20.52 | 44.53 |
We find that:
- With the same number of tasks, raising persona coverage from 4 to 16 improves the average from 40.22 to 44.53 (+4.31) and improves seven of the eight benchmarks.
- The largest gains are on the GUI benchmarks CUG (+8.50) and OSV (+12.08). These results suggest that broader persona coverage improves transfer even with the same number of training tasks.
For cross-interface synthesis, the matched comparison with GUI-only CUA-Gym and MCP-only AWM environments (Table 3) provides direct evidence of the benefits of our cross-interface approach: under the same teacher and data budget, WildAWM transfers best on average, and on cross-interface validation tasks, the single-interface models after RL achieve 12.12 and 6.06 pass@1, compared with 35.98 for WildAWM.
We have updated the paper accordingly, with changes shown in blue: Sec. 5.2 summarizes the persona ablation, and Appendix A.4.1 (Table 13) reports it in full.
Response to Reviewer XKSb (3/3)
Weakness 3
None of synthesized environment's property is validated by human. The proposed generation pipeline generate diverse and solvable tasks. Human judgement even on a small subset would provides more guarantees on the validity of the tasks and the properties above.
Thank you for your suggestion. We have conducted additional human evaluations of both task validity and verifier reliability.
For the tasks, we randomly sampled 100 tasks each from WildAWM, AWM, and CUA-Gym. Three annotators independently read each task description. They counted the applications and data sources involved in each task and independently assessed its validity and whether it requires multiple interaction interfaces, with majority voting used for categorical judgments:
| WildAWM | AWM | CUA-Gym | |
|---|---|---|---|
| Apps and data sources per task | 4.06 | 1.14 | 1.29 |
| Valid tasks | 99% | 87% | 96% |
| Multi-interface tasks | 100% | 0% | 7% |
The annotators agree closely on all three judgments (Krippendorff's α = 0.95 for the counts, Fleiss' κ = 0.97 for multiple interfaces, and Gwet's AC1 = 0.84 for validity).
For the verifier, we manually inspected 100 rollouts, reviewing the task, the trajectory, and compare the initial and final world states without seeing existing verdicts. The reliability study with the new human rows is:
| Comparison | Cons. | Precision | Recall | F1 | κ |
|---|---|---|---|---|---|
| Verifier vs. GLM-5.2 | 0.95 | 0.98 | 0.93 | 0.95 | 0.90 |
| Verifier vs. GPT-5.6-Sol | 0.92 | 0.85 | 0.98 | 0.91 | 0.83 |
| Verifier vs. Claude-Opus-5 | 0.92 | 0.98 | 0.88 | 0.93 | 0.84 |
| Verifier vs. Judge majority | 0.95 | 0.98 | 0.93 | 0.95 | 0.90 |
| Verifier vs. Human (new) | 0.88 | 0.90 | 0.87 | 0.88 | 0.76 |
| GLM-5.2 vs. Opus-5 | 0.97 | 1.00 | 0.95 | 0.97 | 0.94 |
| GPT-5.6-Sol vs. Opus-5 | 0.89 | 1.00 | 0.80 | 0.89 | 0.79 |
| Majority vs. Human (new) | 0.93 | 0.92 | 0.94 | 0.93 | 0.86 |
Against the human labels (52 successes, 48 failures), the errors of each judge are:
| Judge | Cons. | κ | False acceptance | False rejection |
|---|---|---|---|---|
| Our verifier | 0.88 | 0.76 | 10.4% | 13.5% |
| Judge majority | 0.93 | 0.86 | 8.3% | 5.8% |
We find that:
- Task validity: 99% of sampled WildAWM tasks are judged valid, the highest among the three sources, despite involving more than three times as many applications and data sources. All sampled WildAWM tasks require multiple interfaces, supporting the intended cross-interface design.
- Verifier validity: the verifier agrees with humans on 88% of rollouts (κ = 0.76). In a blind corruption audit, it also rejected 39 of 41 final states that an independent agent had corrupted to violate the task.
- Observed solvability: although these tasks are designed to be hard and only about three rollouts per task were inspected, humans confirmed a correct completion by the agent for 24 of the 33 tasks.
We have updated the paper accordingly: Table 4 adds the rows "Human" and "Majority vs. Human", Sec. 5.1 adds sentences reporting the human task assessment, the human comparison, and the corruption audit, Appendix A.6.1 (Table 14) reports the human task assessment, Appendix A.6.2 describes the annotation protocol and reports the per-judge errors (Table 15), and Appendix A.6.3 reports the corruption audit (Tables 16 and 17). We also have case study examples in Appendix C.
Response to Reviewer XKSb (1/3)
Thank you for the constructive review and the clear suggestions for additional comparisons and ablations. We would like to respond with additional results and clarifications as follows. We also updated the PDF, which is marked in blue to highlight the changes.
Weakness 1 & Question 1
Thank you for this helpful suggestion. We would first like to provide some additional context from our existing analyses of the cross-interface design. The policy analysis (Figure 5) shows that training raises the interface-switch rate from 18.4% to 25.5%, close to GLM-5.2, whereas RL from the base model alone lowers it to 15.7%. WildAWM environments are also more diverse than CUA-Gym's (higher Vendi score at every sample size, Figure 3a).
To directly address your question, we conducted a controlled comparison with single-interface environments. We fine-tuned Qwen3.6-35B-A3B on trajectories from CUA-Gym (GUI), AWM (MCP), and WildAWM, using the same teacher, data budget, and training recipe. We then applied RL to each SFT model under matched settings:
We find that:
We have updated the paper accordingly, with changes shown in blue: Table 3 now includes CUA-Gym-SFT, AWM-SFT, WildAWM-SFT, CUA-Gym-RL, and AWM-RL, and point (3) of "Main Results" in Sec. 4.