- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents WIA, a framework that connects world modeling with action generation for digital agents. It introduces WIA-Corpus and WIA-Bench across seven digital domains, trains WIA-WM at different model scales, and explores its use in simulated training and inference. The authors further propose WIA-WAM, combining DPI and CSPO to jointly learn action generation and environment prediction.
Critical strengths
-
I find the multi-domain dataset and benchmark to be a valuable contribution. The paper covers seven different environments and provides both in-domain and out-of-domain evaluations. I also appreciate the extensive experiments across different model scales and downstream tasks.
-
The authors have made a substantial effort to address the concerns raised in the review. In particular, the additional cross-family evaluations, human evaluation, and long-horizon experiments give me more confidence in the world-model results. The new DPI/CSPO ablations are also helpful in understanding the contributions of the proposed components.
Critical weaknesses
-
I would still be interested in seeing more evidence of long-horizon simulation reliability. The additional experiments are encouraging, although the human evaluation covers a relatively small subset and the multi-step tests are limited to 2–3 transitions. Evaluating longer interactions would further strengthen the paper.
-
The new ablations make the role of DPI much clearer. The additional improvements from CSPO are relatively modest, however, and I think its contribution could be better understood through further analysis of prediction quality and downstream behavior. It would also be interesting to explore initialization from the fully trained WIA-WM in future work.
Overall, I appreciate the authors' detailed responses and the additional experiments. Most of my original concerns have been addressed, and I am happy to raise my rating from 2 to 3. I believe this is a useful contribution, and I hope the final version can incorporate these additional results and further clarify the remaining limitations.
Questions for authors
None.
10 comments
Response 1 — W1 part1: Cross-family judge robustness
We evaluate the OOD split with two additional judges, Kimi-K3 and Qwen3.8-max.
Kimi-K3 judge.
Table 1. OOD evaluation under the Kimi-K3 judge.
| Model | Android | MCP | OS | Search | SWE | Terminal | Web | Avg | MAvg |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6-luna | 57.32 | 52.46 | 59.98 | 50.16 | 60.24 | 66.85 | 58.25 | 58.28 | 57.90 |
| GPT-5.6-terra | 57.98 | 51.59 | 61.64 | 50.87 | 61.64 | 67.48 | 58.97 | 59.10 | 58.59 |
| GPT-5.6-sol | 58.91 | 54.07 | 63.41 | 50.83 | 63.36 | 69.48 | 60.13 | 60.42 | 60.03 |
| Claude-Sonnet-5 | 56.87 | 59.26 | 58.70 | 44.06 | 48.08 | 52.99 | 60.11 | 53.48 | 54.30 |
| Claude-Opus-4.8 | 57.24 | 63.92 | 59.51 | 50.22 | 60.12 | 64.26 | 60.59 | 58.94 | 59.41 |
| GLM-5.3 | 49.08 | 60.61 | 54.18 | 35.41 | 33.05 | 51.99 | 55.54 | 46.98 | 48.55 |
| MiMo-v2.5-pro | 53.36 | 56.13 | 56.40 | 47.16 | 55.08 | 60.10 | 56.13 | 54.70 | 54.91 |
| Qwen3.8-27B | 47.13 | 51.17 | 53.64 | 42.18 | 52.51 | 57.66 | 53.15 | 50.95 | 51.06 |
| Qwen3.8-max | 54.15 | 54.79 | 55.86 | 46.24 | 62.07 | 68.42 | 60.45 | 57.69 | 57.43 |
| DeepSeek-V4-Flash | 53.17 | 54.74 | 56.49 | 47.75 | 53.33 | 58.92 | 57.25 | 54.37 | 54.52 |
| DeepSeek-V4-Pro | 54.31 | 54.93 | 57.87 | 49.36 | 54.82 | 62.11 | 57.37 | 55.79 | 55.82 |
| Qwen3.5-4B | 46.72 | 48.45 | 53.93 | 37.14 | 41.95 | 44.18 | 50.79 | 45.54 | 46.16 |
| Qwen3.5-9B | 50.66 | 52.51 | 55.86 | 38.64 | 47.58 | 47.13 | 52.72 | 48.63 | 49.30 |
| Qwen3.5-27B | 54.85 | 55.52 | 58.71 | 40.82 | 53.46 | 59.84 | 57.56 | 53.99 | 54.40 |
| Qwen3.5-35B-A3B | 54.63 | 53.37 | 55.65 | 38.88 | 50.05 | 54.02 | 54.86 | 51.15 | 51.64 |
| Qwen-AgentWorld | 52.76 | 56.46 | 60.62 | 54.59 | 61.27 | 64.40 | 58.32 | 58.52 | 58.35 |
| WIA-WM-4B | 30.77 | 51.82 | 50.24 | 44.18 | 53.62 | 59.44 | 52.01 | 48.81 | 48.87 |
| WIA-WM-9B | 53.87 | 59.39 | 58.17 | 53.60 | 56.60 | 62.90 | 57.17 | 57.19 | 57.38 |
| WIA-WM-27B | 56.03 | 60.58 | 58.59 | 51.72 | 62.29 | 65.12 | 58.90 | 58.91 | 59.03 |
| WIA-WM-35B-A3B | 54.78 | 54.07 | 59.14 | 57.14 | 59.19 | 67.55 | 58.90 | 59.15 | 58.68 |
Qwen3.8-max judge.
Table 2. OOD evaluation under the Qwen3.8-max judge.
| Model | Android | MCP | OS | Search | SWE | Terminal | Web | Avg | MAvg |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6-luna | 46.67 | 45.49 | 49.04 | 36.96 | 50.98 | 54.59 | 51.83 | 48.05 | 47.94 |
| GPT-5.6-terra | 47.03 | 44.18 | 50.51 | 37.40 | 51.73 | 54.49 | 52.40 | 48.47 | 48.25 |
| GPT-5.6-sol | 48.68 | 46.43 | 53.34 | 35.54 | 53.34 | 56.57 | 52.74 | 49.57 | 49.52 |
| Claude-Sonnet-5 | 45.66 | 52.15 | 55.25 | 33.20 | 40.32 | 40.66 | 51.71 | 44.44 | 45.57 |
| Claude-Opus-4.8 | 47.22 | 55.49 | 55.54 | 35.25 | 53.21 | 49.46 | 53.64 | 49.18 | 49.97 |
| GLM-5.3 | 42.87 | 50.76 | 48.72 | 27.62 | 30.69 | 44.55 | 48.61 | 40.68 | 41.98 |
| MiMo-v2.5-pro | 43.87 | 45.97 | 52.96 | 35.27 | 45.96 | 46.60 | 49.17 | 45.32 | 45.68 |
| Qwen3.8-27B | 38.83 | 41.27 | 47.61 | 31.18 | 44.47 | 46.01 | 44.87 | 41.87 | 42.04 |
| Qwen3.8-max | 45.06 | 45.70 | 51.52 | 34.24 | 51.38 | 54.12 | 52.96 | 47.84 | 47.85 |
| DeepSeek-V4-Flash | 44.82 | 45.52 | 51.78 | 35.56 | 44.95 | 47.94 | 48.80 | 45.32 | 45.62 |
| DeepSeek-V4-Pro | 46.52 | 44.26 | 54.67 | 34.66 | 46.95 | 49.78 | 50.70 | 46.63 | 46.79 |
| Qwen3.5-4B | 38.17 | 39.04 | 48.82 | 28.46 | 36.80 | 35.68 | 42.29 | 37.93 | 38.47 |
| Qwen3.5-9B | 41.54 | 41.67 | 50.88 | 28.91 | 41.28 | 37.80 | 44.63 | 40.41 | 40.96 |
| Qwen3.5-27B | 46.57 | 44.45 | 56.13 | 29.77 | 45.58 | 47.99 | 49.56 | 45.30 | 45.72 |
| Qwen3.5-35B-A3B | 46.14 | 42.78 | 49.33 | 28.63 | 41.92 | 43.05 | 45.65 | 42.03 | 42.50 |
| Qwen-AgentWorld | 45.48 | 44.07 | 57.21 | 43.97 | 51.01 | 49.82 | 50.83 | 49.08 | 48.91 |
| WIA-WM-4B | 25.45 | 43.69 | 46.90 | 37.40 | 48.34 | 47.60 | 45.94 | 42.13 | 42.19 |
| WIA-WM-9B | 45.26 | 50.51 | 55.06 | 43.32 | 51.10 | 50.90 | 51.34 | 49.38 | 49.64 |
| WIA-WM-27B | 48.29 | 52.81 | 56.33 | 43.87 | 54.09 | 51.37 | 51.14 | 50.79 | 51.13 |
| WIA-WM-35B-A3B | 47.41 | 46.17 | 56.72 | 49.00 | 52.14 | 53.60 | 52.40 | 51.36 | 51.06 |
The absolute scales differ between judges, but WIA-WM remains competitive across scales. WIA-WM-9B records Avg/MAvg of 57.19/57.38 under Kimi-K3 and 49.38/49.64 under Qwen3.8-max; WIA-WM-27B reaches 58.91/59.03 and 50.79/51.13; and WIA-WM-35B-A3B reaches 59.15/58.68 and 51.36/51.06. Among the WIA-WM variants, 35B-A3B has the highest Avg under both judges, while 27B has the highest MAvg. In Avg, 27B is comparable to Claude-Opus-4.8 under Kimi-K3 (58.91 vs. 58.94) and exceeds both GPT-5.6-sol and Claude-Opus-4.8 under Qwen3.8-max (50.79 vs. 49.57 and 49.18). These results indicate that WIA remains highly competitive with frontier models without relying on a single LLM judge for evaluation.
Response 2 — W1 part2: Programmatic F1 and human cross-evaluator evidence
As a second part of our W1 response, we report programmatic token F1 and a Codex-assisted human evaluation on WIA-Bench-OOD.
1. Non-LLM F1 evaluation. Token F1 is independent of LLM scoring. The table below reports all 19 evaluated models on the full 1,739 OOD rows; Avg weights examples equally and MAvg weights domains equally.
Table 3. OOD programmatic observation-text token-F1 evaluation.
| Model | Android | MCP | OS | Search | SWE | Terminal | Web | Avg | MAvg |
|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6-luna | 62.66 | 35.62 | 61.50 | 22.95 | 56.71 | 63.47 | 67.06 | 53.67 | 52.85 |
| GPT-5.6-terra | 64.82 | 33.84 | 63.87 | 24.47 | 59.19 | 63.70 | 67.32 | 54.91 | 53.89 |
| GPT-5.6-sol | 67.00 | 37.60 | 73.23 | 26.46 | 62.04 | 65.90 | 70.12 | 58.30 | 57.48 |
| Claude-Sonnet-5 | 67.63 | 66.61 | 89.01 | 33.69 | 50.95 | 54.64 | 73.33 | 60.45 | 62.27 |
| Claude-Opus-4.8 | 66.54 | 68.29 | 86.88 | 32.55 | 62.72 | 63.65 | 75.28 | 63.69 | 65.13 |
| MiMo-v2.5-pro | 69.54 | 56.31 | 85.52 | 35.25 | 51.87 | 58.22 | 72.91 | 60.51 | 61.37 |
| Qwen3.8-27B | 65.88 | 44.71 | 77.31 | 33.37 | 50.63 | 56.12 | 63.19 | 55.74 | 55.89 |
| Qwen3.8-max | 74.59 | 54.31 | 85.33 | 32.61 | 59.65 | 66.63 | 78.81 | 64.23 | 64.56 |
| DeepSeek-V4-Flash | 70.17 | 58.06 | 85.20 | 37.50 | 48.70 | 59.41 | 72.31 | 60.64 | 61.62 |
| DeepSeek-V4-Pro | 71.49 | 49.62 | 89.09 | 38.20 | 48.16 | 59.99 | 71.69 | 60.76 | 61.18 |
| Qwen3.5-4B | 62.33 | 47.14 | 75.66 | 29.74 | 32.24 | 34.47 | 61.53 | 47.69 | 49.02 |
| Qwen3.5-9B | 65.13 | 49.86 | 82.20 | 29.85 | 37.02 | 37.73 | 64.25 | 50.90 | 52.29 |
| Qwen3.5-27B | 69.35 | 51.20 | 88.18 | 32.70 | 45.19 | 50.63 | 70.46 | 57.29 | 58.24 |
| Qwen3.5-35B-A3B | 68.94 | 51.27 | 79.34 | 30.18 | 41.91 | 44.20 | 66.22 | 53.42 | 54.58 |
| Qwen-AgentWorld | 68.00 | 44.37 | 86.86 | 37.69 | 57.64 | 59.36 | 71.90 | 61.04 | 60.83 |
| WIA-WM-4B | 4.94 | 51.67 | 80.38 | 38.75 | 37.56 | 58.10 | 65.92 | 47.19 | 48.19 |
| WIA-WM-9B | 71.60 | 64.31 | 89.37 | 46.42 | 42.02 | 61.63 | 72.31 | 62.57 | 63.95 |
| WIA-WM-27B | 74.77 | 62.18 | 87.24 | 43.21 | 53.71 | 64.59 | 69.25 | 64.04 | 64.99 |
| WIA-WM-35B-A3B | 71.64 | 52.59 | 88.54 | 47.06 | 48.64 | 63.65 | 71.52 | 63.08 | 63.38 |
Among the 19 models, F1 Avg ranges from 47.19 to 64.23. WIA-WM-27B is second in Avg and MAvg, while WIA-WM-9B and 35B-A3B exceed GPT-5.6-sol and Qwen-AgentWorld in both averages. Token overlap is a complementary signal, not a replacement for semantic judging: exact wording can differ despite equivalent state content, and overlap cannot validate fields absent from the reference.
2. Human review conducted with the assistance of the Codex tool. We invited five human volunteers to conduct a blinded review of 56 cases, eight per domain, with Codex used as a tool to facilitate the review workflow and record the evaluations; the volunteers, rather than Codex, made the judgments. Domain columns and Panel Avg show mean ranks; Pro Avg shows the original DeepSeek-V4-Pro score on the same cases (20–100). Position columns give overall rankings.
Experimental setup. For each case, all 5 volunteers ranked the outputs from the different models, casting ranking votes on the same set of outputs. We first averaged the 5 volunteers' ranks for each model on each case, then averaged these per-case mean ranks across the cases in each domain and across the full 56-case subset.
Table 4. Codex-assisted human-panel rankings and DeepSeek-V4-Pro scores.
| Model | Android | MCP | OS | Search | SWE | Terminal | Web | Panel Avg rank | Panel position | Pro Avg | Pro position |
|---|---|---|---|---|---|---|---|---|---|---|---|
| GPT-5.6-sol | 2.700 | 2.950 | 3.250 | 3.225 | 4.175 | 2.500 | 3.475 | 3.182 | 3 | 64.29 | 3 |
| Qwen3.8-max | 3.850 | 3.125 | 3.950 | 3.125 | 2.100 | 3.450 | 2.325 | 3.132 | 2 | 64.39 | 2 |
| DeepSeek-V4-Pro | 2.825 | 2.925 | 2.975 | 3.075 | 3.350 | 3.650 | 4.050 | 3.264 | 4 | 62.82 | 4 |
| Qwen3.5-35B-A3B | 3.175 | 3.375 | 3.625 | 4.575 | 4.325 | 3.825 | 3.975 | 3.839 | 5 | 58.43 | 5 |
| WIA-WM-35B-A3B | 2.450 | 2.625 | 1.200 | 1.000 | 1.050 | 1.575 | 1.175 | 1.582 | 1 | 89.86 | 1 |
The volunteer panel and Pro scores produce the same overall ordering on these 56 cases: WIA-WM-35B-A3B, Qwen3.8-max, GPT-5.6-sol, DeepSeek-V4-Pro, and Qwen3.5-35B-A3B. Agreement extends across all five positions, including the relative ordering of the four baselines. This agreement supports the reliability of WIA-Bench's model rankings.
Response 0
Thank you for the careful and constructive review. We will revise the paper to incorporate the additional evaluation, robustness analyses, data-overlap audits, methodological clarifications, and improved presentation of the method figures described below. The revised figures discussed in W2 clarify the training branches, reward flow, and optimization procedure, and we will align the claims and limitations with the resulting evidence. The following responses address W1, W2, Q1, Q2, and Q3 separately.
Completed figure revisions. We have revised the figures to clarify the overall framework and its training and application workflows. The original Figure 2 used three rows: textual modeling examples illustrating the inputs to the three training stages; the CPT–SFT–RL training pipeline; and world-model simulation applications together with the world action model. We have separated this presentation into two figures. The new overview (revised Figure 2) presents the full framework at a higher level: modeling across seven digital domains, world-model training and WIA-Bench, the three world-model simulation applications (simulation SFT, simulation RL, and inference lookahead), and the world action model. The dedicated training figure (revised Figure 3) retains only the input examples and world-model training pipeline from the original Figure 2. Building on the original Figure 3, the revised WIA-WAM figure (now Figure 4) makes the relationships among the CSPO optimization branches, reward propagation, and training objectives more explicit. We have also added an algorithmic workflow diagram for the three world-model simulation applications on the first page of the appendix (Figure 7, Appendix A). Finally, we have improved the appendix visualizations of world-model rollouts across all seven digital domains (Figures 18–24), making the correspondence between textual modeling and visual state transitions clearer in each domain.