- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The motivation is that evaluating computer agents on real desktop apps and live websites is not always feasible because of cost and irreversible consequences (e.g. sending a message). Hence there is a high value in constructing world models (WM) of apps and websites. This paper proposes, Multi-Turn On-Policy Distillation (Multi-Turn OPD), to train an open base model (e.g. Qwen3.5-9B) to predict a text representation of the computer state and availabe actions, given a current trajectory. Multi-Turn OPD has two phases, involving teacher and student models:
- starting with a base set of trajectories and some fixed agent, generate next state predictions using the student WM and train the teacher WM to predict the ground truth next states from agent trajectories generated using the student-WM. The teacher is given privileged data (summaries of the base trajectory data) in this phase.
- Distill the teacher WM into the student WM
Experiments were done on desktop Office apps (Word, Excel, Powerpoint). Evaluation of WM accuracy is done by a surrogate measure of agreement between task outcomes when a fixed agent executes on the WM and on the actual app, as judged by some other strong VLM.
Critical strengths
The topic is timely. That's all I can say about this work.
Critical weaknesses
Critical:
This part of the method makes no sense: using the student WM's predicted state as part of the history, pairing it with the original trajectory's action, and training the teacher WM to predict the original trajectory's next state. Suppose your task is "open the file menu in Excel" and due to some error by the student WM, your history has a blank desktop page as the current predicted state , and your recorded action is click("file menu") (actually correct if excel were open), and suppose your dataset's next state is a correct Excel app with an open file menu. How does this training tuple make any sense? Answer me this: If I tell you the current state is a blank desktop and the action taken is click("file menu"), would you tell me the next state should be an open Excel window with the file menu open? Seriously? This makes no sense.
Other problems:
- Your chosen apps (excel, word, powerpoint) for evaluating the method are all desktop apps where every action is easily reversible. There is no practical motivation use a world model for these apps at all. I know your method can be applied to problems that have irreversible or hard-to-reverse steps (e.g. sending an email, making a payment), but seriously, choosing these desktop apps in a world modeling paper is very poor motivation.
- Unclear from the main paper how many steps are needed for the tasks, so it's unclear how effective the WM will be in practice for long task (e.g. OSWorld 2.0) that would take a human >1hr and hundreds to steps to solve.
- Unclear from the main paper the latency and cost of running the world model. If the world model takes longer to run and needs a GPU (costly these days), I'll just use the actual app for eval anyway and use some other heuristics to get around the irreversible points. A windows virtual machine is less than $100 per month, while a GPU node for running your world model easily costs 10x more.
- The WM not being able to work with pure screenshot-only agents is a critical limitation. Most SoTA agents (opus, gpt) are screenshot-only.
- If the app has some hidden state transitions (e.g. backend database or server state) that is not immediately shown in the UI, then how would the world model construction work?
- The offline action-matching baseline is a terrible baseline. Many computer tasks are pretty much all or nothing, where success requires all actions to be optimal. Saying success when half of predicted actions match a recorded trajectory is a terribly loose condition.
- Task-outcome protocol section: Why not also 5 rollouts per task in the world model? If your intent is to test whether the world model can serve as a surrogate of the VM, then you also need multiple rollouts of the world model to evaluate reliability in multiple runs.
- Your judging protocol assumes that the task completion is checkable from the final state alone. There are many tasks with intermediate criteria that require the full trajectory to be judged.
Minor:
- Section 3.1 writes initial screenshot and initial state separately. The screenshot is already part of the initial state, no point separating them.
- Section 3.1: the available controls at some time step is part of the action space, it should not be confused with the state.
- Section 3.2: You use the term "reference summaries" in the "From response-level OPD..." paragraph without explanation, then you describe it later in another paragraph "Error-conditioned teacher preparation". Improve your writing structure.
- Unclear what model is the teacher. Still Qwen3.5-9B? Not clear from the main paper.
Questions for authors
None that could affect my score.
12 comments
Reply to W2 (Other 1): Motivation for choosing Office applications
We thank the reviewer for the comment. We first clarify the motivation of this paper and then explain why we chose Office as the testbed.
(1) The starting point of this paper is to reduce evaluation cost and improve scalability, not to avoid irreversible operations. The motivation in the introduction is that evaluating in real environments requires setting up the environment, executing actions, and verifying results; repeating these steps across many tasks, agents, and runs is costly, so an evaluation approach is needed that is cheaper but still retains full interaction. Reversibility does not remove reset, execution, and verification overhead; these are the costs measured in our repeated-evaluation setting. In a virtual machine, a snapshot must be restored and a template reopened before every rollout, every step requires taking a screenshot and converting it into a textual description, and timeouts require reruns; these overheads have nothing to do with whether operations are reversible. In our experiments, the VM throughput is only 8.71–15.08 tasks/h, whereas the world model reaches 35.84–114.33 tasks/h, 2.91–7.77× faster; one evaluation of 100 tasks × 5 rollouts takes 33–57 hours on the VM but only 4–14 hours with the world model. Research often requires repeatedly comparing multiple checkpoints, configurations, and agents, which multiplies this gap. In addition, every VM instance used for evaluation requires a license for commercial software such as Office; the larger the degree of parallelism, the heavier the burden of subscription fees and account management. The world model requires real software only when collecting training data, and evaluation can be run in batches directly on GPUs.
(2) We chose Office because it can be used to validate the world model. To test whether a world model can replace the real environment, the same agent must be able to execute the same tasks repeatedly in both environments so that the results can be compared. Irreversible scenarios such as sending emails or making payments cannot be executed repeatedly in this way, so the reliability of the world model cannot be tested there. Only after demonstrating that the world model agrees with the real environment in a verifiable domain is there a basis for extending it to irreversible scenarios.
(3) Office is itself one of the most important and hardest-to-model scenarios for computer-use agents. For example, Excel has on average 374.5 controls per step before filtering, along with many menus, dialogs, and formatting states, which pose a stringent test of state prediction and long-horizon error accumulation.
Reply to W3 (Other 2): Number of task steps and long-horizon tasks
We thank the reviewer for the comment. The reviewer asks how many steps the test tasks require and whether the world model is effective on longer tasks. Statistics on task steps are given in the test-set statistics in the appendix; we explain in two points below.
(1) Task length. The reference trajectories of the 100 test tasks have 2–12 steps, 5.24 steps on average; among them, 28 tasks have 2–3 steps, 31 have 4–5, 27 have 6–8, and 14 have 9–12. PowerPoint tasks are the longest, with 20 of 33 tasks requiring at least 6 steps. For comparison, OSWorld-Human annotates the minimum number of human steps required to complete OSWorld tasks (counting each action as one step): on average 7.5, 7.8, and 13.2 steps for LibreOffice Writer, Impress, and Calc, respectively, showing that our task lengths are of the same order of magnitude as the office tasks in OSWorld.
(2) Reliability on long tasks. The longer the task, the more pronounced the accumulation of the world model's own prediction errors, which is exactly the problem Multi-Turn OPD aims to solve. Table 2 shows that from single-turn replay to multi-turn replay, all state metrics of general-purpose models drop markedly. Taking GPT-5.5 as an example, description consistency, control hit rate, Text+Type, Macro IoU, and Micro IoU drop by 17.59, 10.50, 12.87, 18.26, and 16.15 percentage points, respectively, whereas CUWorldEval's metrics drop by only 4.31–6.47 percentage points. In multi-turn replay, CUWorldEval's description consistency (72.24), Text+Type (54.65), Macro IoU (66.55), and Micro IoU (59.76) are all the highest among all models.
Reply to Reviewer F9tJ
Thank you for your detailed review and for the time you invested in examining our training and evaluation design. Below, we address your questions about the supervision target, evaluation cost, and the reliability of the outcome judgments.
Reply to W1 (Critical): Training the teacher on student-generated histories
We thank the reviewer for this concrete example, which helps clarify our training setup. Besides the state predicted by the student, the teacher model also receives two other kinds of inputs, and taking them into account makes the meaning of this training example clearer.
(1) The history contains not only the states predicted by the student. It also contains the task instruction, the real initial screenshot O_0 and initial state S_0, and all previously recorded actions together with the current action. These all come from the real trajectory rather than being generated by the model.
(2) The teacher model can also see the summary Z of the entire reference trajectory. The very purpose of training the teacher is to enable it, with the help of this privileged information, to predict the correct next state even when the student's history may contain errors (Section 3.2).
With these inputs in mind, consider the example again: Excel is already open at the start of the task, no subsequent action closes Excel, and the recorded action clicks a control that actually exists in Excel. Taking these inputs together, along with the fact that the action trajectory is fixed, "Excel open with the File menu expanded" is the correct inference, whereas "blank desktop" is the one that contradicts all other information and is clearly a prediction error of the student. Z describes the state change at each step of the reference trajectory, and the teacher can use it to recognize this error and give the correct next state.
A predicted state is only the model's estimate, not the environment itself. If the model predicts incorrectly, the state of the real application does not change accordingly, so the real next state remains the correct training target. Another possible design would be to let the model "keep predicting along the erroneous state". We did not adopt this design for three main reasons:
Training on the model's own generated histories with the real trajectory as the target is a standard approach to error accumulation in sequence prediction and multi-step model learning, e.g., Scheduled Sampling [1], Data as Demonstrator [2], and DAgger [3]. On this basis, we additionally provide the teacher model with the privileged summary Z, further removing ambiguity in the target so that the teacher can still provide supervision pointing to the real trajectory when the student's history contains errors. This is the part of our method specifically designed to guide the student model. The experimental results also support this design. Compared with SFT, CUWorldEval's single-step description consistency is essentially unchanged (79.00 and 78.71), but in multi-turn replay, where the history is generated by the model itself, description consistency rises from 70.93 to 72.24, and the task-outcome BA improves on all three agents (Table 3). Meanwhile, continuing SFT with the same data and number of training epochs yields a BA 6.04–11.92 percentage points lower than that of CUWorldEval.
References:
[1] Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks
[2] Improving Multi-Step Prediction of Learned Time Series Models
[3] A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning