- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes MOVI-Agent, a training-free GUI agent that constructs a persistent, queryable software model from instructional videos. The model comprises a function tree, a state-transition graph, and an action database, which support planning, navigation, and action retrieval. A multi-role execution framework incorporates visual grounding and semantic state backtracking to handle execution errors. The paper also introduces ProbeGUI, an annotation-driven benchmark containing 86 tasks across ten professional desktop applications, with evaluation at the step, subtask, task, and error-recovery levels. Experiments report improvements over existing baselines.
Critical strengths
- The paper addresses the difficulty of operating professional software, where agents often lack application-specific knowledge. Organizing instructional-video knowledge into a function tree, state graph, and action database provides a framework for connecting high-level planning with concrete GUI operations without fine-tuning.
Critical weaknesses
- Lack of evaluation on open-source established benchmarks. All experiments are conducted on the newly proposed ProbeGUI benchmark. It therefore remains unclear whether the gains generalize beyond its specific task construction and evaluation metric. Evaluation on an open-source benchmark, such as osworld, with comparable access to instructional demonstrations, would strengthen the evidence for the approach’s broader applicability.
- Limited evidence of generalization to new workflows. Section 4 reports that only 8.1% of demonstration–evaluation pairs differ in workflow structure. The evaluation therefore provides limited evidence that the learned software model can combine knowledge from different demonstrations to solve unseen workflows. Such an evaluation would more directly support the central claim of learning reusable software knowledge.
- The three-part representation needs stronger justification. Table 2 does not isolate the function tree, state graph, and action database, leaving unclear whether graph navigation adds value beyond action retrieval.
- Completion checks under state abstraction are unclear. Since state descriptions omit personalized inputs (§3.2), how does state matching (§3.3) distinguish correct completion from a similar screen containing incorrect task-specific values?
- The motivation needs a clearer visual illustration. Figure 1 explains the overall framework, but does not clearly illustrate why the proposed software model is needed. A motivating example showing where existing agents fail and how the function tree, state graph, and action database address those failures would make the design rationale more convincing.
Questions for authors
see weakness
2 comments
The graph matches the place, and the agent checks the task on the live screen (2/2)
3. Why a function tree, a state graph and an action database (W3): each answers a different question. The function tree selects which function to perform, and the action database retrieves how a similar action was performed. The state graph answers a third question: from this interface state, which transition leads toward the selected function, and what follows it? Similarity rank is not execution order. Of 426 decisions where the graph supplied a next operation, its recorded action text was absent from the three retrieved examples in 321. Even when both adjacent operations had unique full-text matches in retrieval, the later one ranked ahead of the current one in 14 of 39 cases. The graph thus supplies an ordered continuation that retrieval alone does not.
Table 2's "w/ Video Description" variant keeps all four MOVI roles and receives the same videos as procedural text, which can describe several steps at once. The full model leads it on every metric, by 4.7 points in task success (15.1% to 19.8%) and 8.3 points in error recovery (54.2% to 62.5%).
4. Completion under state abstraction (W4): the graph matches the place, and the agent checks the task. Section 3.2 represents what the interface lets a user do, not which personalized value happens to be in a field, so two property panels with different values map to the same node. At execution time, the Reflector sees the actual screenshot and the task instruction; it checks for visible contradictions and can ask the Executor to correct one instead of following the next graph step. Reaching a reference node is not itself a completion decision.
In Logisim-4, the instruction required a tunnel labeled in1 instead of a standard wire. The circuit editor was the right interface, but the screenshot showed a wire connecting IN1 directly to the AND gate. The Reflector flagged that violation and requested removal, and the Executor issued Ctrl+Z, the accepted recovery action. The same check applies to a numeric value: in Logisim-1, the Reflector noticed a gate width of 2 against one-bit connections and the visible "Incompatible widths" warning, and the agent changed Data Bits to 1. In both cases the check used concrete screen content that the reusable graph node intentionally leaves out.
5. Motivating example (W5): the screen shows which menus exist, but often not where a function lives. In Zettlr, both direct GPT-6.1-sol and OS-Symphony looked under Edit to create a snippet. MOVI linked the requested function to Assets Manager, then Snippets, and issued the accepted shortcut for opening the manager. The software model supplies that application-specific link between a goal and its entry point, while the screenshot tells the agent how to act here and now. As you suggest, the revised version will add this Zettlr case to Figure 1 as a motivating panel.
The state graph supplies the next step that action retrieval misses (1/2)
Thank you for the concrete questions, especially on completion checks under state abstraction. An overview of the five answers:
1. Broader applicability (W1): the gains replicate on a second backend and in a second harness. On ProbeGUI, MOVI leads on GPT-6.1-sol (44 tasks vs. 36 for direct prediction and 30 for OS-Symphony), and OS-Symphony itself completes 10 more tasks (+11.63 pp) when it queries the software model. For the OSWorld comparison with comparable access to demonstrations that you suggest, tutorials must first be assembled for its applications and given to every method: its applications (office suite, browser, mail, media, image and code editors) do not overlap ProbeGUI's ten, and its tasks come without tutorials. ProbeGUI complements OSWorld's end-state checks by pairing each task with a tutorial and scoring intermediate actions, so it shows where an agent first went wrong.
2. New workflows (W2): MOVI completes five more subtasks before its first failure than either baseline. On the seven tasks labeled as workflow changes in Section 4, GPT-5.4 MOVI made 87/118 correct decisions on observations shared by both systems, versus 76/118 without video (+9.32 pp). With GPT-6.1-sol, 116 observations were shared by all three systems on those seven tasks:
We also traced retrieved examples back to their tutorial videos across all 86 GPT-6.1-sol tasks. In 11 tasks, examples from different videos reached different decision steps and matched the actions accepted at those steps, and 8 of those tasks succeeded: a single execution drew on more than one demonstration.