- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a method, named StateCraft, to simulate the GUI for a file control system via the world model. It equips the video diffusion model with a VLM to carry out reasoning about system states beyond the current visible frames, maintaining an abstract state using a state graph. To make VDM aware of the current state and generate the corresponding correct content, after reasoning, the retrieved file node from the updated state graph will be encoded into text descriptions and sent to the VDM. Conditioned on these state descriptions, as well as the previous neighboring frames, the VDM generates the current interfaces with correct visual states. The experiments prove its effectiveness.
Critical strengths
This paper addresses an interesting and important subproblem in the field of generative software: modeling of the GUI of the file controlling system. It is suitable for modeling and reasoning about the system control actions using the VLM, representing these elements via text descriptions, which could be conveniently digested by the VDM. Showing that the VDM and VLM could work in concert to tackle the generative software problem is also interesting and informative for the community. The problem formulation and contribution are both clear.
Critical weaknesses
-
As mentioned in the introduction, invisible actions change the states, and these changes may be visible in the future; the world model should be able to handle these cases. However, the memory experiments, which only contain revisits to the same file folder without content changes, seem to lack these cases. Such cases may be added: the user opens a folder, deletes one of the items, then exits the folder and accesses other folders, finally returns to the initial folder, or opens the Recycle Bin. The deleted item should not appear in the revisited folder and should appear in the Recycle Bin. These cases could help examine whether the proposed method could model the state changes beyond the visible frames.
-
The proposed method lacks generalizability to more complicated tasks and scenes beyond the file control system. Since the VDM relies on the text prompt to generate current frames, its performance may be limited when the state graph and node states become more complicated than what can be represented in text prompts. In other real cases, such as interaction with real-world objects and humans, the simple state graph and conditioning strategy may not work well.
Questions for authors
NA
2 comments
Thanks for the rebuttal. My concerns have been addressed. I will raise my score.
Discussion on Invisible State Changes, State Conditioning, and Generalization
We thank the reviewer for recognizing the importance of the problem, the clarity of our formulation, and the value of combining VLM-based state reasoning with video generation.
1. Revisits after deletion
We ran the suggested experiment on 143 benchmark trajectories. We select an item and press Delete, take 64 actions in other folders, revisit the source folder, and then open Trash for the first time. We supply no updated ground-truth frames or state for either location.
The learned update preserves the deletion on revisit and updates Trash while it is out of view. With and without the update, the source folder’s post-deletion layout score is 99.0% vs. 1.0%, with the remaining items shifting to fill the deleted item’s slot. On the first opening of Trash, the item appears in 93.7% vs. 18.9% of clips, and OCR similarity to its name is 0.28 vs. 0.02.
2. State conditioning and generalization
The video model receives state separately from the fixed text prompt. It uses frame-aligned cross-attention over per-element embeddings, so it can attend to each entity individually. The state embeddings have a separate input from the text prompt, which remains fixed, so the prompt length does not limit how many state embeddings the model receives.
The same conditioning interface supports different state schemas. In 3D, room-category and illumination tokens from the simulator condition the generation of unobserved rooms, yielding an illumination AUC of 0.864 on unseen houses vs. 0.542 without state conditioning (§4.7). The state predictor also uses a shared schema for Memory, Battleship, and Blackjack, where it must infer persistent state changes that are implicit in the action. It selects the correct transition in 83–86% of cases (Table 7).
The abstract state captures high-level semantics, while the visual state carries fine-grained appearance. For example, the abstract state can track which room is lit, while the visual state is better suited to representing how that room looks. StateCraft uses both, so the abstract state can guide generation and the visual state can supply the details of appearance.
We hope the deletion experiment and the clarification of our conditioning mechanism address your concerns. We would appreciate your taking them into account in your assessment and are happy to elaborate on either point.