acceptodds
Under review as a conference paper at ICLR 2027

StateCraft: Augmenting Video World Models with Abstract-State Reasoning

20 views

Abstract

Video world models aim to simulate complex environments with realistic appearance and dynamics. Maintaining consistency over long interactions requires tracking past actions and understanding how those actions affect the environment. Video world models can only generate partial observations of the entire world at a time. Remembering past observations alone may not be sufficient in every scenario, as they may no longer reflect the world after a sequence of actions. We therefore introduce a paradigm that decouples a persistent, updatable abstract state from the visual state. The abstract state keeps track of the entities changes as the world evolves, and is used to condition the visual state, which represents the current view. We propose StateCraft, a novel architecture that pairs a language model with a video diffusion model to evolve the dynamic world. We evaluate our approach on generative interfaces, which pose a substantial challenge because a single action may change parts of the state that are not currently visible. To support training and evaluation, we develop a data collection pipeline that produces informative, densely annotated trajectories at scale without human or agent supervision, together with a benchmark measuring generation quality, memory, and controllability. On this benchmark, our approach improves visual memory metrics by 0.30 (SSIM) and PSNR by up to 10 dB over the state-of-the-art baselines. We demonstrate that StateCraft maintains high self-consistency and strong generation quality over long trajectories. Because the abstract state is explicit, it can be edited directly, and generation follows edits to states never observed in the data. Finally, a proof-of-concept 3D experiment, in which actions change the illumination of an unobserved room, shows that the paradigm can also be applied to spatial environments. Dataset, code, and model weights will be made public.

Primary area:
generative models
Keywords:
Video World Models, Memory, Diffusion Models, LLM

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_yB1vOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a method, named StateCraft, to simulate the GUI for a file control system via the world model. It equips the video diffusion model with a VLM to carry out reasoning about system states beyond the current visible frames, maintaining an abstract state using a state graph. To make VDM aware of the current state and generate the corresponding correct content, after reasoning, the retrieved file node from the updated state graph will be encoded into text descriptions and sent to the VDM. Conditioned on these state descriptions, as well as the previous neighboring frames, the VDM generates the current interfaces with correct visual states. The experiments prove its effectiveness.

Critical strengths

This paper addresses an interesting and important subproblem in the field of generative software: modeling of the GUI of the file controlling system. It is suitable for modeling and reasoning about the system control actions using the VLM, representing these elements via text descriptions, which could be conveniently digested by the VDM. Showing that the VDM and VLM could work in concert to tackle the generative software problem is also interesting and informative for the community. The problem formulation and contribution are both clear.

Critical weaknesses
  1. As mentioned in the introduction, invisible actions change the states, and these changes may be visible in the future; the world model should be able to handle these cases. However, the memory experiments, which only contain revisits to the same file folder without content changes, seem to lack these cases. Such cases may be added: the user opens a folder, deletes one of the items, then exits the folder and accesses other folders, finally returns to the initial folder, or opens the Recycle Bin. The deleted item should not appear in the revisited folder and should appear in the Recycle Bin. These cases could help examine whether the proposed method could model the state changes beyond the visible frames.

  2. The proposed method lacks generalizability to more complicated tasks and scenes beyond the file control system. Since the VDM relies on the text prompt to generate current frames, its performance may be limited when the state graph and node states become more complicated than what can be represented in text prompts. In other real cases, such as interaction with real-world objects and humans, the simple state graph and conditioning strategy may not work well.

Questions for authors

NA

2 comments
AuthorsOpenReview ↗

Discussion on Invisible State Changes, State Conditioning, and Generalization

We thank the reviewer for recognizing the importance of the problem, the clarity of our formulation, and the value of combining VLM-based state reasoning with video generation.

1. Revisits after deletion

We ran the suggested experiment on 143 benchmark trajectories. We select an item and press Delete, take 64 actions in other folders, revisit the source folder, and then open Trash for the first time. We supply no updated ground-truth frames or state for either location.

The learned update preserves the deletion on revisit and updates Trash while it is out of view. With and without the update, the source folder’s post-deletion layout score is 99.0% vs. 1.0%, with the remaining items shifting to fill the deleted item’s slot. On the first opening of Trash, the item appears in 93.7% vs. 18.9% of clips, and OCR similarity to its name is 0.28 vs. 0.02.

2. State conditioning and generalization

The video model receives state separately from the fixed text prompt. It uses frame-aligned cross-attention over per-element embeddings, so it can attend to each entity individually. The state embeddings have a separate input from the text prompt, which remains fixed, so the prompt length does not limit how many state embeddings the model receives.

The same conditioning interface supports different state schemas. In 3D, room-category and illumination tokens from the simulator condition the generation of unobserved rooms, yielding an illumination AUC of 0.864 on unseen houses vs. 0.542 without state conditioning (§4.7). The state predictor also uses a shared schema for Memory, Battleship, and Blackjack, where it must infer persistent state changes that are implicit in the action. It selects the correct transition in 83–86% of cases (Table 7).

The abstract state captures high-level semantics, while the visual state carries fine-grained appearance. For example, the abstract state can track which room is lit, while the visual state is better suited to representing how that room looks. StateCraft uses both, so the abstract state can guide generation and the visual state can supply the details of appearance.

We hope the deletion experiment and the clarification of our conditioning mechanism address your concerns. We would appreciate your taking them into account in your assessment and are happy to elaborate on either point.

Thanks for the rebuttal. My concerns have been addressed. I will raise my score.

Reviewer_UkiqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

StateCraft splits a video world model into two parts: an explicit abstract state and a visual renderer. A VLM predicts how an action changes the state without looking at pixels. A video diffusion model then renders the next frame conditioned on the predicted state. The method is built for a GUI file manager, with an accessibility-tree-based data collector and a new benchmark covering reconstruction, revisit memory and state edits. It beats NeuralOS and the unconditioned backbones by a large margin and follows direct state edits 81% of the time.

Critical strengths
  • The decomposition is well motivated, and a GUI is a good test case because pixel-only memory clearly fails there. The benchmark is carefully built, and the predicted state performs close to the ground-truth state.
  • Being able to edit the state directly is a good property, and the edit experiments show that it works.
Critical weaknesses
  • The main motivation is not fully supported by the experiments. The introduction emphasizes tracking changes that happen outside the current view, such as deleted files appearing in Trash. However, the memory benchmark keeps folder contents fixed, as mentioned in the appendix. The hidden-change experiment also uses manual state edits rather than changes caused by actual user actions. As a result, the experiments seem to test whether the model can remember and retrieve information, rather than whether it can track how the environment changes over time.

  • It is unclear how much of the improvement comes from the learned model. StateCraft uses structured information from the accessibility tree, including folder paths and element names, which already provides much of the information needed to generate the next screen. In Table 1, using ground-truth state gives almost the same results, suggesting that most of the improvement comes from having this structured state. The key question is then how well the VLM can learn state transitions, but Table 7 only reports in-domain accuracy on held-out navigation. Also, the method requires a manually defined state schema and is only fully tested on one app. The 3D example uses ground-truth state rather than predicted state, so it is still unclear how well the approach would generalize to other environments.

  • Several figures are hard to read. Figures 2, 3 and 4 are too small, and the text and frames inside them are hard to make out at normal zoom. I had trouble following the overview in Fig. 2 in particular, which made the method harder to understand.

Questions for authors
  1. The collected data does include filesystem mutations (Appendix A.2). Can you evaluate revisits after an action changes a folder while it is out of view?
  2. How does quality degrade over long rollouts as state-prediction errors accumulate?
  3. What is the in-domain state-prediction accuracy on non-navigation actions, such as rename, delete or move?
3 comments
AuthorsOpenReview ↗

Additional Details on State Changes, Long-Rollout Quality, and State-Prediction Accuracy

We thank the reviewer for recognizing the motivation for separating state prediction from rendering, the care taken in building the benchmark, and the effectiveness of direct state editing.

1. Can the model track changes caused by actions?

The new deletion test uses a Delete action and a learned state update. In 143 clips, we select item XX and press Delete, take 64 actions in other folders, then revisit the source folder and open Trash. We hold the renderer fixed and supply no updated ground-truth frames or state. The source revisit tests whether the model retains the deletion. Trash is opened for the first time in every clip, so it tests whether the model renders a change to a destination that was out of view when the action occurred.

With and without the state update, the post-deletion layout score is 99.0% vs. 1.0%, SSIM for repositioned items is 0.81 vs. 0.56, and OCR similarity to the deleted name at the source is 0.20 vs. 0.71 (lower is better). The item appears in Trash in 93.7% vs. 18.9% of clips.

The manual edits in Table 3 test whether the renderer follows a supplied state change. In 45 hidden-retype clips, the generated view follows the edit on re-entry in 98% of cases, with a Follow margin of 0.236 (95% CI [0.200,0.274][0.200,0.274]). With vs. without the edit, SSIM to the requested appearance is 0.91 vs. 0.36, OCR similarity to the requested name is 0.73 vs. 0.12, and similarity to the old name is 0.10 vs. 0.74. Control SSIM is similar (0.87 vs. 0.84).

2. How does visual quality change over long rollouts?

We render the same 184 rollouts (about 544 frames each) with CF++, using either closed-loop VLM predictions or ground-truth state. Using predicted state has little effect on visual quality over the tested horizons. Both SSIM and PSNR are close when we predict the state and when we use the GT state, thus we can conclude that the VLM prediction doesn’t degrade the quality of the world model. The visual quality remains high, and it can be improved over long rollouts by applying established distillation techniques, which we don’t incorporate in this work.

FramesVLM state accuracySSIM pred.SSIM GTPSNR pred.PSNR GT
8–640.9640.9480.94027.2227.14
64–1280.9310.9300.92425.2525.22
128–1920.9040.9110.91323.8224.72
192–2560.9090.9010.90523.2324.29
256–3200.9030.8950.89823.2024.04
320–3840.9070.8880.89022.8423.59
384–4480.9040.8830.88522.7823.46
448–5120.9050.8770.87822.4623.07
512–5520.8880.8730.87522.6323.07

3. How accurate is state prediction for non-navigation actions?

On held-out trajectories (41,010 free-running steps), the predicted path is correct for 99.5% of window closes, 91.4% of context-menu actions, and 93.2% of non-navigation clicks (93.7% overall). Element-name F1 is 0.87 and box IoU is 0.98.

For file mutations, we run 284 deletions and renames on the folders in our memory benchmark and score the complete predicted folder state, including the new layout after deletion. The in-domain fine-tuned predictor recovers the exact state in every tested deletion and rename, zero-shot and with no regression in navigation accuracy on the validation set. Fine-tuning substantially improves accuracy over prompting the same base model with few-shot prompting.

State predictorDelete, right itemDelete, exact stateRename, right itemRename, exact state
Qwen3.5-0.8B base, few-shot4/1410/14187/13976/139
Qwen3.5-0.8B fine-tuned, zero-shot143/143143/143141/141141/141

Our end-to-end evaluation uses one application with a manually defined schema. The 3D experiment tests rendering with simulator state, and Table 7 separately evaluates state prediction in Memory, Battleship, and Blackjack (83–86% transition accuracy). These experiments test individual components beyond the file manager.

4. Figure readability

We will enlarge Figures 2–4 and increase the size of the labels and example frames. In Fig. 2, we will simplify the layout and label the arrows so the flow from action and retrieval to state prediction and rendering is readable at normal zoom.

We hope the new deletion and state-prediction results answer your questions. If they do, we would welcome a reconsideration of your score and are happy to discuss any remaining concerns.

I thank the authors for the rebuttal. Some of my concerns remain.

  • W1/Q1: Only deletion is tested, and the learned update performs about the same as a manual cache update. This suggests the gain comes mainly from the structured state rather than from learned state transitions.
  • W2: Generalization beyond one app with a manually defined schema is still untested.
  • W3: The figures are only promised to be improved.

I therefore maintain my score.

Thank you for the follow-up.

The file-operation tests cover both deletion and renaming. Matching the manual update shows that the predictor can maintain this state automatically, without manually supplied updates. This supports our main contribution to improve world-model consistency through persistent structured state and learned updates.

We demonstrate our contributions by modeling a fully functioning file explorer in latent space, previously limited in the state of the art. We also complement with the use of structured state in simulated 3D environments and in three different games. Together, these contributions provide a foundation for larger-scale applications requiring persistent state and consistent visual generation.

The figure improvements are already included in the revised manuscript. We enlarged the labels, figures and examples to make the method easier to follow.

We thank you for the discussion, we are happy to answer any further question.

Reviewer_4XiWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

StateCraft maintains an explicit, structured environment state, uses a finetuned VLM to predict action-induced state updates, and conditions a video world model on the updated state. In file-manager simulation, the paper reports improved generation quality and long-horizon revisit consistency across DFoT and CF++ backbones. It also evaluates responsiveness to state edits and demonstrates state-conditioned generation in a simulated-room illumination experiment.

Critical strengths
  • Separating persistent state tracking from visual generation provides a reasonable mechanism for preserving entities and their properties across interactions, including changes outside the current view.
  • The experiments cover overall and region-specific image quality, revisit consistency, and controllability across two video backbones. State-predictor comparisons and conditioning ablations provide evidence for the contributions of both components.
  • Editing the supplied state tests whether generated frames follow that state despite conflicting visual history. Evaluating both edited and unedited regions also reveals limitations in preserving unaffected content.
Critical weaknesses
  • The method description isn't well formulated and contains ambiguities. The relationship between retrieved context nodes and the inputs depicted in Figure 2 is unclear. Although the text describes destination-node retrieval, it does not fully explain the selection of the context-node set. Likewise, “graph tokens” are introduced without explicitly connecting them to the previously described component embeddings. Equation (1) conditions frame generation on ata_t, whereas Figure 2 describes conditioning on an action history. The role of MtM_t, which appears as conditioning on the right-hand side but not explicitly on the left, also warrants clarification.
  • ROI evaluation details are insufficiently specified. The paper describes using recorded element bounds and OCR-localized text boxes, but does not clearly state where OCR is applied or how regions are matched between reference and generated frames. It is therefore unclear how displaced, resized, or missing elements are handled. Without this information or complementary localization measures, the reported scores are difficult to interpret in terms of content fidelity versus spatial alignment.
  • The controllability metrics provide an incomplete picture. Follow indicates whether a generated result is closer to the edited target than to the original target, but does not quantify how much closer it is. E-PSNR provides complementary fidelity measurements, yet the paper does not report the continuous distance margin underlying Follow. Finally, the omission of E-SSIM and C-SSIM is unexplained, despite the use of SSIM elsewhere.
Questions for authors
  • Please address the ambiguities, namely, how are context nodes selected and supplied to the VLM, and how should the retrieval path in Figure 2 be interpreted? Are “graph tokens” the component embeddings described in §3.2.3? Please also clarify the action-history conditioning and the relationship between MtM_t and the history in Equation (1).
  • Are OCR boxes obtained from reference frames, generated frames, or both? How are corresponding regions matched when generated elements move, change size, or disappear? Could localization or overlap measurements help separate spatial errors from content errors?
  • Can you report the magnitude of the distance preference underlying Follow, such as (dold−dnew)/dold(d_{\mathrm{old}}-d_{\mathrm{new}})/d_{\mathrm{old}} for each edit category, or E-SSIM and C-SSIM to determine whether they support the conclusions drawn from PSNR?
4 comments
AuthorsOpenReview ↗

Response: Graph Retrieval, ROI Evaluation, and Edit Fidelity

We thank the reviewer for recognizing the value of separating persistent state tracking from video generation and for highlighting the evaluation across both backbones.

1. Graph retrieval and history conditioning

In Eq. (1), oto_t is the currently focused node, and MtM_t is the persistent graph that summarizes the interaction history on the left-hand side. The VLM receives oto_t, the current action ata_t, and any context retrieved from MtM_t. It predicts the next node ot+1o_{t+1} and the update Δt\Delta_t, which we apply as Mt+1=Mt⊕ΔtM_{t+1}=M_t\oplus\Delta_t.

For a click, the VLM receives at most one retrieved node. We check whether the click falls inside a component box of oto_t. If it does, we combine the current path with the component’s relative path and look up that address in MtM_t. The context contains the stored destination node if found and is empty otherwise. We will update the context label in Fig. 2 to show both cases.

The graph tokens are the component embeddings in §3.2.3. We serialize the predicted next node ot+1o_{t+1} into at most 32 strings, consisting of the path and one kind name @box string per component. Frozen Qwen3-Embedding-8B embeds each string independently. We truncate each embedding to 1,024 dimensions and use the resulting tokens to condition the corresponding frame through cross-attention.

The video model uses the action history shown in Fig. 2. Eq. (1) currently lists only ata_t in the video factor. In practice, the model uses recent frames from x≤tx_{\leq t} and their aligned action embeddings. Action ata_t leads to frame xt+1x_{t+1}, which is conditioned on the predicted node ot+1o_{t+1}. DFoT keeps the eight most recent frames, and CF++ caches up to 21 previous frames. We will revise Eq. (1) to show the action history explicitly.

2. ROI matching and OCR

The original evaluation uses the same crop coordinates for the reference and generated frames. For region scores, we use element bounds recorded with the reference frame. For text scores, we detect OCR boxes on the reference frame. In both cases, we crop the same rectangle from both frames, so displaced, resized, or missing elements can lower the score alongside content errors.

To separate localization from text accuracy, we also detect text boxes on generated frames. Within each frame and screen zone, we match generated and reference boxes one-to-one using the Hungarian algorithm to maximize total IoU. After matching, a pair counts as localized if IoU≥0.5\mathrm{IoU}\geq 0.5. Unmatched reference boxes and pairs below the threshold count as misses. We compute character error rate (CER) only on localized pairs.

StateCraft localizes more reference text boxes and achieves lower CER on the boxes it localizes.

Canvas metricStateCraftNeuralOSCF++
Localization recall ↑87%47%9%
CER on localized pairs ↓0.180.640.82

3. Follow margin and SSIM

The continuous Follow margin shows how much closer the output is to the requested edit. We report dold−dnewd_{\mathrm{old}}-d_{\mathrm{new}}, where each distance is the mean absolute difference within the edit mask on a [0,1][0,1] scale. The mean margin is 0.147 (95% bootstrap CI [0.125,0.170][0.125,0.170]). The margin is significantly positive in six of eight categories and largest for hidden changes on re-entry (0.236).

SSIM improves in the edited region and stays similar in the control region. E-SSIM to the requested appearance rises from 0.47 without the edit to 0.80 with it. C-SSIM is 0.805 with the edit and 0.801 without it. We will report both alongside PSNR.

We hope these clarifications and measurements resolve the issues you raised. We welcome follow-up questions and would appreciate your considering an updated score in light of these results.

Thank you for addressing all of my questions and providing additional experiments. The clarifications on graph retrieval and history conditioning are very clear. I look forward to the updated manuscript incorporating these explanations and resolving the ambiguities in the original presentation.

The continuous Follow margin is a useful addition. But I am curious about its distribution: the 95% bootstrap confidence interval for the mean has a positive lower bound of 0.125, while the binary Follow rate of 0.81 indicates that approximately 19% of cases have nonpositive margins. Although these statistics are mathematically compatible, could you provide more details, such as some margin percentiles, to help characterize this variation?

Finally, could you explain the different behavior of C-SSIM and C-PSNR? C-SSIM remains nearly unchanged with and without editing (0.805 versus 0.801), whereas Table 3 reports a substantial C-PSNR decrease with editing, from 37.7 to 23.88 dB. Do you have any qualitative analysis that helps explain this difference and what it implies for preserving unedited content?

AuthorsOpenReview ↗

Follow Margin and Control-Region Fidelity

We thank the reviewer for the careful analysis of the results and for the follow-up questions.

Applying the edit reverses which target is closer at the median. Across the 321 edits in Table 3, the median Follow margin dold−dnewd_{\mathrm{old}}-d_{\mathrm{new}} shifts from −0.154-0.154 without the edit to 0.1390.139 with it, moving from a preference for the original target to a preference for the requested edit.

PercentileWith editWithout edit
10th−0.107−0.449
25th0.057−0.319
50th0.139−0.154
75th0.312−0.102
90th0.4410.003

Control-region metrics. In our previous reply, we reported two C-SSIM values from separate runs of the editing protocol. We apologize for this reporting mistake. The standard unedited baseline in Table 3 yields a C-SSIM of 0.980 and a C-PSNR of 37.73 dB, which does not affect any of our conclusions. The original protocol improves edit fidelity at a cost to control-region fidelity, as both metrics show when compared with the correct baseline.

The reviewer’s question also prompted us to test a way to better preserve unedited content. The original evaluation focused on our main contribution, editing through explicit state updates. We propose complementing our approach with standard diffusion inpainting, a well-established technique for local image editing. The state update identifies the regions that need to change, so we can use them as the inpainting mask and preserve the unedited render elsewhere. We evaluated this approach on the same 321 edits.

SettingFollow (no-edit control)E-PSNR (dB)C-PSNR (dB)C-SSIM
Table 3 protocol0.81 (0.11)27.623.880.805
Edit-region inpainting0.85 (0.08)29.537.700.980
Standard unedited generation—17.637.730.980

Inpainting brings control-region fidelity back to the level of standard unedited generation while improving Follow and edited-region PSNR. The mean Follow margin rises from 0.147 to 0.162 (95% bootstrap CI [0.141,0.184][0.141,0.184]). E-SSIM is 0.855 with the edit and 0.477 in the no-edit control.

We will summarize this experiment in the main text and include the inpainting details and results in the supplement. We hope these results address your remaining concerns and would appreciate your reconsidering your score in light of them.

Thanks for the detailed response and supplementary information. I think my concerns are addressed and I will update the score.

Reviewer_RZWeOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

StateCraft augments video world models with a persistent, structured state. A language model predicts state updates from user actions, and a video diffusion model generates observations conditioned on the resulting state components. The paper introduces an instrumented data collection pipeline for a file manager and evaluates reconstruction, directory revisits, and explicit state edits. Experiments with DFoT and CF++ show substantial improvements in content reconstruction and retention under the specified initialization protocol. The paper also studies a gate for reducing language-model calls and a 3D experiment in which simulator-provided state controls the illumination of an unseen room.

Critical strengths
  1. Maintaining explicit state is a sensible approach to the consistency problems in generative interfaces, and the reconstruction gains are substantial. In Table 1, adding StateCraft to CF++ improves PSNR from 18.49 to 28.78 dB. More importantly, the improvement extends to variable interface content: canvas text-region SSIM increases from 0.16 to 0.85 in Table 2. The consistent gains with DFoT provide useful evidence that the conditioning approach works across the two backbones. These results make a convincing case for preserving structured content outside the video model’s recent visual context.

  2. The revisit evaluation is carefully controlled. The 184 trajectory families keep the target content and corresponding observations fixed while varying the interval before returning. Figure 10 additionally measures reconstruction against ground truth, addressing an important limitation of self-consistency alone. StateCraft maintains approximately 0.92–0.94 ROI SSIM across the tested horizons. Together, these experiments provide good evidence for long-term retention of previously observed content.

Critical weaknesses
  1. My main concern is that the experiments do not directly establish the ability that motivates the paper: learning how actions change unseen state and maintaining those changes until a later visit. The memory benchmark keeps target content unchanged, as explicitly acknowledged in Appendix A.3.1. The editing experiment supplies the modified conditioning state directly, while the 3D experiment obtains state updates from the simulator. These are useful tests of retention and conditional rendering, but they leave the learned update mechanism largely untested on the specific problem used to motivate it. The ablations do not resolve this: Table 8 uses ground-truth states, and Table 9 removes conditioning channels from a trained model. Neither isolates the benefit of learned state updates over a structured cache with simple navigation updates. I would need more direct evidence here to support the paper’s claim about abstract-state reasoning.

  2. The editing results leave the reliability of control unclear. The 81% follow rate in Table 3 measures whether an output is closer to the edited image than to the original image within the change mask. It does not establish that the requested filename, object type, or object count is correct. Meanwhile, PSNR in the unedited control region drops from 37.7 to 23.88 dB, and insertion reaches only 48% follow. The authors acknowledge these limitations, but the amount of collateral change matters for a system intended to maintain a consistent environment. A semantic check of the requested edit, together with preservation of unaffected objects, would make the controllability result much easier to assess.

Questions for authors
  1. Targeted validation: Can the existing evaluation be broken down to show actions that change the contents of an already observed but currently unseen directory, followed by a later revisit? Moving files between directories or deleting files and revisiting Trash would be relevant examples. Please report both state and image correctness without supplying the updated ground-truth state. A comparison between the learned updater and a simple structured-cache update, with the renderer and initialization held fixed, would directly address my main concern.

  2. Editing evaluation: What are the sample counts for the edit categories in Table 3? Among the cases counted as following the edit, how often are the requested names, types, and counts correct while unaffected objects remain unchanged? Does the drop in control-region PSNR mainly reflect appearance changes, or does it include changes to object content?

  3. Baseline clarification: Section 4 says NeuralOS follows its public training schedule and iteration count, whereas Appendix A.1.2 describes 110k updates with a 40k/30k/40k stage allocation. The original NeuralOS training procedure includes scheduled sampling and context-length extension. How were these stages incorporated into the implementation used here, and was the model still improving at the end of training? This clarification would affect my interpretation of the external baseline’s long-horizon performance.

1 comment
AuthorsOpenReview ↗

Clarifications: Learned State Updates, Edit Fidelity, and NeuralOS Training

We thank the reviewer for highlighting the reconstruction gains across both backbones and the evidence for long-term retention from our controlled revisit benchmark.

1. Learned state updates after deletion

We tested deletion on 143 clips (N=64N=64). We delete item XX during the first folder visit, take 64 actions in other folders, then revisit the source folder and open Trash. Trash is opened for the first time in every clip. The source revisit tests whether the model retains the deletion. The first opening of Trash tests whether the model can render an update to a destination that was out of view when the action occurred. We provide no updated ground-truth frames or state for either location.

We fix the renderer, checkpoint, and eight initial frames across all three conditions. The learned predictor recovers the exact post-deletion source state, including the new layout, in 143/143 clips. Its state sequence matches the manually updated state in 140/143 clips.

Image measureLearned updateManual cache updateNo update
Post-deletion layout score on revisit99.0%98.2%1.0%
OCR similarity to the deleted name ↓0.200.200.71
Deleted item appears in Trash93.7%94.4%18.9%

The learned update performs about as well as the manual cache update on deletion. Both outperform the no-update control at the source and in Trash.

2. Edit fidelity and effects on unaffected regions

Table 3 covers 293 clips and 321 edits. We will add the following category counts to the table.

Edit categoryClips
Rename, ground-truth / generated context45 / 36
Retype, ground-truth / generated context61 / 36
Retype of a directory entered one frame earlier18
Hidden retype on re-entry45
Multi-retype25 (53 edits)
Insertion27

Among edits counted as followed, OCR similarity (1−CER1-\mathrm{CER}) is 0.65 to the requested name and 0.14 to the original name. OCR never reads the full original name in an edited region. For retypes, SSIM to the requested icon is 0.85 with the edit and 0.34 without it.

The lower control-region PSNR is consistent with differences in appearance. PSNR penalizes pixel differences even when the text stays the same. Control-region SSIM is similar with and without the edit (0.805 vs. 0.801), and OCR similarity is close to the baseline (0.37 vs. 0.38). These results are consistent with differences in rendering appearance. Control-region SSIM changes little when the edit is applied, and text fidelity is comparable to the baseline.

3. NeuralOS training

We used the public NeuralOS implementation and its three-stage structure, matching the total number of updates to our models. NeuralOS ran for 110k updates (40k/30k/40k). In the final stage, we used the released settings for scheduled sampling (S=0.05S=0.05) and context extension to 64 frames. Training loss was flat over the final 10k updates. We will clarify the schedule in Section 4.

If these results address your concerns, we would appreciate your reconsidering the score. We are happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank the reviewers for their useful feedback which helped us further strengthen the conclusions of our paper.

Changes outside the current view (Reviewers RZWe, Ukiq, yB1v). The learned model updates locations outside the current view and retains changes on revisits. Its deletion updates closely match manually specified updates.

Reliability of state prediction (Reviewers RZWe, Ukiq). Fine-tuning improves deletion and rename accuracy. Over long sequences, predicted state yields image quality close to that obtained with correct state. We clarified how NeuralOS was trained and that its loss had plateaued by the end of training.

Edit accuracy and preservation (Reviewers RZWe, 4XiW). We reported sample counts and added checks of requested names, types, and how consistently outputs follow edits. Complementing our approach with established inpainting techniques improves edit accuracy while preserving surrounding content.

Method and evaluation clarity (Reviewers 4XiW). We clarified how stored information and past actions guide generation. We separated text placement from text accuracy so misplaced elements and incorrect text can be assessed independently.

Generalization and figures (Reviewers Ukiq, yB1v). Tests in games assess state prediction, while tests in simulated rooms assess rendering with supplied state. We clarified the scope of these results and will enlarge figure labels and examples.

We have addressed all concerns raised in the reviews and follow-up discussion and updated the manuscript. We are happy to address any remaining questions and continue the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.