- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces Typed Spatial Markdown (TSM), a graphical user interface state representation that preserves hierarchical relationships, element types, locations, and control states to support decision-making with world-model foresight. The authors construct AtomBlock-WebUI by injecting predefined labels into HTML to reduce annotation errors from interpreting heterogeneous DOM structures. They propose MKD2World, a web world model trained in two stages to reconstruct screenshots in TSM and predict the next state given the current screenshot and an action. They also introduce an inference strategy in which a web agent uses predicted outcomes of candidate actions to extract task-relevant evidence and select an action. Evaluations cover annotation quality, state prediction, and downstream performance on Online-Mind2Web and VisualWebArena. MKD2World-9B achieves an out-of-domain EleF1 score of 26.50 and improves Qwen3.5-35B’s success rate from 22.94% to 37.61% on Online-Mind2Web.
Critical strengths
-
Using MKD2World’s next-state predictions in TSM format to provide task-relevant evidence for action selection is convincing. The framework improves downstream performance over the base agents in most evaluated settings on Online-Mind2Web and VisualWebArena (Tables 3 and 4), supporting its practical usefulness for web-agent decision-making. The ablation results also suggest a benefit from the margin-ranking objective.
-
Pre-injecting element labels into generated HTML and extracting bounding boxes directly from the corresponding DOM nodes is a useful approach for obtaining more consistent annotations. This reduces the ambiguity of inferring element types from noisy DOM structures. The annotation-quality results in Table 1 support the usefulness of this pipeline for constructing supervision for the UI detector used by the TSM renderer.
Critical weaknesses
- The additional benefit of TSM over existing structured representations is not clearly isolated.
The paper contains useful engineering contributions, particularly the injected-label annotation pipeline. However, predicting candidate-action outcomes for action selection and representing user interfaces through structured elements are already established. Table 5 examines training objectives, TSM-aware agent training, and the digest step, but does not vary the state representation. A controlled comparison using the same model backbone, two-stage training procedure, and downstream inference pipeline, but replacing TSM with an alternative representation such as MobileDreamer’s textual sketch or ScreenParse's ScreenTag, would help establish TSM’s specific contribution. Without such a comparison, the additional value attributable to TSM remains unclear.
- Annotation recall is not explicitly evaluated.
In Table 1, annotation quality is compared through measures including object validity, boundary alignment, and semantic correctness. However, the VLM judge and human audit evaluations do not explicitly measure false negatives, which is, visible UI elements within each dataset’s intended taxonomy that are missing from the annotations. Therefore, the reported results support the quality of the provided annotations but do not establish or allow a comparison of annotation completeness.
Questions for authors
-
Which Multimodal-Mind2Web splits were used as layout references for AtomBlock-WebUI? Since Online-Mind2Web includes original and adapted Mind2Web tasks, could you clarify what overlap checks were performed between the layout references, world-model and agent training data, and the 109 evaluation tasks? Otherwise there is a risk of information leak.
-
For the Base agents in Tables 3 and 4, was the unified prompt in Appendix I.4 used? If so, did you compare their performance with their recommended prompts and native action formats, where applicable, to check if the prompt you introduced downgraded the baselines?
-
Was annotation recall evaluated for AtomBlock-WebUI and the comparison datasets? Could an independent (e.g. human) audit estimate the proportion of visible elements missing from the annotations, within each dataset’s intended taxonomy?
-
How were the evaluation subsets selected? In particular, could you provide more explicit exclusion criteria or a breakdown of the Online-Mind2Web exclusions, and explain how the 233 VisualWebArena tasks were selected?
-
How frequently is the world model invoked during inference, and how does the proposed inference pipeline affect overall efficiency? Could you report the invocation rate and inference latency or computational cost compared with the base agent?
-
Do you have controlled results comparing TSM with an alternative structured representation under comparable backbone, data, two-stage training, agent adaptation, and inference-budget settings? If not, could a feasible, focused comparison or ablation help isolate TSM’s specific benefit from the other components of the framework?
Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns through the new comparisons with tree search and other world models, the inference-time cost analysis, and the TSM component ablations.
I believe these additional results and clarifications have adequately addressed my main concerns. Incorporating them into the revised manuscript would further strengthen the paper and provide a more comprehensive evaluation of the proposed method.
I have no further concerns at this point, and I appreciate the authors' thoughtful responses.