acceptodds
Under review as a conference paper at ICLR 2027

MKD2World: A Typed Spatial Markdown World Model for Web Agents

Abstract

Graphical user interface (GUI) agents can improve decision-making with world-model foresight, provided that predicted states accurately reflect post-action interfaces. Existing state representations fail to jointly capture interface layout, UI semantics, and explicit locations. We introduce Typed Spatial Markdown (TSM), a Markdown-style textual state representation for desktop web interfaces. TSM serializes block-level and atomic-level UI elements according to their spatial relationships, jointly encoding their types, visual semantics, explicit locations, optional control states, and hierarchical membership. To train a screenshot-to-TSM renderer, we construct AtomBlock-WebUI, a dataset with precise annotations at both granularities. By injecting element-type labels into HTML and directly extracting bounding boxes after rendering, we avoid annotation errors caused by post-hoc inference from webpage structures. Building on TSM, we propose MKD2World, a desktop web world model trained in two stages for current-state description and action-conditioned next-state prediction. During inference, the web agent queries MKD2World as needed and performs a "digest" step, in which it interprets each predicted state and extracts task-relevant evidence for selecting the best action. Experiments show that MKD2World-9B achieves an EleF1 score of 26.50 on the out-of-domain test set, outperforming GPT-5.6-Sol(14.98) by 11.52 points. Across Online-Mind2Web tasks, MKD2World-9B improves the success rate of the Qwen3.5-35B web agent from 22.94% to 37.61%, achieving competitive performance among leading open-source GUI world models.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Agentic System, Web Modelling, Cross-modal Learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Mw4mOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Typed Spatial Markdown (TSM), a graphical user interface state representation that preserves hierarchical relationships, element types, locations, and control states to support decision-making with world-model foresight. The authors construct AtomBlock-WebUI by injecting predefined labels into HTML to reduce annotation errors from interpreting heterogeneous DOM structures. They propose MKD2World, a web world model trained in two stages to reconstruct screenshots in TSM and predict the next state given the current screenshot and an action. They also introduce an inference strategy in which a web agent uses predicted outcomes of candidate actions to extract task-relevant evidence and select an action. Evaluations cover annotation quality, state prediction, and downstream performance on Online-Mind2Web and VisualWebArena. MKD2World-9B achieves an out-of-domain EleF1 score of 26.50 and improves Qwen3.5-35B’s success rate from 22.94% to 37.61% on Online-Mind2Web.

Critical strengths
  1. Using MKD2World’s next-state predictions in TSM format to provide task-relevant evidence for action selection is convincing. The framework improves downstream performance over the base agents in most evaluated settings on Online-Mind2Web and VisualWebArena (Tables 3 and 4), supporting its practical usefulness for web-agent decision-making. The ablation results also suggest a benefit from the margin-ranking objective.

  2. Pre-injecting element labels into generated HTML and extracting bounding boxes directly from the corresponding DOM nodes is a useful approach for obtaining more consistent annotations. This reduces the ambiguity of inferring element types from noisy DOM structures. The annotation-quality results in Table 1 support the usefulness of this pipeline for constructing supervision for the UI detector used by the TSM renderer.

Critical weaknesses
  1. The additional benefit of TSM over existing structured representations is not clearly isolated.

The paper contains useful engineering contributions, particularly the injected-label annotation pipeline. However, predicting candidate-action outcomes for action selection and representing user interfaces through structured elements are already established. Table 5 examines training objectives, TSM-aware agent training, and the digest step, but does not vary the state representation. A controlled comparison using the same model backbone, two-stage training procedure, and downstream inference pipeline, but replacing TSM with an alternative representation such as MobileDreamer’s textual sketch or ScreenParse's ScreenTag, would help establish TSM’s specific contribution. Without such a comparison, the additional value attributable to TSM remains unclear.

  1. Annotation recall is not explicitly evaluated.

In Table 1, annotation quality is compared through measures including object validity, boundary alignment, and semantic correctness. However, the VLM judge and human audit evaluations do not explicitly measure false negatives, which is, visible UI elements within each dataset’s intended taxonomy that are missing from the annotations. Therefore, the reported results support the quality of the provided annotations but do not establish or allow a comparison of annotation completeness.

Questions for authors
  1. Which Multimodal-Mind2Web splits were used as layout references for AtomBlock-WebUI? Since Online-Mind2Web includes original and adapted Mind2Web tasks, could you clarify what overlap checks were performed between the layout references, world-model and agent training data, and the 109 evaluation tasks? Otherwise there is a risk of information leak.

  2. For the Base agents in Tables 3 and 4, was the unified prompt in Appendix I.4 used? If so, did you compare their performance with their recommended prompts and native action formats, where applicable, to check if the prompt you introduced downgraded the baselines?

  3. Was annotation recall evaluated for AtomBlock-WebUI and the comparison datasets? Could an independent (e.g. human) audit estimate the proportion of visible elements missing from the annotations, within each dataset’s intended taxonomy?

  4. How were the evaluation subsets selected? In particular, could you provide more explicit exclusion criteria or a breakdown of the Online-Mind2Web exclusions, and explain how the 233 VisualWebArena tasks were selected?

  5. How frequently is the world model invoked during inference, and how does the proposed inference pipeline affect overall efficiency? Could you report the invocation rate and inference latency or computational cost compared with the base agent?

  6. Do you have controlled results comparing TSM with an alternative structured representation under comparable backbone, data, two-stage training, agent adaptation, and inference-budget settings? If not, could a feasible, focused comparison or ablation help isolate TSM’s specific benefit from the other components of the framework?

Reviewer_CWhVOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a world model that predicts future states to improve web agent decision-making. It introduces Typed Spatial Markdown (TSM), a structured representation of UI elements, and trains an action-conditioned world model to predict next states (so basically, the web agent predicts candidate actions -> the world model predicts the next 'TSM' -> the agent reevaluates the action based on the predicted next state). Experiments show improved next-state prediction accuracy and web agent task success rates.

Critical strengths

TSM itself makes sense and the motivation is clear.

Critical weaknesses
  1. Computational overhead. This method requires an additional world model, requiring additional training and inference costs. Especially, inference cost is a bottleneck. The agent should generate 1) multiple candidates with reasoning, 2) the world model needs to predict the next state, and 3) then do state-value prediction again to select the action.

  2. Weak OOD generalization. Despite the claims, prediction accuracy remains low (EleF1: 26.50), raising doubts about generalization to unseen websites.

  3. Unclear motivation for world modeling. This is the most critical part. Browsers already provide exact state transitions + very easy and cheap to simulate (unlike physical world models). Why do we need an approximate world model (that can be wrong + that cannot generalize to OOD). I partially agree with "irreversible action" (e.g., deleting), but we can have two simulations to prevent this (e.g., having a copied simulation used for next-state simulation).

  4. Marginal improvements. The improvements are marginal to the strongest baselines (37.61% vs. 35.78% on Online-Mind2Web), despite introducing an additional trained 9B model and agent-side adaptation. More importantly, performance on VisualWebArena decreases from 17.17% to 16.74% for DeepSeek-V4.1-Flash. This raises concerns about whether the approach can meaningfully benefit stronger agents, where the value of an auxiliary world model is arguably most important.

Questions for authors

See the weakness above.

Reviewer_FHa1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

To support GUI agents’ decision-making, they propose MKD2World, an action-conditioned world model that predicts the next interface state from the current screenshot and a candidate action. Its central component is Typed Spatial Markdown (TSM), which represents UI elements through their types, semantics, locations, control states, and hierarchical relationships. They construct AtomBlock-WebUI to support screenshot-to-TSM conversion and train MKD2World through current-state reconstruction followed by next-state prediction. At inference time, the GUI agent interprets predicted states through a digest step and selects an action. Experiments on Online-Mind2Web and VisualWebArena demonstrate improved task success in several agent configurations.

Critical strengths
  • Suitable state representation for GUI decision-making: TSM integrates UI semantics, element types, spatial locations, control states, and hierarchical relationships into a readable textual representation. By preserving both action-induced changes and their surrounding context, it provides useful information for comparing candidate actions without requiring pixel-level reconstruction.
  • Effective methods for data construction, world-model training, and action selection: The pipeline combines pre-injected type labels for direct bounding-box extraction, training objectives that emphasize action-induced changes, and a digest step that connects predictions to task progress. The ablation results support the effectiveness of the proposed training and inference components.
Critical weaknesses
  • Insufficient isolation of TSM’s contribution: Although TSM is reasonably motivated, the experiments do not isolate its contribution from training and inference differences, and TSM-aware training measures agent adaptation rather than the representation’s advantage. Controlled comparisons against alternative representations and ablations of spatial information, control states, and hierarchy would clarify which properties drive the improvements.
  • Insufficient analysis of inference-time cost versus effectiveness: Candidate-specific prediction, digest generation, and action selection introduce additional computation, but invocation frequency, token usage, latency, and total cost per task are not reported. Comparisons under matched inference budgets would clarify the practical value of the gains. Analyzing the confidence threshold for invoking the world model would further characterize the trade-off between computation and task success.
Questions for authors
  • Could you compare MKD2World with alternative approaches under similar inference-time budgets, such as tree-search agents [1] or an agent that generates and evaluates its own action candidates without a separate world model? Such comparisons would help distinguish the benefits of world-model predictions from those of additional inference-time computation.
  • Could you clarify how TSM facilitates interface understanding and learning action-conditioned transitions, ideally through controlled comparisons with alternative representations? What limitations does TSM have, and what extensions might address them?
  • Why are comparisons with other world models restricted to Qwen3.5-35B-A3B on Online-Mind2Web and Qwen3.5-9B on VisualWebArena? If this choice reflects computational constraints, could you nevertheless evaluate the competing methods with both Qwen3.5-9B and Qwen3.5-35B-A3B on both benchmarks to assess whether the relative gains hold across agent sizes?

[1] Tree Search for Language Model Agents

1 comment

Thank you for the detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns through the new comparisons with tree search and other world models, the inference-time cost analysis, and the TSM component ablations.

I believe these additional results and clarifications have adequately addressed my main concerns. Incorporating them into the revised manuscript would further strengthen the paper and provide a more comprehensive evaluation of the proposed method.

I have no further concerns at this point, and I appreciate the authors' thoughtful responses.

Comments on the paper

1 comment
Area_Chair_jWhROpenReview ↗

AC Note: Rebuttal Received, Please Engage in Discussion

Dear authors and reviewers,

Thank you to the authors for their detailed rebuttal and for the time and care they put into addressing the reviewers' comments. I appreciate the additional clarifications and results.

Reviewers: please read the authors' responses carefully and reply to them before the end of the discussion period. Even a short acknowledgment is valuable. It is especially helpful to say whether your concerns have been addressed and, if not, what remains unresolved. If the rebuttal changes your assessment, please update your score and review accordingly, whether up or down. If your score stays the same, a brief explanation of why helps me and the senior area chairs.

Authors: I will follow the discussion closely and take your rebuttal fully into account when forming my recommendation.

Thank you all for your time and effort.

Best regards, Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.