acceptodds
Under review as a conference paper at ICLR 2027

mxSRBench: Towards Spatial Understanding and Diagrammatic Reasoning in mxGraph for Large Language Models

Abstract

While LLMs have advanced code synthesis and semantic reasoning, their ability to bridge the spatial–semantic gap in diagrams remains poorly understood. Diagrams demand satisfaction of semantic topology and metric geometry, yet existing benchmarks, focused on VQA or pixel generation, fail to isolate the resulting symbolic reasoning failures. We introduce mxSRBench, a benchmark of 3,990 instruction–XML pairs built on mxGraph, a high-fidelity symbolic language that explicitly encodes spatial logic. mxSRBench offers three tiers: T1 validates XML and structural renderability; T2 probes spatial reasoning; and T3 evaluates professional diagram synthesis across 11 types. For T2, we establish an Evaluation Protocol covering three core capabilities: (1) Understanding (interpreting spatial logic from XML), (2) Generation (grounding textual spatial constraints into valid coordinate XML), and (3) Multihop Reasoning (compositional multi-step spatial inference). Systematic benchmarking of LLMs/VLMs reveals a spatial–semantic gap: top models achieve up to 91.7% in understanding but degrade sharply on constrained generation and multihop reasoning, confirming they are sophisticated logicians yet poor drafters. mxSRBench provides a quantitative, diagnostic foundation for neuro-symbolic and tool-augmented diagram generation.

Primary area:
datasets and benchmarks
Keywords:
Spatial Reasoning, Diagram Understanding, Diagram Generation, Multi-hop Reasoning, mxGraph XML, Large Language Model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ootUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces mxSRBench, a benchmark for evaluating the spatial understanding and structured diagram generation capabilities of large language models using mxGraph XML. The benchmark consists of three tiers: T1 validates XML correctness and renderability, T2 evaluates spatial understanding, constrained generation, and multihop reasoning across six dimensions, and T3 assesses professional diagram synthesis across different diagram types. The dataset contains 3,990 instruction–XML pairs constructed through LLM-assisted generation, rule-based validation, and quality control. The authors evaluate various LLMs using accuracy, decomposed requirement satisfaction, and VLM-based diagram quality scores. Experimental results reveal performance differences across models and tasks, highlighting challenges in semantic grounding and complex diagram generation.

Critical strengths

S1. The paper addresses an interesting and practical topic of evaluating LLMs on structured and editable diagram generation, which is not fully covered by existing spatial reasoning benchmarks.

S2. The proposed benchmark provides a systematic evaluation framework covering format validation, spatial reasoning, and professional diagram synthesis. The experiments cover multiple representative LLMs, and the error analysis provides useful insights into their limitations.

Critical weaknesses

W1. The experimental results do not sufficiently support the central claim of a spatial–semantic gap. The evaluation relies on different metrics across tasks, making it difficult to draw direct conclusions about the relative capabilities of spatial understanding and generation.

W2. The validity and generalizability of the benchmark remain insufficiently established. The use of synthetic diagrams and reference-based VLM evaluation may introduce biases, while the impact of mxGraph-specific representation knowledge on model performance has not been sufficiently quantified.

Questions for authors

Q1. In Table 3, Gemini-3-Pro achieves 91.7% accuracy on Understanding, 90.57 DRFR on Generation, and 92.85 DRFR on Multihop Generation. These results do not clearly support the claimed sharp performance degradation from understanding to generation. Moreover, Accuracy and DRFR measure different aspects of performance and are not directly comparable. Could the authors clarify how the spatial–semantic gap is established from these results? Would an instance-level success rate, requiring all constraints to be satisfied, provide a more meaningful comparison?

Q2. For T2 Generation, the authors adopt DRFR based on automatically decomposed requirements. Although the reported agreement between the automated judge and human annotators is high, this only validates the individual verification decisions rather than the completeness of the generated checklists. How do the authors ensure that all essential constraints are included and appropriately weighted? Could the authors provide further validation of the requirement decomposition process?

Q3. Section 4.3 identifies grounding-related errors, such as invalid mxGraph shape names and missing style attributes, as well as constraint propagation errors. However, the analysis does not quantify how much of the performance degradation results from unfamiliarity with mxGraph rather than spatial reasoning limitations. Could the authors provide a controlled experiment, such as supplying models with an mxGraph shape dictionary or API documentation, to better distinguish these factors?

Q4. According to Section 3.2 and Appendix C.3, the T3 dataset is constructed by using Gemini-3-Pro to generate diagram descriptions, translating them into Mermaid code, converting the Mermaid code into mxGraph XML, and then regenerating natural-language instructions from the resulting XML.

Q5. The T3 evaluation uses a VLM to compare rendered images of the generated and reference diagrams. However, different diagrams may satisfy the same user instruction while having different visual layouts. Conversely, visually similar diagrams may contain incorrect underlying structures. How does DSS account for these situations? Although the paper provides illustrative expert rankings in Figure 4 and reports cross-judge correlations, could the authors provide a more systematic quantitative evaluation of DSS against human judgments on semantic and structural correctness?

Q6. Appendix B.1 reports substantial XML generation failure rates for some models, particularly on T3, where Qwen3-VL-235B has an overall failure rate of 49.94%. How are these invalid or non-renderable outputs handled when calculating the main evaluation scores in Tables 3 and 5? Are they excluded or assigned zero scores? This distinction could significantly affect the reported model comparisons.

1 comment

Thanks for the authors' response, and most of my concerns are addressed. As the authors have also acknowledged, several fundamental issues still remain, such as the over-claimed understanding–generation gap, the generalizability of the synthetic benchmark, the limitations of DSS. They require more refinement of the benchmark design and evaluation, because they directly affect the motivation and overall contribution. Therefore, I will maintain my original rating.

Reviewer_ZbpKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces mxSRBench, a benchmark of 3,990 instruction–XML pairs for evaluating diagram understanding and generation in mxGraph. Its three-tier evaluation covers format validity, spatial understanding and constrained generation (including multihop reasoning) and professional diagram synthesis. The benchmark spans six spatial dimensions and eleven diagram types, with evaluations of multiple LLMs and VLMs, human validation, and analyses of representative failures.

Critical strengths
  1. The benchmark addresses a useful application: generating editable diagrams with explicit structural and geometric constraints. Its coverage of spatial dimensions and professional diagram types provides a more detailed assessment than a single aggregate diagram-generation score.
  2. The evaluation includes useful validation efforts. The automated T2 generation evaluator achieves 96.1% agreement with human annotations across 383 atomic checks. The alternative-judge analysis further supports the stability of the overall T3 model ranking.
  3. The error analysis covers both grounding errors, such as incorrect shape/style names and missing attributes, and constraint propagation errors involving scheduling semantics and temporal ordering. These observations are informative for improving practical diagram-generation systems.
Critical weaknesses
  1. The claim in the abstract and conclusion that models are strong interpreters but poor drafters needs stronger support. Table 3 shows no consistent understanding–generation gap across the closed-source models: the two Gemini models have similar aggregate scores, while GPT-5.2 and both Claude models have numerically higher generation scores. More fundamentally, understanding accuracy and partial-credit DRFR are not directly comparable. The results clearly show a drop on multihop understanding, but do not establish a general sharp degradation in generation.
  2. Format familiarity and spatial reasoning are not sufficiently separated. The XML instructions in Appendix E are helpful, but several highlighted failures concern mxGraph-specific shape names, style tokens, or required attributes. These are relevant practical failures, yet it remains unclear how much they reveal about spatial reasoning rather than knowledge of the format and its libraries.
  3. The model coverage limits the conclusions about the benchmark's difficulty for more recent models. The main evaluation uses GPT-5.2, Claude 4.5, and Gemini 3 variants, while GPT-5.4 is already used as an alternative judge. Evaluating a more recent representative model as a participant would strengthen the empirical picture.
Questions for authors
  1. Could you report the percentage of generation examples satisfying all requirements, alongside DRFR, and clarify which results support the claimed understanding–generation gap? If the tasks and metrics are not directly comparable, narrowing the conclusion would be appropriate.
  2. On a representative subset, could you evaluate newer models, with and without references for the relevant mxGraph shapes and styles? This would help assess both the remaining difficulty and the contribution of format familiarity. Other targeted analyses addressing these concerns would also be welcome.
1 comment

Thanks for your response. Most of my concerns have been addressed. I agree that the general claim about the understanding–generation gap should be dropped. I still see value in a benchmark that captures differences across editable diagram tasks. Given GPT-6-Sol’s high scores on both T2 understanding and generation in the subset evaluation, I would encourage future work to examine how well DSS captures structural correctness in professional diagrams. I will raise my rating.

Reviewer_uz7MOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces mxSRBench, the first understanding and diagrammatic reasoning benchmark for LLMs built on mxGraph (XML specification of draw.io), including 3990 natural language instruction-XML pairs.

The benchmark adopts a three-tier progressive evaluation suite: 1) Format Validation to test whether generated code renders properly in draw.io. 2) Spatial Reasoning to evaluate spatial logic and coordinate grounding; and 3) Professional Diagram Generation to evaluate end-to-end diagram synthesis.

Critical strengths
  • Compared to semantic DSLs like Mermaid or DOT, which specify logical connections but leaving spatial placement to an automatic black-box renderer, xmGraph combines explicit 2D canvas coordinates with structured relational tags, making both layout accuracy and logical topology directly verifiable.
  • The benchmark is claimed to be human-aligned. Tier-2 utilizes the DRFR that breaking prompts into atomic Boolean checks verified by an automated judge, achieving 96.1% agreement with human annotators. Tier-3 uses rubic-based VLM evaluation and the judge analysis using gpt-5.4 and claude opus 4.5 suggesting the model-level ranking robust to the choice of judge.
Critical weaknesses
  • While the author positions mxSRBench as a foundation for tool-augmented generation, all models are evaluated zero-shot without any external aids. However, as a foundation tool-augmented diagram, no tool-augmented baseline is reported.
  • Semantic generation is the weakest dimension for all 12 models, which the paper attributes to models hallucinating draw.io shape names. This measures recall of a tool-specific output rather than spatial reasoning, so treating it as a spatial dimension is misleading. In practice, it could be handled by retrieval or tool calls.
  • In Tier-2, models are given only raw XML text. Without an image-only or image+XML condition, it is unclear whether errors reflect spatial reasoning or simply parsing long XML.
Questions for authors
  1. Which reasoning configuration was used for each model? Could the gap between GPT-5.2 vs. GPT-5.2-Codex reflect a difference in reasoning effort rather than code pre-training?
  2. Tool-augmented baseline. Since the benchmark is positioned as a foundation for tool-augmented generation, could you evaluate at least one setting?
  3. How does mxSRBench differ from VCG-Bench, which also builds an xmGraph benchmark with DRFR-style evaluation?

Su, X., Dong, P., Tang, Z., Tang, S., Zhai, Y., and others. VCG-Bench: Towards A Unified Visual-Centric Benchmark for Structured Generation and Editing. ICML 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.