- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces mxSRBench, a benchmark for evaluating the spatial understanding and structured diagram generation capabilities of large language models using mxGraph XML. The benchmark consists of three tiers: T1 validates XML correctness and renderability, T2 evaluates spatial understanding, constrained generation, and multihop reasoning across six dimensions, and T3 assesses professional diagram synthesis across different diagram types. The dataset contains 3,990 instruction–XML pairs constructed through LLM-assisted generation, rule-based validation, and quality control. The authors evaluate various LLMs using accuracy, decomposed requirement satisfaction, and VLM-based diagram quality scores. Experimental results reveal performance differences across models and tasks, highlighting challenges in semantic grounding and complex diagram generation.
Critical strengths
S1. The paper addresses an interesting and practical topic of evaluating LLMs on structured and editable diagram generation, which is not fully covered by existing spatial reasoning benchmarks.
S2. The proposed benchmark provides a systematic evaluation framework covering format validation, spatial reasoning, and professional diagram synthesis. The experiments cover multiple representative LLMs, and the error analysis provides useful insights into their limitations.
Critical weaknesses
W1. The experimental results do not sufficiently support the central claim of a spatial–semantic gap. The evaluation relies on different metrics across tasks, making it difficult to draw direct conclusions about the relative capabilities of spatial understanding and generation.
W2. The validity and generalizability of the benchmark remain insufficiently established. The use of synthetic diagrams and reference-based VLM evaluation may introduce biases, while the impact of mxGraph-specific representation knowledge on model performance has not been sufficiently quantified.
Questions for authors
Q1. In Table 3, Gemini-3-Pro achieves 91.7% accuracy on Understanding, 90.57 DRFR on Generation, and 92.85 DRFR on Multihop Generation. These results do not clearly support the claimed sharp performance degradation from understanding to generation. Moreover, Accuracy and DRFR measure different aspects of performance and are not directly comparable. Could the authors clarify how the spatial–semantic gap is established from these results? Would an instance-level success rate, requiring all constraints to be satisfied, provide a more meaningful comparison?
Q2. For T2 Generation, the authors adopt DRFR based on automatically decomposed requirements. Although the reported agreement between the automated judge and human annotators is high, this only validates the individual verification decisions rather than the completeness of the generated checklists. How do the authors ensure that all essential constraints are included and appropriately weighted? Could the authors provide further validation of the requirement decomposition process?
Q3. Section 4.3 identifies grounding-related errors, such as invalid mxGraph shape names and missing style attributes, as well as constraint propagation errors. However, the analysis does not quantify how much of the performance degradation results from unfamiliarity with mxGraph rather than spatial reasoning limitations. Could the authors provide a controlled experiment, such as supplying models with an mxGraph shape dictionary or API documentation, to better distinguish these factors?
Q4. According to Section 3.2 and Appendix C.3, the T3 dataset is constructed by using Gemini-3-Pro to generate diagram descriptions, translating them into Mermaid code, converting the Mermaid code into mxGraph XML, and then regenerating natural-language instructions from the resulting XML.
Q5. The T3 evaluation uses a VLM to compare rendered images of the generated and reference diagrams. However, different diagrams may satisfy the same user instruction while having different visual layouts. Conversely, visually similar diagrams may contain incorrect underlying structures. How does DSS account for these situations? Although the paper provides illustrative expert rankings in Figure 4 and reports cross-judge correlations, could the authors provide a more systematic quantitative evaluation of DSS against human judgments on semantic and structural correctness?
Q6. Appendix B.1 reports substantial XML generation failure rates for some models, particularly on T3, where Qwen3-VL-235B has an overall failure rate of 49.94%. How are these invalid or non-renderable outputs handled when calculating the main evaluation scores in Tables 3 and 5? Are they excluded or assigned zero scores? This distinction could significantly affect the reported model comparisons.
Thanks for the authors' response, and most of my concerns are addressed. As the authors have also acknowledged, several fundamental issues still remain, such as the over-claimed understanding–generation gap, the generalizability of the synthetic benchmark, the limitations of DSS. They require more refinement of the benchmark design and evaluation, because they directly affect the motivation and overall contribution. Therefore, I will maintain my original rating.