- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Knossos, a synthetic benchmark containing 19,200 diagrams across six domains for complete diagram-to-graph topology extraction. It also proposes Ariadne, a two-stage framework that first identifies diagram nodes and then predicts edges through source-conditioned grouping. Experiments demonstrate substantial improvements over direct graph extraction
Critical strengths
1: Large-scale benchmark: Knossos provides complete node, edge, relation-type, and geometric annotations across six diagram domains, offering valuable supervision for structured diagram understanding.
2: Strong experimental results: Ariadne significantly outperforms existing baselines on the synthetic benchmark, with ablations demonstrating the effectiveness of staged extraction.
3: Comprehensive evaluation: The paper compares diverse open- and closed-source VLMs, includes external generalization experiments, and analyzes structural complexity and geometric grounding.
Critical weaknesses
1: Limited methodological novelty: The proposed framework mainly relies on decomposing graph extraction into node identification and source-group edge prediction. The technical contribution beyond structured prompting and task-specific fine-tuning appears relatively incremental.
2: Limited real-world generalization: The benchmark is entirely synthetic, relying on predefined graph grammars and limited entity vocabularies. Performance drops substantially on the external TopoBench benchmark, raising concerns about generalization to more diverse real-world diagrams.
3: Efficiency concerns: The staged framework requires multiple inference calls, introducing considerable computational overhead. The accuracy–efficiency trade-off deserves further consideration.
4: Incomplete evaluation of graph completeness: Although the paper emphasizes complete topology extraction, the primary evaluation relies on Node F1 and Edge F1. Graph-level exact-match accuracy would provide stronger evidence of whether the entire topology is correctly recovered.
Questions for authors
See weakness.
6 comments
4. Response to Question/Weakness 4
Thank you for this valuable suggestion, which strengthens our assessment of complete topology extraction. We have computed Graph Exact Match (Graph EM), requiring a complete one-to-one node match and an identical directed edge multiset, with no missing or extra nodes or edges. This metric evaluates topology only, excluding relation types and connector geometry. We report results on Knossos because its difficulty annotations allow us to assess complete graph extraction across different levels of diagram complexity. As shown in the table below, Ariadne-Qwen achieves 45.08% Graph EM, compared with 30.33% for GPT-5.6, with higher accuracy across all three difficulty levels. Graph EM consistently decreases as difficulty increases across the evaluated models, supporting the relevance of Knossos's difficulty stratification and highlighting the growing challenge of extracting an entire graph correctly.
We have added the metric definition, full results, and accompanying discussion to Appendix D.5, “Graph-level completeness across difficulty levels.” We have also expanded Section 4.4, “What diagram structures remain challenging?”, to discuss this trend and direct readers to the appendix.
| Model | Simple (164) | Medium (571) | Difficult (465) | Overall (1,200) |
|---|---|---|---|---|
| Qwen3-VL-8B | 11.59% | 1.93% | 0.00% | 2.50% (30/1,200) |
| GPT-4o | 12.20% | 1.93% | 0.00% | 2.58% (31/1,200) |
| GPT-5.6 | 75.00% | 35.55% | 8.17% | 30.33% (364/1,200) |
| Ariadne-Qwen | 84.76% | 58.49% | 14.62% | 45.08% (541/1,200) |
We thank the reviewer for the feedback, particularly the suggestion to evaluate graph completeness. We hope the added results address the concerns raised and strengthen confidence in the work.
Thanks for the reply, which addressed some of my concerns. Regarding the novelty, the author said "During development, we investigated geometry-assisted approaches, including joint connector-trace and edge prediction, as well as reinforcement learning with typed-edge rewards. These more elaborate alternatives did not provide consistent improvements". I would like to learn more about this. Could the author compare a bit more about different methods? I would raise my score if this is properly compared. Also, please clarify the major difference with the paper: "DiagramEval: Evaluating LLM-Generated Diagrams via Graphs". Though it is conducted in SVG format, the methodology behind it is quite similar.
1. Response to Question/Weakness 1
Regarding the reviewer's concern about novelty, we would like to clarify two points.
First, our work makes contributions in both benchmark construction and methodology: Knossos provides complete topology and geometry supervision across six domains, while Ariadne offers an effective framework for learning complete diagram topology extraction. Its central objective is to improve VLMs’ visual perception of diagram entities and their connections through structured supervision. We therefore believe that describing Ariadne primarily as structured prompting understates its focus on learning visual topology perception.
Second, Ariadne's simplicity is motivated by human annotation and supported by empirical exploration. Ariadne is inspired by an intuitive human annotation strategy: first identify the nodes, then focus on one node and annotate its connections before moving to the next. This provides an orderly way to track which parts of the diagram have been covered. During development, we investigated geometry-assisted approaches, including joint connector-trace and edge prediction, as well as reinforcement learning with typed-edge rewards. These more elaborate alternatives did not provide consistent improvements, leading us to retain the simpler topology-first approach as the most effective design among the alternatives we evaluated.
We believe that conceptual simplicity does not diminish a method's contribution. Methods such as ReAct and Chain-of-Thought likewise draw on intuitive human problem-solving strategies. Ariadne's value lies in a logically motivated, effective solution to complete topology extraction, together with the benchmark that enables its systematic study.
2. Response to Question/Weakness 2
We would like to clarify why synthetic generation is appropriate for this task and how we assess its real-world value.
First, synthetic image generation is a common approach to constructing data for diagram-oriented VLM learning and evaluation. For example, PlotQA programmatically renders charts from real-world numerical data, while FlowVQA generates flowchart images by rendering Mermaid code derived from procedural content. These benchmarks illustrate how generated visual data can support structured visual understanding at scale. As models demand more supervision, this approach helps overcome the limited availability and annotation cost of real-world diagram data.
Second, synthesis is particularly well suited to diagrams. Diagrams are deliberately constructed representations of entities and relations, often created with drawing software. Programmatically composing nodes, labels, and connectors reflects this construction process. Generating images and annotations from the same symbolic representation also provides scalable, internally consistent supervision for every node, relation, and connector path, alleviating the substantial cost of complete manual annotation.
Third, to our knowledge, Knossos is the first large-scale benchmark for complete diagram topology extraction, jointly providing typed topology and connector geometry annotations. Its value extends beyond synthetic evaluation: on 162 real-world TopoBench images, both Ariadne variants improve over their corresponding unadapted backbones on both external subsets (Table 1), with gains of 18.2 and 24.5 percentage points in edge F1 on AI2D/Web for GLM and Qwen, respectively. These results support the practical value of our complementary contributions: Knossos provides scalable training supervision, and Ariadne turns that supervision into improved topology extraction on real-world diagrams.
In the newly added Table 2c, Ariadne-Qwen also improves average Edge F1 over the Knossos-trained one-step baseline from 54.8% to 60.9%, with gains on both external subsets. This supports the effectiveness of staged extraction on the evaluated real-world diagrams.
3. Response to Question/Weakness 3
Thank you for raising this concern. The manuscript already examines the accuracy–efficiency trade-off: Section 4.4 discusses the model-dependent benefits, while Appendix D.4 (Table 9) quantifies the protocol overhead in calls, tokens, cost, and latency. Staged extraction benefits smaller, task-adapted VLMs but does not improve GPT-5.6, so its additional cost is not justified for every backbone. To better address the reviewer's concern, we have further expanded Limitations and outlook to clarify Ariadne's intended deployment in offline diagram digitization with task-adapted open-source VLMs, where accuracy and completeness can justify additional inference cost. We also discuss investigating whether decomposition can benefit larger VLMs on more demanding perception tasks as a future research direction.