- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies vision-language models' understanding of symbolic drawings, focusing on mechanical CAD drawings and mathematical geometry diagrams. The authors design domain-specific structured representations and use an existing VLM to extract objects, dimensions, annotation bindings, and geometric relations from images. These representations are supplied as JSON/XML to assist question answering. The paper further introduces Structure-Guided On-Policy Distillation (SG-OPD), in which a structure-assisted teacher provides token-level supervision on trajectories generated by an image-only student. The resulting student requires no additional structured input at inference time.
The experiments show that providing structured information generally improves answer accuracy. SG-OPD achieves substantial improvements on the evaluated MathVista subsets, while its gains on MechVQA and most MathVerse variants are comparatively small.
Critical strengths
-
The motivation is clear and relevant. Object–attribute binding, annotation association, and cross-view correspondence are important challenges in understanding symbolic drawings.
-
Combining structure-assisted reasoning with an image-only deployment setting is practically useful. The distilled student retains the original image-and-question interface and avoids the additional structure-generation step at inference time.
-
The structure-assisted experiments cover multiple models and include JSON/XML serialization, JSON-only inference, native JSON generation, and consistent object-ID renaming. These experiments provide useful evidence for the value of structured descriptions and reveal differences across models and serialization formats.
Critical weaknesses
-
The question-independent formulation needs clarification in light of the appendix prompt. L.170 states that the generator cannot access the external problem statement or question. However, the mathematics prompt in the appendix permits reading “visible problem text” and explicitly requests query fields such as
find_length,find_angle, andprove. It is unclear which inputs were actually provided to the generator and whether query fields were retained. This may simply be a mismatch between the documented prompt and the implementation; it does not by itself establish information leakage. -
The transfer gains vary considerably across benchmarks, and the stability of the smaller gains is unclear. SG-OPD produces substantial improvements on MathVista, but only a 0.60 percentage-point gain on MechVQA and gains of 0.12–0.25 percentage points on four of the five MathVerse variants. The paper appropriately acknowledges limited transfer in these settings. Nevertheless, without information about result variability, it is difficult to assess whether these small improvements are robust. Claims about internalizing structural understanding should be interpreted in light of this variation.
Questions for authors
-
What inputs were actually provided to the mathematics structure generator? Did it receive any external problem text, and were the
queryfields retained in the generated JSON? Could the descriptions of question independence in the main text and appendix be reconciled? -
Could the authors provide a small qualitative comparison of the reasoning traces of the original image-only student, the structure-assisted teacher, and the SG-OPD-trained student on the same examples? It would be helpful to identify whether improvements correspond to correcting object identification, dimension binding, or relation-use errors. Including both successful and unsuccessful transfer cases would help explain the results. Such traces would provide behavioral evidence, although similarity between traces alone would not establish that the models use the same internal reasoning mechanism.
-
Do the authors have any existing repeated-run results or other information about result variability that could help assess the stability of the small improvements on MechVQA and most MathVerse variants?
-
As an optional extension, it would be interesting to explore whether the approach also helps with circuit schematics, which are mentioned among the examples of symbolic drawings in L.120. Public benchmarks such as CircuitVQA and CircuitSense could offer useful testbeds. This is a suggestion for future exploration rather than a request for additional experiments in the revision; the current focus on CAD and mathematical diagrams is reasonable.