acceptodds
Withdrawn from ICLR 2027

Learning to Understand Symbolic Drawings with Structure-Guided On-Policy Distillation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Vision language models (VLMs) perform well on natural images and documents but often struggle with complex symbolic drawings, such as mechanical CAD drawings and mathematical geometry diagrams. These drawings encode information through sparse geometry, numerical annotations, multiple views, and formal relations, requiring accurate object identification and annotation binding. We first study structure-assisted reasoning, where a question-independent structured description is provided alongside the image to explicitly represent geometric elements, dimensions, cross-view correspondences, and constraints. Experiments show that such information consistently improves question-answering performance, suggesting that many failures arise from unstable visual-to-structural interpretation. We further propose Structure-Guided On-Policy Distillation (SG-OPD) to transfer this benefit into an image-only model. During training, a structure-assisted teacher provides token-level supervision on trajectories generated by the image-only student. Experiments on mechanical CAD drawings and mathematical geometry diagrams show that SG-OPD consistently improves image-only baselines without requiring structured inputs at inference time. These results demonstrate that explicit structure can facilitate symbolic visual reasoning and be effectively internalized through on-policy distillation.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
CAD Understanding, Geometry Reasoning, Vision-Language Models, On-Policy Distillation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_oiW3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies vision-language models' understanding of symbolic drawings, focusing on mechanical CAD drawings and mathematical geometry diagrams. The authors design domain-specific structured representations and use an existing VLM to extract objects, dimensions, annotation bindings, and geometric relations from images. These representations are supplied as JSON/XML to assist question answering. The paper further introduces Structure-Guided On-Policy Distillation (SG-OPD), in which a structure-assisted teacher provides token-level supervision on trajectories generated by an image-only student. The resulting student requires no additional structured input at inference time.

The experiments show that providing structured information generally improves answer accuracy. SG-OPD achieves substantial improvements on the evaluated MathVista subsets, while its gains on MechVQA and most MathVerse variants are comparatively small.

Critical strengths
  1. The motivation is clear and relevant. Object–attribute binding, annotation association, and cross-view correspondence are important challenges in understanding symbolic drawings.

  2. Combining structure-assisted reasoning with an image-only deployment setting is practically useful. The distilled student retains the original image-and-question interface and avoids the additional structure-generation step at inference time.

  3. The structure-assisted experiments cover multiple models and include JSON/XML serialization, JSON-only inference, native JSON generation, and consistent object-ID renaming. These experiments provide useful evidence for the value of structured descriptions and reveal differences across models and serialization formats.

Critical weaknesses
  1. The question-independent formulation needs clarification in light of the appendix prompt. L.170 states that the generator cannot access the external problem statement or question. However, the mathematics prompt in the appendix permits reading “visible problem text” and explicitly requests query fields such as find_length, find_angle, and prove. It is unclear which inputs were actually provided to the generator and whether query fields were retained. This may simply be a mismatch between the documented prompt and the implementation; it does not by itself establish information leakage.

  2. The transfer gains vary considerably across benchmarks, and the stability of the smaller gains is unclear. SG-OPD produces substantial improvements on MathVista, but only a 0.60 percentage-point gain on MechVQA and gains of 0.12–0.25 percentage points on four of the five MathVerse variants. The paper appropriately acknowledges limited transfer in these settings. Nevertheless, without information about result variability, it is difficult to assess whether these small improvements are robust. Claims about internalizing structural understanding should be interpreted in light of this variation.

Questions for authors
  1. What inputs were actually provided to the mathematics structure generator? Did it receive any external problem text, and were the query fields retained in the generated JSON? Could the descriptions of question independence in the main text and appendix be reconciled?

  2. Could the authors provide a small qualitative comparison of the reasoning traces of the original image-only student, the structure-assisted teacher, and the SG-OPD-trained student on the same examples? It would be helpful to identify whether improvements correspond to correcting object identification, dimension binding, or relation-use errors. Including both successful and unsuccessful transfer cases would help explain the results. Such traces would provide behavioral evidence, although similarity between traces alone would not establish that the models use the same internal reasoning mechanism.

  3. Do the authors have any existing repeated-run results or other information about result variability that could help assess the stability of the small improvements on MechVQA and most MathVerse variants?

  4. As an optional extension, it would be interesting to explore whether the approach also helps with circuit schematics, which are mentioned among the examples of symbolic drawings in L.120. Public benchmarks such as CircuitVQA and CircuitSense could offer useful testbeds. This is a suggestion for future exploration rather than a request for additional experiments in the revision; the current focus on CAD and mathematical diagrams is reasonable.

Reviewer_kM1hOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper investigates how explicit structural descriptions can improve vision-language model understanding of mechanical CAD drawings and mathematical geometry diagrams. It first evaluates structure-assisted inference, where the model receives a description of geometric entities, measurements, and relationships alongside the image. It then proposes Structure-Guided On-Policy Distillation (SG-OPD), which trains an image-only student to match a structure-assisted teacher’s next-token distributions on student-generated trajectories. Direct access to structure generally improves performance across the evaluated models. The distilled student achieves substantial gains on the reported MathVista subsets, but improvements are small on MechVQA and minimal on most MathVerse settings.

Critical strengths
  1. The paper addresses a concrete and practically relevant challenge: correctly associating annotations and geometric relationships with objects in symbolic drawings. The experiments across multiple VLMs provide useful evidence that explicit structural descriptions can help with these tasks.

  2. The MathVista results are promising: the distilled student improves by 8.78–11.54 percentage points on the evaluated subsets without requiring structural inputs at inference. This suggests that the training approach can provide a meaningful benefit in some settings, although the source of that benefit remains unresolved.

Critical weaknesses
  1. The experiments do not isolate the contribution of SG-OPD.

Section 5.3 compares the trained student with the original image-only checkpoint, without matched-data SFT, distillation without structural context, or off-policy distillation from the same structure-assisted teacher. The gains therefore cannot be attributed specifically to structural guidance or on-policy training rather than additional domain training. This is particularly consequential because the method closely follows the privileged-context distillation approaches discussed in Section 2.2, with drawing structure as its main distinguishing feature. Controlled comparisons are needed to establish the value of that feature and the proposed training procedure.

  1. Evidence for effective structural transfer is limited, and the claimed mechanism remains insufficiently tested.

Although MathVista shows substantial gains, MechVQA improves by only 0.60 percentage points and four MathVerse settings by just 0.12–0.25 points. The largest MathVerse gain occurs in Text Dominant. Without uncertainty estimates or repeated runs, the reliability of the small improvements is unclear. These results provide limited support for effective transfer across the evaluated tasks.

The diagnostic experiments also leave alternative explanations unresolved. Since an additional VLM generates the descriptions, comparisons with OCR text or ordinary descriptions from the same generator are needed to isolate the benefit of explicit structure. Consistent ID renaming preserves relational bindings and therefore does not test their importance. Moreover, final-answer accuracy alone does not establish improved annotation binding or cross-view correspondence. The evidence supports the usefulness of auxiliary descriptions more strongly than the broader claim that SG-OPD internalizes structural understanding.

Questions for authors
  1. How does SG-OPD compare with standard SFT on the same training data and off-policy distillation from the same structure-assisted teacher, under comparable training budgets? These comparisons would help distinguish the benefits of additional domain training from those of the proposed on-policy procedure.

  2. What evidence establishes the added value of the structured representation over an auxiliary textual description? A comparison with a similarly detailed prose description from the same generator would help clarify whether the explicit schema and relational organization are important to the gains.

  3. How reliable are the small improvements reported in Tables 3–4? Please provide the evaluated sample counts and paired confidence intervals, together with variation across training seeds if available, particularly for the 0.60-point MechVQA gain and the 0.12–0.25-point gains on four MathVerse settings.

  4. Please clarify the inputs used for mathematical structure generation. Section 3.2 excludes the external question, while Appendix A.1 permits visible problem text and includes a query field. Is the generator given only the image, including any text embedded in it, or also a separate problem statement? How is the query field handled in the reported experiments?

Reviewer_TWcpOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the problem of exploiting visual structure in image interpretation. This is an important challenge, and one of the key areas where associative models fail. The approach exploited is structure-assisted reasoning, and the paper proposes a method labelled structure guided on-policy distillation. This method improves upon the performance of existing approaches for images with suitable structure.

The paper addresses one of the key ongoing problems in computer vision and makes a valuable contribution. The proposed method is a tep on the path, rather than the final solution to the problem, but a valuable step at that.

Critical strengths

The paper approaches one of the core challenges in computer vision, and makes progress towards a solution. It would be unreasonable to expect any single paper to solve such a challenge, so the fact that some progress is made is commendable.

The method presented makes some progress, and improves performance in specific circumstances, which is enough.

Critical weaknesses

The primary weakness of the paper is that the method it proposes requires a-priori enumeration of all of the relationships required to answer an unseen set of {image,question} tuples. This is impossible.

A secondary weakness is the assumption that every important image structure can be represented as text, and that a method that can achieve the transformation from pixels to corresponding text can be generated. This is not the case.

In both cases I think that it is clear that the proposed method improves performance in the cases where the required relationships are in fact predicted, and methods for identifying them available.

The references are fundamentally inadequate. The use of structure in image interpretation has been an ongoing subject in computer vision since the beginning and the oldest reference I can see is from 2021. The paper should compare its contribution against the last 100 years of progress. Some places to start might be: 1/ Wertheimer (1923), “Laws of Organization in Perceptual Forms.” 2/ Attneave (1954), “Some informational aspects of visual perception.” but there are thousands of publications on the issue.

Questions for authors

SOme minor points:

  • The language used needs to be much more accurate. It is currently so inaccurate as to impede understanding. Some examples: 1/ Line 152: 'a symbolic drawing image' -> 'an image depicting a symbolic drawing'. This is a critical distinction 2/ Line 152: 'q its question' -> images don't have questions as they are inanimate objects. I assume also that your method is applicable in the case where there are multiple questions per image. 3/ Line 153: Your 'image-only VLM' seems to take as input a question q as natural language. It's thus not image-only. 4/ Line 158: 'preceding prefix' is a tautology, and doesn't actually say what y_{<t} is 5/ Line 164 'For a drawing domain d' -> 'For drawings falling within the domain d' or something like that. 6/ Line 167: I'd encourage you to have a really hard look at this as I think having two versions of G_d with different numbers of parameters is confusing, particularly as you have stated that parameters q, y, and \rho have no effect

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.