acceptodds
Under review as a conference paper at ICLR 2027

A Dataset for Understanding and Reconstructing Handwritten Computer Science Diagrams

Abstract

Handwritten technical diagrams are widely used in science, engineering, and education to communicate concepts, relationships, and processes. Automatic interpretation of such diagrams has applications in the digitization of handwritten material, evaluation of student work, error identification, and conversion of handwritten diagrams into editable digital representations. However, handwritten technical diagrams remain underrepresented in existing datasets, which limits the data available for training and evaluating models for diagram understanding and generation. We present AutoDiagram-1000-v1, a dataset of 1,000 handwritten diagrams from automata theory paired with natural language descriptions that describe visual content without requiring automata theory terminology. Each description is manually verified against the handwritten diagram and further validated through diagram reconstruction. Specifically, the description is converted into TikZ code and rendered as a digital diagram, after which the reconstruction is compared with the original handwritten diagram to verify the correspondence between the description and the source diagram. In addition to verified descriptions, the dataset includes incorrect descriptions, their resulting reconstructions, and annotations of the observed errors. This combination provides examples of both correct and incorrect mappings between handwritten diagrams and their textual representations. We further fine tune and evaluate two models to establish an initial benchmark for handwritten diagram description and generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
hand written diagram

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ZnFQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces AutoDiagram-1000-v1, a dataset of handwritten computer science diagrams with human-verified descriptions, TikZ code, rendered images, and error annotations. The authors also establish baseline evaluations using existing vision-language models.

Critical strengths
  1. Meaningful dataset contribution. The paper addresses the underexplored task of understanding handwritten technical diagrams.
  2. Clear baseline evaluation. The paper provides baseline results using existing VLMs, serving as a reference for future research.
Critical weaknesses
  1. Limited reconstruction evaluation. The experiments focus on text descriptions, without evaluating diagram reconstruction or structural correctness.
  2. Limited evaluation reliability. The test set contains only 60 diagrams, which may not be enough to reliably evaluate model performance. The data splitting strategy is also unclear.
  3. Insufficient annotation quality analysis. Quantitative inter-annotator agreement is missing, making annotation consistency difficult to assess.
  4. Limited overall contribution. The dataset is relatively small and restricted to automata-related coursework.
Questions for authors

Data Splitting and Evaluation: How were the training and test sets split? Given the small test set, could the authors consider five-fold cross-validation to provide more reliable results?

Reviewer_kdMCOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces AutoDiagram-1000-v1, a dataset of 1,000 handwritten computer science diagrams, primarily from automata theory. Each diagram is accompanied by human-verified descriptions, TikZ representations, rendered images, and error annotations. The authors also evaluate several vision-language models and conduct fine-tuning experiments to demonstrate the potential utility of the dataset.

Critical strengths
  1. The paper addresses an interesting and relatively underexplored problem of understanding and reconstructing handwritten technical diagrams.

  2. The dataset provides multiple forms of annotations, including natural language descriptions, executable TikZ code, and structural error annotations, which could support different diagram understanding tasks.

Critical weaknesses
  1. The dataset scale and generalizability are limited. AutoDiagram-1000-v1 contains only 1,000 images collected primarily from automata theory courses, covering a relatively narrow range of diagram types and sources. Moreover, the fine-tuning experiments use only 540 training images and 60 test images. Such a limited scale raises concerns about the diversity, representativeness, and generalizability of the dataset, particularly for broader handwritten computer science diagram understanding tasks. The paper also does not sufficiently clarify whether the data splits prevent overlap in question templates or diagram structures.

  2. The experimental evaluation is insufficient to establish the dataset's research value. Although the dataset provides rich annotations for diagram reconstruction and structural error analysis, the experiments mainly focus on text description generation using BLEU, ROUGE, and similar metrics. These metrics cannot adequately evaluate the correctness of nodes, edges, connectivity, and other critical diagram structures. Furthermore, comparisons with existing specialized diagram recognition methods are missing, and the quality of the reconstructed diagrams is not systematically evaluated. Consequently, the experiments do not sufficiently demonstrate the advantages and practical utility of the proposed dataset.

Questions for authors
  1. Could the authors clarify how training and test samples are separated, particularly whether diagrams from the same questions, students, or similar structural templates appear across different splits?

  2. Given the limited dataset size and its concentration on automata theory, how can the authors establish its representativeness and generalizability to broader handwritten technical diagrams?

  3. Could the authors provide evaluations based on structural correctness, such as node recognition, edge connectivity, and graph reconstruction accuracy, rather than relying mainly on text similarity metrics?

  4. Could the authors compare existing specialized diagram recognition or reconstruction methods on the proposed dataset to establish its value as a benchmark?

  5. Could the authors provide a more systematic evaluation of annotation quality, particularly the consistency between handwritten diagrams, descriptions, and TikZ reconstructions, including cases labeled as unacceptable?

Reviewer_GrE1OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper introduces AutoDiagram-1000-v1, a dataset of handwritten diagrams, paired with human-reviewed descriptions, TikZ code, rendered diagrams, and error annotations. It also evaluates diagram description generation and reports improvements after fine-tuning Qwen2.5-VL-7B and PaliGemma2-3B.

The data collection is useful, but the positioning and evaluation are insufficient to establish a strong benchmark contribution.

Critical strengths
  1. The dataset discusses a useful problem.
  2. The paper is easy-to-follow.
Critical weaknesses
  1. The evaluated models are relatively dated for an ICLR 2027 submission. Results on Qwen2.5-VL, PaliGemma2, Claude 4 Sonnet, and Gemini 2.5 Pro do not adequately characterize the capabilities of contemporary VLMs.
  2. The evaluation relies on text-similarity metrics, which do not reliably measure diagram correctness.
  3. The dataset sizes (train/val/test) in Experiment 2 are quite small.
  4. DeTikZify and SkeTikZ are not discussed in related works, which also provide sketch diagrams.
Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.