A Dataset for Understanding and Reconstructing Handwritten Computer Science Diagrams

Handwritten technical diagrams are widely used in science, engineering, and education to communicate concepts, relationships, and processes. Automatic interpretation of such diagrams has applications in the digitization of handwritten material, evaluation of student work, error identification, and conversion of handwritten diagrams into editable digital representations. However, handwritten technical diagrams remain underrepresented in existing datasets, which limits the data available for training and evaluating models for diagram understanding and generation. We present AutoDiagram-1000-v1, a dataset of 1,000 handwritten diagrams from automata theory paired with natural language descriptions that describe visual content without requiring automata theory terminology. Each description is manually verified against the handwritten diagram and further validated through diagram reconstruction. Specifically, the description is converted into TikZ code and rendered as a digital diagram, after which the reconstruction is compared with the original handwritten diagram to verify the correspondence between the description and the source diagram. In addition to verified descriptions, the dataset includes incorrect descriptions, their resulting reconstructions, and annotations of the observed errors. This combination provides examples of both correct and incorrect mappings between handwritten diagrams and their textual representations. We further fine tune and evaluate two models to establish an initial benchmark for handwritten diagram description and generation.