- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces AutoDiagram-1000-v1, a dataset of handwritten computer science diagrams with human-verified descriptions, TikZ code, rendered images, and error annotations. The authors also establish baseline evaluations using existing vision-language models.
Critical strengths
- Meaningful dataset contribution. The paper addresses the underexplored task of understanding handwritten technical diagrams.
- Clear baseline evaluation. The paper provides baseline results using existing VLMs, serving as a reference for future research.
Critical weaknesses
- Limited reconstruction evaluation. The experiments focus on text descriptions, without evaluating diagram reconstruction or structural correctness.
- Limited evaluation reliability. The test set contains only 60 diagrams, which may not be enough to reliably evaluate model performance. The data splitting strategy is also unclear.
- Insufficient annotation quality analysis. Quantitative inter-annotator agreement is missing, making annotation consistency difficult to assess.
- Limited overall contribution. The dataset is relatively small and restricted to automata-related coursework.
Questions for authors
Data Splitting and Evaluation: How were the training and test sets split? Given the small test set, could the authors consider five-fold cross-validation to provide more reliable results?