- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes EQUATION10K, a procedurally generated benchmark for diagnosing handwritten, step-by-step arithmetic solutions. It evaluates whether VLMs can not only recognize steps but also verify transformations, pinpoint the earliest error, and model error propagation, addressing a gap left by benchmarks that focus on final correctness or coarse error detection. Experiments show that performance on joint diagnosis drops with structural complexity and improves with cleaner text inputs, yet substantial diagnostic mistakes persist even with oracle text, suggesting hard limitations in visual transcription and process-level verification.
Critical strengths
The paper generates correct reasoning traces in an executable format, then introduces an error at a chosen step and labels later steps as PROPAGATION if they follow from that wrong state. This setup lets the evaluation check whether a model can identify where the first mistake occurs and whether it keeps making consistent downstream judgments, instead of only scoring the final answer. It also makes the benchmark easier to reproduce because the ground-truth process is defined and testable.
Critical weaknesses
- The benchmark draws on synthetic handwriting in 12 font styles, yet it never checks whether the results carry over to real student work. More fundamentally, it assumes a single linear path from problem to answer. Real students, though, skip steps or rework earlier lines out of order. The framework labels any locally valid transformation after an error as PROPAGATION, which presumes students keep a coherent thread through corrupted states, an assumption classroom data does not support. Its apparent VLM reasoning failures may therefore be artifacts of its own rigid design.
- The five error categories underpin all evaluation metrics, yet they lack grounding in how teachers actually distinguish errors. The paper acknowledges the taxonomy is model-based rather than derived from authentic classroom data. No teacher study validates whether educators can reliably apply these categories or whether real student mistakes even conform to these boundaries. Without inter-rater reliability evidence, claiming JSA and FEIA measure genuine diagnostic capability remains speculative.
- Furthermore, although ETA measures overall error-type classification accuracy, the paper provides no per-category breakdown showing how VLMs perform on each error type. Given that order-of-operations errors are structurally rarer and may require different reasoning patterns than value or calculation errors, disaggregated accuracy by error category would clarify where diagnostic failures originate. Without this analysis, claims about fine-grained diagnostic capability remain coarse.
Questions for authors
The paper evaluates the taxonomy's diagnostic capability on synthetic data. What would be the minimal evidence that would support claims about real-world pedagogical validity? What are the key obstacles to collecting such evidence, and how might they be addressed?
2 comments
Thank you for the response. I acknowledge the added experiments and revisions; several of my concerns have been partially addressed. However, the central validity concern remains only partially resolved, so I have raised my score by one point to reflect the partial progress.
Dear Reviewer,
Thank you for your thoughtful review. We respond to each point below.
[W1a] Synthetic handwriting; no validation on real student work.
We agree that the generalizability of synthetic handwriting is a limitation, already noted in Section 7, and will make it more prominent. We also add an empirical validation: volunteers hand-copy benchmark traces on paper, preserving the original problems and step text while introducing natural pen variation. We evaluate the same models on these scans and report the Spearman rank correlation between synthetic- and handwritten-data leaderboards, together with absolute score differences.
[W1b] The single linear path and PROPAGATION as a framework artifact.
Equation10K tests a necessary condition: a grader unable to reliably verify clean sequences of transitions cannot be expected to handle skipped, reordered, or crossed-out work. We will clarify this scope and avoid presenting the benchmark as representative of all student work.
The PROPAGATION distinction also reflects established educational practice. Teachers routinely award credit for work that is correct given an earlier mistake, commonly termed "follow-through" or "error carried forward." We acknowledge that students may also make multiple independent errors and therefore add a controlled probe set containing a second independent error after the first. We will also clarify that each solution in the main split contains exactly one primary error.
[W2] The five categories lack teacher grounding and inter-rater reliability.
The four primary categories (wrong value, wrong operator, order of operations, and calculation error) reflect recurring computational error patterns in mathematics-education literature. However, we agree that this does not establish consistent application by teachers.
We therefore (i) clarify that JSA and FEIA measure agreement with taxonomy-based diagnostic labels rather than validated pedagogical diagnosis; and (ii) conduct a blinded annotation audit in which teachers or annotators label a stratified sample of solutions from images alone. We report inter-annotator agreement (Fleiss' kappa), agreement with generator labels, and commonly confused label pairs.
Because some injected causes are not identifiable from the displayed trace (e.g., an operator substitution when only the resulting value is shown), we also report scores that accept indistinguishable alternative labels alongside strict scores.
[W3] No per-category breakdown of ETA.
We address this gap by reporting precision, recall, F1, and confusion matrices for all five labels across models, with ORDER analyzed separately due to its lower structural frequency (Section 3.3). We also distinguish primary-error classification accuracy from propagation recall, since PROPAGATION accounts for 65.8% of erroneous steps (9,611/14,611) and substantially influences aggregate ETA.
[Question] Minimal evidence for real-world pedagogical validity and obstacles.
We consider three types of evidence essential: (i) teacher agreement on synthetic items; (ii) teacher-annotated, de-identified real student solutions using the same taxonomy; and (iii) rank agreement between synthetic and real-data leaderboards. Key obstacles include student privacy and consent, annotation costs, and inherent label ambiguity.
In this revision, we address (i) through the blinded annotation study and (iii) through the volunteer hand-copy study, which introduces natural handwriting variation while preserving exact labels. For (ii), we propose collaborating with instructors under institutional review and releasing only de-identified data, leaving this as future work.
Correction.
During revision, we identified that Table 3 reported unweighted means across difficulty levels rather than the step-level micro-averages defined in Appendix B. We also corrected the Qwen2.5-VL-7B handwritten JSA in Figure 3 from 0.157 to 0.147. We recompute all metrics using the stated aggregation definition and will report the corrected values consistently throughout the paper.
We sincerely appreciate your constructive feedback. We believe these clarifications and additional evaluations strengthen the rigor and practical relevance of our work, and respectfully ask you to reconsider your assessment.