acceptodds
Under review as a conference paper at ICLR 2027

Equation10K: Teacher-Style Error Diagnosis in Handwritten Arithmetic Solutions

Abstract

Vision-language models (VLMs) are increasingly considered for educational assessment, yet existing mathematical benchmarks mostly ask models to solve a problem, recognize handwritten expressions, or locate a single error. A teacher must do more: read a multi-line handwritten solution, verify each transformation, identify the earliest misconception, distinguish a new mistake from downstream error propagation, and remain reliable across diverse arithmetic structures and handwriting styles. We introduce \dataset, a procedurally generated benchmark for fine-grained diagnosis of handwritten step-by-step arithmetic. The benchmark contains 10,000 solution images spanning 10 problem families, five difficulty levels, 12 handwriting styles, and 4,272 operator-structure templates. Each solution is paired with executable step traces and labels for correctness, first-error position, and five diagnostic states: wrong value, wrong operator, incorrect order of operations, calculation error, and propagated error. We evaluate representative proprietary and open VLMs using step correctness, error-type accuracy, joint step accuracy, exact solution grading, and first-error localization. Our experiments reveal two consistent patterns: joint diagnostic accuracy decreases at higher structural difficulty levels, and performance improves when handwritten inputs are replaced by cleaner printed or oracle-text representations, while non-trivial errors remain even under oracle text. These results indicate that both visual transcription and step-level process verification remain challenging for current VLMs. \dataset provides a scalable testbed for studying multimodal process supervision and reliable AI-assisted grading. We open source the project at https://anonymous.4open.science/r/Equation10K-61BE.

Primary area:
datasets and benchmarks
Keywords:
Handwritten Mathematics, Mathematical Reasoning, Step-by-Step Grading, Error Diagnosis, Error Propagation, Multimodal Benchmark, Educational AI

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_oaGtOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes EQUATION10K, a procedurally generated benchmark for diagnosing handwritten, step-by-step arithmetic solutions. It evaluates whether VLMs can not only recognize steps but also verify transformations, pinpoint the earliest error, and model error propagation, addressing a gap left by benchmarks that focus on final correctness or coarse error detection. Experiments show that performance on joint diagnosis drops with structural complexity and improves with cleaner text inputs, yet substantial diagnostic mistakes persist even with oracle text, suggesting hard limitations in visual transcription and process-level verification.

Critical strengths

The paper generates correct reasoning traces in an executable format, then introduces an error at a chosen step and labels later steps as PROPAGATION if they follow from that wrong state. This setup lets the evaluation check whether a model can identify where the first mistake occurs and whether it keeps making consistent downstream judgments, instead of only scoring the final answer. It also makes the benchmark easier to reproduce because the ground-truth process is defined and testable.

Critical weaknesses
  1. The benchmark draws on synthetic handwriting in 12 font styles, yet it never checks whether the results carry over to real student work. More fundamentally, it assumes a single linear path from problem to answer. Real students, though, skip steps or rework earlier lines out of order. The framework labels any locally valid transformation after an error as PROPAGATION, which presumes students keep a coherent thread through corrupted states, an assumption classroom data does not support. Its apparent VLM reasoning failures may therefore be artifacts of its own rigid design.
  2. The five error categories underpin all evaluation metrics, yet they lack grounding in how teachers actually distinguish errors. The paper acknowledges the taxonomy is model-based rather than derived from authentic classroom data. No teacher study validates whether educators can reliably apply these categories or whether real student mistakes even conform to these boundaries. Without inter-rater reliability evidence, claiming JSA and FEIA measure genuine diagnostic capability remains speculative.
  3. Furthermore, although ETA measures overall error-type classification accuracy, the paper provides no per-category breakdown showing how VLMs perform on each error type. Given that order-of-operations errors are structurally rarer and may require different reasoning patterns than value or calculation errors, disaggregated accuracy by error category would clarify where diagnostic failures originate. Without this analysis, claims about fine-grained diagnostic capability remain coarse.
Questions for authors

The paper evaluates the taxonomy's diagnostic capability on synthetic data. What would be the minimal evidence that would support claims about real-world pedagogical validity? What are the key obstacles to collecting such evidence, and how might they be addressed?

2 comments

Dear Reviewer,

Thank you for your thoughtful review. We respond to each point below.

[W1a] Synthetic handwriting; no validation on real student work.

We agree that the generalizability of synthetic handwriting is a limitation, already noted in Section 7, and will make it more prominent. We also add an empirical validation: volunteers hand-copy benchmark traces on paper, preserving the original problems and step text while introducing natural pen variation. We evaluate the same models on these scans and report the Spearman rank correlation between synthetic- and handwritten-data leaderboards, together with absolute score differences.

[W1b] The single linear path and PROPAGATION as a framework artifact.

Equation10K tests a necessary condition: a grader unable to reliably verify clean sequences of transitions cannot be expected to handle skipped, reordered, or crossed-out work. We will clarify this scope and avoid presenting the benchmark as representative of all student work.

The PROPAGATION distinction also reflects established educational practice. Teachers routinely award credit for work that is correct given an earlier mistake, commonly termed "follow-through" or "error carried forward." We acknowledge that students may also make multiple independent errors and therefore add a controlled probe set containing a second independent error after the first. We will also clarify that each solution in the main split contains exactly one primary error.

[W2] The five categories lack teacher grounding and inter-rater reliability.

The four primary categories (wrong value, wrong operator, order of operations, and calculation error) reflect recurring computational error patterns in mathematics-education literature. However, we agree that this does not establish consistent application by teachers.

We therefore (i) clarify that JSA and FEIA measure agreement with taxonomy-based diagnostic labels rather than validated pedagogical diagnosis; and (ii) conduct a blinded annotation audit in which teachers or annotators label a stratified sample of solutions from images alone. We report inter-annotator agreement (Fleiss' kappa), agreement with generator labels, and commonly confused label pairs.

Because some injected causes are not identifiable from the displayed trace (e.g., an operator substitution when only the resulting value is shown), we also report scores that accept indistinguishable alternative labels alongside strict scores.

[W3] No per-category breakdown of ETA.

We address this gap by reporting precision, recall, F1, and confusion matrices for all five labels across models, with ORDER analyzed separately due to its lower structural frequency (Section 3.3). We also distinguish primary-error classification accuracy from propagation recall, since PROPAGATION accounts for 65.8% of erroneous steps (9,611/14,611) and substantially influences aggregate ETA.

[Question] Minimal evidence for real-world pedagogical validity and obstacles.

We consider three types of evidence essential: (i) teacher agreement on synthetic items; (ii) teacher-annotated, de-identified real student solutions using the same taxonomy; and (iii) rank agreement between synthetic and real-data leaderboards. Key obstacles include student privacy and consent, annotation costs, and inherent label ambiguity.

In this revision, we address (i) through the blinded annotation study and (iii) through the volunteer hand-copy study, which introduces natural handwriting variation while preserving exact labels. For (ii), we propose collaborating with instructors under institutional review and releasing only de-identified data, leaving this as future work.

Correction.

During revision, we identified that Table 3 reported unweighted means across difficulty levels rather than the step-level micro-averages defined in Appendix B. We also corrected the Qwen2.5-VL-7B handwritten JSA in Figure 3 from 0.157 to 0.147. We recompute all metrics using the stated aggregation definition and will report the corrected values consistently throughout the paper.

We sincerely appreciate your constructive feedback. We believe these clarifications and additional evaluations strengthen the rigor and practical relevance of our work, and respectfully ask you to reconsider your assessment.

Thank you for the response. I acknowledge the added experiments and revisions; several of my concerns have been partially addressed. However, the central validity concern remains only partially resolved, so I have raised my score by one point to reflect the partial progress.

Reviewer_Vk6pOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Equation10K is a procedurally generated benchmark for diagnosing multi-line handwritten arithmetic solutions. It contains 10,000 images and 37,048 displayed steps across ten problem families, five difficulty levels, and twelve handwriting fonts. An exact rational-arithmetic AST solver constructs traces, injects a primary error, and labels subsequent propagation. Models predict correctness for each line and, for incorrect lines, one of five diagnostic error types. The evaluation reports binary and joint step accuracy, error-type accuracy, exact solution grading, and first-error localization for ten VLMs, with analyses by difficulty and problem family and comparisons against printed images and oracle text. The intended contribution is a controlled, scalable resource for multimodal process verification.

Critical strengths
  1. A useful combination of controlled generation and step-level provenance. Separating expression generation, exact execution, error injection, and rendering makes the benchmark reproducible in principle and supports controlled changes to mathematical structure and visual presentation. Its scale and explicit distinction between a primary mistake and its consequences complement existing handwritten-grading and text-based process-verification resources. This is a meaningful benchmark contribution without requiring a new model.
  2. Evaluation beyond final-answer accuracy. The paper evaluates ten models and reports step-level, first-error, and whole-solution metrics, with difficulty and family breakdowns. The matched handwritten/printed comparison is useful, and the oracle-text experiment probes performance when symbolic steps are available. The manuscript also explicitly acknowledges the limitations of synthetic fonts and injected mistakes for real classrooms. These choices make the resource more informative than a single aggregate grading score.
Critical weaknesses
  1. Generator provenance does not establish identifiable diagnostic labels. The benchmark treats the injected cause as the uniquely correct error type, although that cause is not always observable in the displayed trace. In Figure 4(c), 7 times 5 becomes 1.4 and is labeled OPERATOR, but a division symbol is never displayed. An observer could reasonably describe the same visible transition as a multiplication calculation error. Figure 4(a), where 1.5 minus 5.5 becomes 7, presents the same issue. These are not claims that the generator's internal record is wrong; they show that exact provenance alone does not justify a single diagnostic label from the model's input. Since ETA, JSA, and SEM require exact type agreement, this ambiguity can penalize defensible diagnoses. A blinded annotation audit or explicit treatment of admissible alternative labels is needed to establish the validity of the central task. The propagation task is also narrower than the teacher-style framing suggests. Section 3.2 injects one primary error and then continues with locally valid steps. After finding the first error, a system can label the remaining suffix PROPAGATION without testing for later independent errors. Propagation accounts for 9,611 of the 14,611 erroneous steps, so always choosing it already gives 65.8% ETA. The paper acknowledges imbalance, but does not report primary-type performance or a first-error-plus-suffix baseline. Such controls, or examples with multiple independent errors, would establish what propagation diagnosis adds beyond first-error detection.
  2. The scoring aggregation and modality comparison require reconciliation. Appendix B defines JSA as a micro-average over all displayed steps, but the Table 3 values match the unweighted averages of the five difficulty scores in Table 4. For GPT-5.6 Sol, that average is exactly 0.780. This is difficult to reconcile with the stated step weighting: Section 3.3 implies 2,000 solutions per level and gives mean lengths of 2.70 at Level 1 and 4.70 at Level 5. Applying Appendix B's step-level definition to the level-specific scores, even assigning every remaining step the highest Level 2-4 score, 0.811, yields an overall micro-JSA below 0.774 after allowing for rounding, rather than the reported 0.780. The authors should clarify the actual aggregation and recompute the tables consistently with the stated metric, or revise the definition. This matters for the benchmark's primary score and model comparisons. The oracle-text comparison introduces an additional protocol change. Figure 6 asks image models to identify and grade all visible steps in one response, whereas the text prompt grades only a supplied CURRENT step. Thus the reported gain mixes transcription removal with known segmentation and a different prediction task. Section 6.4 appropriately cautions that oracle text is not a perfect single-factor control, but a matched whole-trace text condition is still needed before attributing the gap to visual transcription or interpreting residual errors specifically as reasoning failures.
Questions for authors
  1. What observable rule uniquely distinguishes OPERATOR from CALCULATION in the transitions in Figure 4(a,c)? How often do blinded annotators disagree with the injected label, and how do scores change if reasonable alternative labels are accepted or the altered operation is explicitly displayed?
  2. Are the main scores micro-averaged over steps or macro-averaged over difficulty levels? Please provide per-level step counts and raw correct counts, reconcile Tables 3-4 with Appendix B, and specify how missing or extra predicted lines, malformed outputs, and inconsistent correctness/type pairs are aligned and scored.
  3. Does the oracle-text gain persist when the same full solution, prompt scope, and output structure are used for both modalities? Please also reconcile the handwritten Qwen2.5-VL JSA of 0.147 in Table 3 with 0.157 in Figure 3, including any difference in evaluated samples.
  4. How much does the benchmark distinguish full diagnostic systems from a first-error detector followed by automatic PROPAGATION labels? Primary-error class scores and a controlled test with a later independent error would materially clarify the claimed diagnostic capability.
3 comments

Dear Reviewer,

Thank you for your exceptionally careful review. Your comments helped us identify several reporting errors and methodological limitations, which we acknowledge and address below.

CORRECTIONS

[C1] Aggregation (Tables 3–4 vs. Appendix B).

You are correct. Although Appendix B defines SCA and JSA as micro-averages over steps, Table 3 mistakenly reports unweighted means across five levels. For GPT-5.6-Sol, the level-macro averages are JSA = 0.780 and SCA = 0.831. This inconsistency affects all ten models. Given the unequal level sizes (approximately 5,400 steps at Level 1 and 9,400 at Level 5), the estimated step-weighted JSA is approximately 0.77, consistent with your bound of < 0.774.

We retain micro-averaging as the primary metric, recompute all five metrics for all models using per-step logs, and report per-level step counts, raw correct counts, and macro-averages separately.

[C2] Qwen2.5-VL-7B handwritten JSA.

Tables 3–4 report 0.147 (per-level mean: 0.1468), whereas Figure 3 and Section 6.4 report 0.157. The former is correct; thus, the printed-image gain is +0.050 rather than +0.040. Figure 3 will be regenerated following C1.

[C3] Claims in the Conclusion.

We agree that the claims identifying "order-of-operations reasoning" and "propagation tracking" as specific bottlenecks are not sufficiently supported. We will remove them and replace them with findings from the per-class analysis.

WEAKNESSES

[W1] Provenance does not guarantee identifiable labels (Figure 4a,c).

We agree that exact provenance does not necessarily make an error label identifiable from the input. For example, 7×5 → 1.4 and 1.5−5.5 → 7 could plausibly be interpreted as calculation errors rather than operator substitutions. Strict matching may therefore penalize defensible diagnoses.

We will address this through three additions:

  1. Observability-aware evaluation: Automatically identify admissible causes for each transition and report strict and lenient scores, together with an observability flag.
  2. Blinded annotation audit: Annotate a stratified image-only sample and report agreement with injected labels, Fleiss' kappa, and confused label pairs.
  3. Explicit-operation subset: Regenerate a subset displaying altered operations, making the underlying causes observable.

[W1b] PROPAGATION is narrower than "teacher-style" prediction.

We verified your calculation: always predicting PROPAGATION achieves 65.8% ETA (9,611/14,611). Three models fall below this baseline, while the strongest exceed it by 21–24 percentage points. Nevertheless, aggregate ETA conflates primary-error detection and propagation labeling, leaving the latter's contribution unclear.

We will therefore add:

  • Per-class precision, recall, F1, and confusion matrices, separating primary-error classification from propagation recall.
  • A first-error-plus-suffix baseline using an oracle first-error detector followed by automatic PROPAGATION labels.
  • An independent-second-error probe, where the suffix heuristic necessarily fails.

We will also clarify that each solution in the main split contains exactly one primary error (Section 3.2).

[W2] Aggregation and scoring details.

Beyond C1, Appendix B will specify step-matching rules and the treatment of missing lines, extra lines, malformed JSON, and inconsistent correctness/type predictions, including their per-model frequencies.

[W3] The oracle-text comparison changes the protocol.

We agree that the original conditions are not matched: Figure 6b evaluates a supplied current step with known segmentation, whereas Figure 6a requires full-trace transcription and segmentation. Thus, the reported gain conflates transcription removal with segmentation and output-format differences.

We will add a matched whole-trace text condition using the same complete solution, instructions, output schema, and single-response evaluation as the image condition. We will also clarify the conclusions supported by each comparison and avoid attributing residual oracle-text errors solely to reasoning failures without evidence from the matched condition.

QUESTIONS

Q1 (OPERATOR vs. CALCULATION). Without a displayed operation, these labels may be indistinguishable. We address this through observability-aware scoring, blinded annotation, and the explicit-operation subset (W1).

Q2 (Micro vs. macro, counts, and alignment). We correct the aggregation error and will report micro- and macro-averages separately, alongside per-level counts and alignment rules (C1, W2).

Q3 (Oracle-text gain; 0.147 vs. 0.157). We correct handwritten JSA to 0.147 and introduce a matched whole-trace text comparison (C2, W3).

Q4 (Full system vs. first-error detector plus PROPAGATION). We will report primary-error classification results, compare against the first-error-plus-suffix baseline, and evaluate independent subsequent errors (W1b).

Dear Reviewer, I hope this message finds you well. As the discussion period draws to a close, I would like to confirm whether we have satisfactorily addressed all of your concerns. Please let us know if there are any further points or feedback you would like to discuss. Your insights are invaluable to us, and we are keen to resolve any remaining issues to improve our work. Thank you for the time and effort you have dedicated to reviewing our paper.

Thank you for the response.I appreciate the authors’ detailed responses and their transparent corrections of the aggregation, the 0.147/0.157 discrepancy and the overstated claims in the conclusions. However, my main concerns are whether the error labels are identifiable, what propagation diagnosis adds beyond first-error detection, and the unmatched oracle-text comparison. These are addressed only through planned additions rather than results, so I shall keep my score.

Reviewer_wPcxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes EQUATION10K, a benchmark for evaluating VLMs for fine-grained diagnosis of handwritten step-by-step arithmetic. The authors introduce the dataset construction pipeline from expression template with different difficulty levels, then they inject an error for teacher-style step-by-step annotation. As a result, the newly created benchmark includes 10,000 solution images spanning 10 problem families, 5 difficulty levels, and 12 handwriting styles. In the experiments with several proprietary and open VLMs, it shows that open and small model performs worse than large and proprietary models, and step-by-step accuracy is still worse and thus more difficult. In the ablation study with different input presentations, printed out image and oracle text further improve the performance, but still the challenge of grading step-by-step remains.

Critical strengths
  • The paper address a gap in existing benchmarks by focusing on teacher style step-by-step grading.
  • The presented analysis of different input presentations provides insights regarding burdens for VLMs for the task.
  • Writing is easy to follow
Critical weaknesses
  • The main experimental evaluation and analysis provide no surprise. First, it is expected that, with more difficulty, performance of VLMs may decrease. Second, the performance of frontier models such as GPT-5.6 sol is already quite good on the benchmark, and Qwen3.8-Max and Qwen3.8-Flash-Next also perform fairly well. Therefore, the newly introduced benchmark may not clearly reveal the limitations of them.
  • Even though this paper provides comparison with different VLMs, as a benchmark paper, it is expected to propose a strong baseline proposed by the paper. But it was missing, and thus the paper has less contribution.
  • The benchmark has limitations. First, the dataset was generated, so its validity was not confirmed. It is desirable to check its quality by human or VLM-as-judge. Second, unlike prior work such as FERMAT that includes not only Arithmetic but also Algebra, Mensuration, Geometry, Probability, Statistics, Trigonometry and Calculus, the benchmark only focuses on arithmetic. Therefore, its significance compared to existing benchmarks is rather limited.
Questions for authors
  • Can you clarify why you focus on arithmetic and why it is more important than others like Algebra, Statistics and Calculus?

Typo -#242: rimary error -> Primary error

4 comments

Dear Reviewer , thank you for the careful review. We respond to each point below.

[W1] "No surprise"; frontier models already perform well, so the benchmark may not reveal their limitations.

We would like to point to what the results show beyond "harder is worse.

  1. Frontier models are far from reliable. The best model reaches SEM = 0.613 and FEIA = 0.693. Roughly 39% of solutions contain at least one wrongly graded step, and for roughly 31% the first error is not correctly localized. This is arithmetic that a calculator verifies instantly, and the prompt states the labeling rules explicitly.
  2. The difficulty effect is not the expected one. If difficulty simply degraded everything, step correctness should fall with level. It does not: GPT-5.6-Sol's SCA stays within 0.812-0.844 across all five levels (Appendix G), while its JSA falls from 0.822 to 0.655. The failure is concentrated in diagnosis (typing the error and separating new from propagated errors), not in noticing that a line is wrong. A single accuracy score would hide this.
  3. Rankings change with difficulty (Section 6.2): the best model at Level 1 is not the best at Level 5.
  4. The benchmark separates perception from reasoning. Oracle text lifts GPT-5.6-Sol to 0.929 JSA, so a diagnostic gap remains once transcription is removed, while weaker models gain up to +0.43.

Equation10K is a diagnostic benchmark with controlled factors, not one designed to push every frontier score toward zero. We will state this more explicitly in the introduction.

A correction we made while preparing this response: following a check prompted by another reviewer, we found that the SCA/JSA values in Table 3 were unweighted means over the five difficulty levels, whereas Appendix B defines step-level micro-averages. We are recomputing all metrics under the stated definition . Our rough estimate is that the headline JSA values shift by about one point, so the conclusions above are unaffected. We also corrected one value in Figure 3 (Qwen2.5-VL-7B handwritten JSA: 0.147, not 0.157).

[W2] A benchmark paper should propose a strong baseline.

We agree that stronger reference points make the benchmark more useful. Because the benchmark is procedurally generated, we can provide two without touching the test set:

  1. Transcribe-then-verify pipeline. A VLM transcribes the visible lines, and the exact rational-arithmetic AST executor from our generator verifies each transition and assigns labels deterministically. This shows how much of the gap is transcription.
  2. Supervised fine-tuning on a disjoint training split (different seeds and held-out expression templates). Both will be released with the benchmark. Their purpose is to show what is achievable and where headroom remains, not to claim a new method.

[W3a] The dataset is generated and its validity was not confirmed.

Correctness and error provenance come from an executable trace, so step correctness can be checked mechanically. We re-executed all 37,048 displayed steps with an independent rational-arithmetic checker . For what a mechanical check cannot cover (legibility of the rendering, and whether the injected label is the one a teacher would give). [W3b and Question] Why arithmetic, and why not algebra, statistics, calculus as in FERMAT?

We do not claim arithmetic is more important than the other areas. We chose it because it is the setting where all five properties listed in our introduction can hold at once: every intermediate state has an exact, executable ground truth, so a later line can be provably locally valid but globally wrong; difficulty can be enumerated (4,272 templates); and labels scale without manual annotation. FERMAT's curated solutions cover broader topics but, as Table 1 shows, have no propagation labels and no procedural scale, so the two benchmarks are complementary. A grader that cannot reliably verify arithmetic transitions will also not do so inside an algebra or calculus derivation, where each step reduces to arithmetic and symbolic rewriting, so our results are a lower bound for broader settings. The same construction extends in principle to algebra via an AST rewriting solver with rule-level error injection. We will add this discussion and state the scope limit more prominently in the abstract and Section 7.

We hope these additions address your concerns and would be grateful if you would reconsider your assessment.

Dear Reviewer, I hope this message finds you well. As the discussion period draws to a close, I would like to confirm whether we have satisfactorily addressed all of your concerns. Please let us know if there are any further points or feedback you would like to discuss. Your insights are invaluable to us, and we are keen to resolve any remaining issues to improve our work. Thank you for the time and effort you have dedicated to reviewing our paper.

I thank the authors for providing the response. Even though the response partially addressed my concerns, my two concerns still remain:

  1. A new baseline model: the authors suggested the ideas of reasonable baselines and I appreciate, but the effectivenesses of them were not demonstrated with experimental result. Therefore, the contribution of the paper is still rather weak as a benchmark paper.
  2. Scope of the topics covered in the benchmark: I agree that arithmetic is a natural choice for step level verification. However, e.g., algebra or calculus still may be relevant to have solution steps that one can check its validity. Therefore, the weakness of the benchmark still persists.

Therefore, I maintain my original rating.

Thank you for your reply.

It is undeniable that calculus and higher algebra are indeed much more difficult than these simple equations and can better test the grading capabilities of vision large models. However, according to the current experimental results, vision large models still have considerable room for improvement in grading equations solved step by step. In addition, the handwriting we generate resembles that of young children, which further increases the difficulty of evaluating vision large models’ ability to grade step-by-step equation solutions. From the experimental results, even the strongest models, such as GPT-5.6 Sol, still rely on OCR followed by programmatic computation tools and then judge whether the answer is correct based on the computation results, rather than identifying the specific type of answer error as humans do. In this sense, the Equation10K benchmark we constructed is still of great significance.

We are also attempting to construct a handwritten response benchmark for calculus or higher algebra, but progress is not smooth at present. When rendering handwritten responses, we encounter difficulties in rendering symbols such as the integral symbol (∫), summation symbol (∑), and partial derivative symbol (∂), and the real handwritten response samples we have collected are limited. Perhaps in the near future we will be able to construct a handwritten response benchmark for calculus or higher algebra. We welcome attention to our work.

Your positive feedback serves as the motivation for us to complete the development of this benchmark( calculus, higher algebra).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.