- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MathTypes, a typed language for mathematical reasoning that enables deterministic step-level checking through a registry of SymPy-backed operations. It combines natural-language explanations with machine-checkable steps and uses a compile-verify-correct loop to provide feedback to LLMs. The authors also propose an automated pipeline for constructing exam-specific type registries from existing mathematical solutions.
Experiments on 171 difficult problems from six mathematics examinations show that MathTypes outperforms six baseline methods across all seven evaluated models, improving accuracy by 9.1--17.6 percentage points over the strongest baselines. Ablation studies suggest that step-level diagnostic feedback is a major contributor to these improvements.
Critical strengths
- MathTypes offers an interesting alternative to informal LLM-based verification and fully formal proof assistants. Its registry of mathematical operations and deterministic feedback provide a practical way to check parts of mathematical reasoning without requiring complete formalization.
- The method achieves accuracy improvements across multiple models and challenging mathematical examinations. The ablation studies also provide useful evidence that detailed verification feedback contributes to the improvements.
Critical weaknesses
(1) On the notion of "type". The paper motivates MathTypes by referring to type theory, including the Curry-Howard correspondence and Martin-Lof type theory. However, the "types" in MathTypes appear to be registered mathematical operations with predefined input signatures and verification procedures, rather than types in the usual type-theoretic sense. The system seems to perform function-signature checking and deterministic evaluation rather than type checking in the sense of a formal type system.
Since the notion of "type" is central to the proposed framework, its intended meaning and relationship to established type-theoretic approaches should be clarified.
(2) Although MathTypes checks registered operations deterministically, this does not guarantee the correctness of the overall mathematical reasoning. Natural-language steps remain unverified, and locally correct operations do not necessarily establish the validity of the inferences connecting them. Moreover, some theorem-verification procedures are generated by LLMs, and the authors report that certain procedures fail to reject false claims.
(3) On Readability of proofs. The paper emphasizes that MathTypes produces more readable mathematical reasoning than proof assistants. However, no direct human evaluation of readability, comprehensibility, or explanatory value is provided. Retaining natural-language explanations does not necessarily make a mathematical argument easier to understand or more informative.
Questions for authors
-
What exactly is meant by a "type" in MathTypes? How does this notion relate to types in modern type theories, such as Martin-Lof type theory?
-
Does a PASS verdict guarantee the correctness of a mathematical reasoning step, even when the step depends on previous steps or on an LLM-generated theorem-verification procedure? If not, what exactly does a PASS verdict guarantee?
-
Do the authors have evidence that MathTypes solutions are more readable or comprehensible to human readers than solutions produced using existing formal verification approaches?
4 comments
Thank you for the detailed responses and additional experiments.
Regarding W1, I appreciate the clarification that the notion of "type" in MathTypes is not directly related to type theory. However, the proposed connection to HOL and LCF remains unclear. In particular, LCF's use of an abstract theorem type and trusted inference rules is fundamentally different from, and has little to do with, MathTypes' registry of mathematical operations and verification procedures. I encourage the authors to clarify this distinction rather than introduce potentially misleading analogies.
Also, "Higher-Order Logic (HOF)" appears to be a typo for "Higher-Order Logic (HOL)."
Regarding W2, I appreciate the clarification that a PASS verdict does not guarantee the correctness of the overall reasoning. However, the statement that PASS ensures a step contains "sound logic" seems too strong. Appendix H reports that the induced theorem-verification procedures accept approximately 85–88% of deliberately falsified claims. This suggests that a PASS verdict guarantees only acceptance by the corresponding verification procedure, not the mathematical correctness of the claim. The additional results concerning the association between PASS and final-answer accuracy are useful, but they do not resolve this issue.
Regarding W3, I appreciate the additional readability experiment. However, the study involves only four participants with different levels of familiarity with the evaluated languages, including one extensively trained in MathTypes. This raises concerns about the reliability and generalizability of the results. I do not think such a small study provides sufficient evidence to support the claim that MathTypes is more readable than existing approaches.
Overall, I appreciate the authors' efforts to address my concerns. However, my main concerns remain, particularly regarding the interpretation and reliability of PASS verdicts. I therefore maintain my original rating.
Author Response to Reviewer CfZj’s comments
We thank the reviewer for their comments on our rebuttal. We have a few clarifications to make.
W2 (PASS verdicts)
We agree that a PASS does not guarantee mathematical correctness, just as expecting correct programmatic logic from a step that compiles is unreasonable. We will replace the term “sound logic” to reflect this clearly.
Regarding the reliability finding the reviewer notes, we rechecked the 85.4-88.0% figure in Appendix H and found that the false-acceptance rate experiment only perturbs the claimed_result field everywhere, including theorems. However, theorems never check the claimed_result field; they only check conclusion_claimed. Thus, the 85.4–88.0% figure falsely deflates theorems’ soundness.
We reran the experiment by perturbing just conclusion_claimed fields for theorems. Only 1.6-4.2% of theorems actually accept a falsified claim. We have updated Appendix H and present the revised table with the updated caption here:
Table 15: False-acceptance rate (false accepts/scored trials) when each passing step's claim is replaced by a wrong value. The claim is claimed_result, except on apply_theorem steps, whose only claim is conclusion_claimed. Scored trials include ERROR outcomes (272, 126, and 52 for the three sources).
| Verification strategy | Code edits | GLM 5.3 Flash | Muse Spark 1.3 |
|---|---|---|---|
| Fixed SymPy strategies, pooled | 0.84% (66 / 7,828) | 1.31% (31 / 2,362) | 0.80% (18 / 2,242) |
— worst single strategy (verify_condition_implication) | 100% (20 / 20) | 100% (4 / 4) | 100% (4 / 4) |
| — next worst | sympy_solve 3.1% | sympy_solve 9.8% | verify_add_equations 6.2% |
| — strategies with no false accepts | 14 of 19 | 15 of 21 | 15 of 21 |
theorem_lookup (induced theorem bodies) | 4.2% (35 / 827) | 1.6% (4 / 252) | 2.0% (5 / 249) |
| All strategies, pooled | 1.17% (101 / 8,655) | 1.34% (35 / 2,614) | 0.92% (23 / 2,491) |
We reviewed which theorems accept falsified steps, and they localize to just two: zero_derivative_implies_constant which fails to assert a numeric range, so a perturbed numerical value by a frontier LLM gets accepted; and p_series_convergence, which matches a convergence status by keyword matching. These theorems can easily be fixed with a manual review to drop the false-acceptance rate to 0%.
So, the takeaway is that:
- PASS means the logic was acceptable to a fixed, auditable procedure
- The measured failure rate of this procedure is ~1% across types and theorems, pooled
We will state this clearly in the main paper and regret the error in the submitted appendix.
W1 (LCF/HOL)
We agree that analogies with LCF, HOL, or Type Theory are not foolproof. As stated, we will make it clear in Section 2.2 that Type Theory was cited as early background work leading up to modern techniques such as HERMES and SymCode. Our main point of comparison with LCF was its use of a small, fixed set of trusted inference rules, just as MathTypes’ steps are checked by one of 31 fixed verification strategies (Appendix I, Table 16). The analogy is not like-for-like, as MathTypes is not an extension of LCF, and we thank the reviewer for cautioning us in this regard.
We also thank the reviewer for noticing the typo in our rebuttal, which we have corrected.
W3 (Readability)
We agree that four participants cannot support a general claim. Yet, we would like to surface two observations:
- All four participants independently produce the same ranking (CoT > MathTypes > SymCode > Lean), so the ranking holds even with the participant with MathTypes syntax training held out. Further, we would also like to note that we included a participant with Lean training and professional Python programmers, so one participant with MathTypes training is reasonable.
- Binary inter-annotator agreement was 66% (moderate).
The readability study is a pilot, as allowed by the rebuttal’s timeline. We are currently recruiting a larger annotator pool, including subject experts in each tested exam, such as JEE Advanced, and intend to update the study by the camera-ready deadline.
Thank you for engaging with our paper. Your comments have made our work stronger. We kindly request that you reconsider your stance based on this clarification.
Thank you for the clarification and responses. I will reply specifically to W1.
Having a small, fixed set of trusted inference rules is a general property of many logical systems, not a distinctive feature of type theory. If that is the main point of comparison, then the connection to type theory in the theoretical sense is absent. Relying on a superficial overlap in the word "type" to associate lightweight schema constraints with foundational frameworks like MLTT or HOL is misleading, even as historical background in Section 2.2. I strongly recommend removing these references.
Author Response to Reviewer CfZj
W1 / Q1
We intended to cite Curry-Howard and Martin-Löf as background on verification systems in Section 2.2 and would like to clarify that the “type registry” does not directly correlate to type theory. Instead, MathTypes’ types are closer to Mike Gordon’s Higher-Order Logic (HOL), which is commonly used in Isabelle, and Robin Milner’s Logic for Computable Functions (LCF). The link here is that both use types as rules. LCF, for example, uses theorems as abstract data types that define a small set of inference rules, similar to our design.
We would also like to clarify that MathTypes doesn’t simply check if a function’s signature was used correctly. Each step does three things:
Steps #1 and #3 are precisely what separates it from existing programming languages, allows MathTypes to give high-quality step-level feedback, and makes it lightweight. Therefore, isolating just the compiler’s function (step #2) is not ideal.
Section 2.2 ends with the negative effects of over-formalization on high school and college-level maths, which we precisely aim to solve. We will add this extra qualification to the write-up to make the difference clear.
W2 / Q2
We thank the reviewer for raising the point and would be happy to clarify. Just as a line of Python code that compiles does not guarantee correct programming, a PASS is not a guarantee of absolutely correct mathematical reasoning required at the correct step. It ensures that the step contains sound logic and fits into the theorems, formulas, or conjectures the model uses.
Given that no programming language ensures absolutely correct programming logic or satisfaction of a problem statement, we do not expect the same from MathTypes. Empirically, we show the positive impact of MathTypes verification in Table 1.
Table 1. Empirical analysis of programs (accuracy) of programs that PASS vs. those that fail MathTypes compilation.
A program that PASSes is about 21 points more likely to have the right answer than a rejected one, and the gap holds on every tested exam (+14 to +35 points). Further, we also note that a majority of wrong-answer outcomes that still contain a PASS are linked to a setup derived from a former informal step.
W3 / Q3
We thank the reviewer for raising this point. We conducted a human subject study to test the readability of our language. We showed annotators code snippets or full programs written in MathTypes, Lean, or the SymPy-based SymCode blindly. The task defined the problem, explained the solution, explained what the steps do, and asked the annotator to choose the more readable version. The annotator pool is varied:
We add Chain-of-Thought as a control to this experiment. Being pure natural language, it is expected to be the most readable. We calculated Elo scores based on Bradley-Terry on the preference annotations. The results are summarized in Table 2.
CoT tops the readability test, as expected, with MathTypes following closely behind. Both SymCode and Lean trail MT by ~300 points, proving the readability argument we make in the paper.
Table 2. Elo scores from the readability analysis, centered at 1500.
We will add this human study to the paper to back up our claim with full details of the task design, annotator recruitment, and payment procedure.
We hope these new experimental results and discussion address your concerns about the submission. We request that you consider adjusting the score. Thank you for taking the time to evaluate our work.