acceptodds
Under review as a conference paper at ICLR 2027

MathTypes: Towards Deterministic Step-level Checking for Reasoning in Math Exams

Abstract

Mathematical reasoning with Large Language Models (LLMs) mostly remains final-answer verifiable, with existing step-level checks being either probabilistic (LLM judges, process reward models), unstructured (pairing a model with a CAS), or too heavyweight for exam-style reasoning (proof assistants). We present MathTypes, a typed language in which reasoning steps are calls into a registry of SymPy-backed operations carrying a claimed result that is evaluated deterministically, and a compile-verify-correct evaluator loop assay built on it. Our method makes models select from a fixed type registry instead of authoring verification in free-text, with unverifiable steps left to escape hatches, thereby providing step-level feedback on natural-language mathematical reasoning akin to interpreted programming languages, while ensuring reasoning chains are more readable than proof assistants. On difficulty-filtered problems from six competitive exams, MathTypes is the most accurate method for all seven models tested, beating the strongest of six baselines by 9.1–17.6 points and extending the accuracy-cost Pareto frontier. Detailed ablations show that the verifier's step-level diagnostics, not registry knowledge alone, drive the gain; that most of it lands within two repair passes; and that the required type registry can be induced cheaply. Through MathTypes, we establish the promise of readable math reasoning whose typed steps are machine-checked.

Primary area:
neurosymbolic & hybrid AI systems (physics-informed, logic & formal reasoning, etc.)
Keywords:
math reasoning, formal verification, LLM reasoning, process evaluators

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_CfZjOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MathTypes, a typed language for mathematical reasoning that enables deterministic step-level checking through a registry of SymPy-backed operations. It combines natural-language explanations with machine-checkable steps and uses a compile-verify-correct loop to provide feedback to LLMs. The authors also propose an automated pipeline for constructing exam-specific type registries from existing mathematical solutions.

Experiments on 171 difficult problems from six mathematics examinations show that MathTypes outperforms six baseline methods across all seven evaluated models, improving accuracy by 9.1--17.6 percentage points over the strongest baselines. Ablation studies suggest that step-level diagnostic feedback is a major contributor to these improvements.

Critical strengths
  1. MathTypes offers an interesting alternative to informal LLM-based verification and fully formal proof assistants. Its registry of mathematical operations and deterministic feedback provide a practical way to check parts of mathematical reasoning without requiring complete formalization.
  2. The method achieves accuracy improvements across multiple models and challenging mathematical examinations. The ablation studies also provide useful evidence that detailed verification feedback contributes to the improvements.
Critical weaknesses

(1) On the notion of "type". The paper motivates MathTypes by referring to type theory, including the Curry-Howard correspondence and Martin-Lof type theory. However, the "types" in MathTypes appear to be registered mathematical operations with predefined input signatures and verification procedures, rather than types in the usual type-theoretic sense. The system seems to perform function-signature checking and deterministic evaluation rather than type checking in the sense of a formal type system.

Since the notion of "type" is central to the proposed framework, its intended meaning and relationship to established type-theoretic approaches should be clarified.

(2) Although MathTypes checks registered operations deterministically, this does not guarantee the correctness of the overall mathematical reasoning. Natural-language steps remain unverified, and locally correct operations do not necessarily establish the validity of the inferences connecting them. Moreover, some theorem-verification procedures are generated by LLMs, and the authors report that certain procedures fail to reject false claims.

(3) On Readability of proofs. The paper emphasizes that MathTypes produces more readable mathematical reasoning than proof assistants. However, no direct human evaluation of readability, comprehensibility, or explanatory value is provided. Retaining natural-language explanations does not necessarily make a mathematical argument easier to understand or more informative.

Questions for authors
  1. What exactly is meant by a "type" in MathTypes? How does this notion relate to types in modern type theories, such as Martin-Lof type theory?

  2. Does a PASS verdict guarantee the correctness of a mathematical reasoning step, even when the step depends on previous steps or on an LLM-generated theorem-verification procedure? If not, what exactly does a PASS verdict guarantee?

  3. Do the authors have evidence that MathTypes solutions are more readable or comprehensible to human readers than solutions produced using existing formal verification approaches?

4 comments
AuthorsOpenReview ↗

Author Response to Reviewer CfZj

W1 / Q1

We intended to cite Curry-Howard and Martin-Löf as background on verification systems in Section 2.2 and would like to clarify that the “type registry” does not directly correlate to type theory. Instead, MathTypes’ types are closer to Mike Gordon’s Higher-Order Logic (HOL), which is commonly used in Isabelle, and Robin Milner’s Logic for Computable Functions (LCF). The link here is that both use types as rules. LCF, for example, uses theorems as abstract data types that define a small set of inference rules, similar to our design.

We would also like to clarify that MathTypes doesn’t simply check if a function’s signature was used correctly. Each step does three things:

  1. Assigning a typed judgment to a claimed belief state, which is used to check the LLM’s internal computation
  2. The MathTypes compiler matches the arguments to the rule’s signatures
  3. A deterministic code- or rule-based procedure to affirm whether the claim inhabits the judgment

Steps #1 and #3 are precisely what separates it from existing programming languages, allows MathTypes to give high-quality step-level feedback, and makes it lightweight. Therefore, isolating just the compiler’s function (step #2) is not ideal.

Section 2.2 ends with the negative effects of over-formalization on high school and college-level maths, which we precisely aim to solve. We will add this extra qualification to the write-up to make the difference clear.

W2 / Q2

We thank the reviewer for raising the point and would be happy to clarify. Just as a line of Python code that compiles does not guarantee correct programming, a PASS is not a guarantee of absolutely correct mathematical reasoning required at the correct step. It ensures that the step contains sound logic and fits into the theorems, formulas, or conjectures the model uses.

Given that no programming language ensures absolutely correct programming logic or satisfaction of a problem statement, we do not expect the same from MathTypes. Empirically, we show the positive impact of MathTypes verification in Table 1.

ExamAccuracy if the program verifiesAccuracy if it does not
JEE Advanced82.6%50.7%
Oxford TMUA81.2%57.4%
CSIR-NET60.7%39.5%
CMI46.3%18.4%
Brazil IME42.7%8.2%
Brazil ITA32.9%16.3%

Table 1. Empirical analysis of programs (accuracy) of programs that PASS vs. those that fail MathTypes compilation.

A program that PASSes is about 21 points more likely to have the right answer than a rejected one, and the gap holds on every tested exam (+14 to +35 points). Further, we also note that a majority of wrong-answer outcomes that still contain a PASS are linked to a setup derived from a former informal step.

W3 / Q3

We thank the reviewer for raising this point. We conducted a human subject study to test the readability of our language. We showed annotators code snippets or full programs written in MathTypes, Lean, or the SymPy-based SymCode blindly. The task defined the problem, explained the solution, explained what the steps do, and asked the annotator to choose the more readable version. The annotator pool is varied:

  1. A professional Python programmer with extensive training on MathTypes syntax
  2. A competitive programmer (ICPC-qualified)
  3. A professional Python programmer without extensive MathTypes syntax training
  4. A CS & Math graduate student with exposure to Lean

We add Chain-of-Thought as a control to this experiment. Being pure natural language, it is expected to be the most readable. We calculated Elo scores based on Bradley-Terry on the preference annotations. The results are summarized in Table 2.

CoT tops the readability test, as expected, with MathTypes following closely behind. Both SymCode and Lean trail MT by ~300 points, proving the readability argument we make in the paper.

MethodElo Score
Chain-of-Thought (CoT)1690
MathTypes1634
SymCode1342
Lean1333

Table 2. Elo scores from the readability analysis, centered at 1500.

We will add this human study to the paper to back up our claim with full details of the task design, annotator recruitment, and payment procedure.

We hope these new experimental results and discussion address your concerns about the submission. We request that you consider adjusting the score. Thank you for taking the time to evaluate our work.

Thank you for the detailed responses and additional experiments.

Regarding W1, I appreciate the clarification that the notion of "type" in MathTypes is not directly related to type theory. However, the proposed connection to HOL and LCF remains unclear. In particular, LCF's use of an abstract theorem type and trusted inference rules is fundamentally different from, and has little to do with, MathTypes' registry of mathematical operations and verification procedures. I encourage the authors to clarify this distinction rather than introduce potentially misleading analogies.

Also, "Higher-Order Logic (HOF)" appears to be a typo for "Higher-Order Logic (HOL)."

Regarding W2, I appreciate the clarification that a PASS verdict does not guarantee the correctness of the overall reasoning. However, the statement that PASS ensures a step contains "sound logic" seems too strong. Appendix H reports that the induced theorem-verification procedures accept approximately 85–88% of deliberately falsified claims. This suggests that a PASS verdict guarantees only acceptance by the corresponding verification procedure, not the mathematical correctness of the claim. The additional results concerning the association between PASS and final-answer accuracy are useful, but they do not resolve this issue.

Regarding W3, I appreciate the additional readability experiment. However, the study involves only four participants with different levels of familiarity with the evaluated languages, including one extensively trained in MathTypes. This raises concerns about the reliability and generalizability of the results. I do not think such a small study provides sufficient evidence to support the claim that MathTypes is more readable than existing approaches.

Overall, I appreciate the authors' efforts to address my concerns. However, my main concerns remain, particularly regarding the interpretation and reliability of PASS verdicts. I therefore maintain my original rating.

AuthorsOpenReview ↗

Author Response to Reviewer CfZj’s comments

We thank the reviewer for their comments on our rebuttal. We have a few clarifications to make.

W2 (PASS verdicts)

We agree that a PASS does not guarantee mathematical correctness, just as expecting correct programmatic logic from a step that compiles is unreasonable. We will replace the term “sound logic” to reflect this clearly.

Regarding the reliability finding the reviewer notes, we rechecked the 85.4-88.0% figure in Appendix H and found that the false-acceptance rate experiment only perturbs the claimed_result field everywhere, including theorems. However, theorems never check the claimed_result field; they only check conclusion_claimed. Thus, the 85.4–88.0% figure falsely deflates theorems’ soundness.

We reran the experiment by perturbing just conclusion_claimed fields for theorems. Only 1.6-4.2% of theorems actually accept a falsified claim. We have updated Appendix H and present the revised table with the updated caption here:

Table 15: False-acceptance rate (false accepts/scored trials) when each passing step's claim is replaced by a wrong value. The claim is claimed_result, except on apply_theorem steps, whose only claim is conclusion_claimed. Scored trials include ERROR outcomes (272, 126, and 52 for the three sources).

Verification strategyCode editsGLM 5.3 FlashMuse Spark 1.3
Fixed SymPy strategies, pooled0.84% (66 / 7,828)1.31% (31 / 2,362)0.80% (18 / 2,242)
— worst single strategy (verify_condition_implication)100% (20 / 20)100% (4 / 4)100% (4 / 4)
— next worstsympy_solve 3.1%sympy_solve 9.8%verify_add_equations 6.2%
— strategies with no false accepts14 of 1915 of 2115 of 21
theorem_lookup (induced theorem bodies)4.2% (35 / 827)1.6% (4 / 252)2.0% (5 / 249)
All strategies, pooled1.17% (101 / 8,655)1.34% (35 / 2,614)0.92% (23 / 2,491)

We reviewed which theorems accept falsified steps, and they localize to just two: zero_derivative_implies_constant which fails to assert a numeric range, so a perturbed numerical value by a frontier LLM gets accepted; and p_series_convergence, which matches a convergence status by keyword matching. These theorems can easily be fixed with a manual review to drop the false-acceptance rate to 0%.

So, the takeaway is that:

  • PASS means the logic was acceptable to a fixed, auditable procedure
  • The measured failure rate of this procedure is ~1% across types and theorems, pooled

We will state this clearly in the main paper and regret the error in the submitted appendix.

W1 (LCF/HOL)

We agree that analogies with LCF, HOL, or Type Theory are not foolproof. As stated, we will make it clear in Section 2.2 that Type Theory was cited as early background work leading up to modern techniques such as HERMES and SymCode. Our main point of comparison with LCF was its use of a small, fixed set of trusted inference rules, just as MathTypes’ steps are checked by one of 31 fixed verification strategies (Appendix I, Table 16). The analogy is not like-for-like, as MathTypes is not an extension of LCF, and we thank the reviewer for cautioning us in this regard.

We also thank the reviewer for noticing the typo in our rebuttal, which we have corrected.

W3 (Readability)

We agree that four participants cannot support a general claim. Yet, we would like to surface two observations:

  • All four participants independently produce the same ranking (CoT > MathTypes > SymCode > Lean), so the ranking holds even with the participant with MathTypes syntax training held out. Further, we would also like to note that we included a participant with Lean training and professional Python programmers, so one participant with MathTypes training is reasonable.
  • Binary inter-annotator agreement was 66% (moderate).

The readability study is a pilot, as allowed by the rebuttal’s timeline. We are currently recruiting a larger annotator pool, including subject experts in each tested exam, such as JEE Advanced, and intend to update the study by the camera-ready deadline.

Thank you for engaging with our paper. Your comments have made our work stronger. We kindly request that you reconsider your stance based on this clarification.

Thank you for the clarification and responses. I will reply specifically to W1.

Having a small, fixed set of trusted inference rules is a general property of many logical systems, not a distinctive feature of type theory. If that is the main point of comparison, then the connection to type theory in the theoretical sense is absent. Relying on a superficial overlap in the word "type" to associate lightweight schema constraints with foundational frameworks like MLTT or HOL is misleading, even as historical background in Section 2.2. I strongly recommend removing these references.

Reviewer_szgGOpenReview ↗
Rating
4(+1)
Confidence
4
Informativeness
2
Presentation
3(+1)
Soundness
2
Read review
Paper summary

The authors propose a lightweight type-checking based system to perform verified mathematical reasoning in high-school/early graduate level mathematics. Their system sits in between unverifiable natural language reasoning and fully formalized proofs. It first uses an inductive phase to build up a registry of reasoning primitives that can consist of theorems and other symbolic computations. These reasoning programs can be mechanically verified by a symbolic checker like SymPy.

In the induction phase, an annotator LLM proposes items for the registry that are verified with SymPy and also reviewed by another judge LLM before being added to the registry. The paper evaluates many different LLMs as annotators and also takes some basic measures to ensure the registry is minimal.

After building this registry, an LLM can be used to generate proofs using primitives in the registry.

Critical strengths
  1. The paper explores an important gap that exists between informal reasoning and fully formal reasoning. This is a timely contribution to the area.
  2. The idea of inductively constructing a library of primitives is intuitive and also has a basis in prior literature. The authors have applied it to mathematical reasoning with LLMs and conducted extensive model evaluation as seen in figure 2.
Critical weaknesses
  1. Although the methodology is novel and represents a good first step towards a more flexible reasoning verification pipeline, it is unclear if it has any value for future research in improving reasoning in LLMs since models have progressively become reliable verifiers of natural language proofs.
  2. It is unclear how a branching/hierarchical structure would be represented in such a straight line program. Proofs often contain case analyses, intermediate lemmas, etc.
Questions for authors
  1. What are important takeaways from this work that can guide future research in verifiable reasoning?
  2. How would a MathTypes program handle structures like induction, case analysis, and intermediate lemmas?
  3. In table 2, there are frontier models with very low proposal precision. Could you give examples on the types of proposals that are rejected? Are there failure modes of the reviewer LLM (gemini 3.1 pro)?

While the paper is a good contribution, I believe that the presentation needs to be improved. I have a few clarification questions that also require better explanations in the main body.

  1. There needs to be a section discussing limitations and scope for future work.
  2. The definitions of the metrics like Binds Claim and Claim Bind are not clear at all even after consulting the appendix. Could you explain these?
  3. What does accuracy mean in table 3? Is it just based on the final answer or does it consider step-level correctness?
  4. The results section is hard to read and can be organized such that the key results are highlighted.
3 comments
AuthorsOpenReview ↗

Author Response to Reviewer szgG

W1, Q1

Section 4.3 (Lines 465–575) discusses LLM-based Process Reward Models (PRMs):.

  1. We show that using Gemini 3.1 Pro (which is our reviewer) as a PRM yields the same verdict only 82.7% of the time. The remainder is lost to LLMs’ probabilistic nature.
  2. Further, the LLM’s verdict is biased by its own idea of the problem’s correct answer. This suggests that an LLM’s effectiveness as a PRM is limited by its own performance ceiling.

MathTypes targets solution-based problems where the answer is unknown. It's not a proof-checking system. For more evidence, we test a 2nd PRM backbone (Muse Spark 1.3) and run our 171-problem test set with PRMs as step-level judges following ProcessBench. We note that PRMs leave accuracy unchanged over CoT, especially as they reassert the LLM’s own belief in 87-91% of cases. They lack the additional math-solving capability to offer meaningful feedback. MathTypes solves this problem and achieves a 12.2-13.8% accuracy gain.

Setting (Solver: Muse Spark 1.3)Accuracy
MathTypes69.6% ± 1.5
CoT57.4% ± 0.8
PRM loop, w/ Gemini 3.1 Pro57.3%
PRM loop, w/ Muse self-judge55.8% ± 0.9
Table 1. MathTypes, PRM, and CoT results on the 171-problem eval set.

Important takeaways: MathTypes provides deterministic, checkable feedback on supported steps, addressing PRMs' inconsistency and answer bias

W2, Q2

Thank you for raising this point. We clarify that MathTypes is not designed to be a proof assistant, unlike Lean or Coq. However, given the flexibility it allows, models wrap structured proof steps around escape hatches. Regex analysis of completed programs with >2 distinct steps gives:

MethodCount%
Case analysis1704.4%
Proof by contradiction1453.8%
Named lemma/claim280.7%
Induction160.4%
WLOG80.2%

Encoding such structured proof techniques into MathTypes remains a future work direction. Specialized expertise and exhaustive effort are required for correct design, yet only ~5% of all programs would use them today. Further, MathTypes’ core idea delivered accuracy gains in proof-heavy physics problems (table: reviewer NE3L’s Q2).

Q3

Proposing the right theorems and rules for induction into MathTypes is difficult. We approached it cautiously to avoid the type registry from getting unnecessarily verbose and bloated.

  • Rejected proposal examples:

    • ~40% of rejections came from fake/tautological verification steps (e.g., theorems that just assert True without any real mathematical calculation or checks a natural language string instead of any math).
    • Types/theorems hardcoded to one problem that might not generalize: 30%. E.g., Muse proposed a derivative x*(2 - cos x) with intervals hardcoded to the problem at hand.
    • Proposals redundant with an existing type: 21%. These are mostly specialized types that can be expressed simply. E.g., Gemini 3.1 Pro proposed inverse_tangent_difference_identity with a tautological tan(atan(A)) == tan(atan(A)) check. It was redundant with rewrite_expression.
  • Reviewer failure modes: Gemini is overly strict in checking proposals, including demoting operations such as counting integers in an interval to informal_step. Binary agreement between the 2 tested reviewers (Opus 5, Gemini 3.1 Pro) is 63%, so judgments are mildly subjective.

We will add this analysis with illustrative examples to the appendix.

C1

We will include a dedicated limitations section in the final version, covering the partiality of MathTypes verification, limited scope to exam-style math problems, and required future work on learning-based methods, expert-written type registries, and expansion to proofs.

C2

“Binds claim” is the rate at which an annotator model uses the claimed_result or conclusion_claimed types. It’s a litmus test for whether the annotator wants the MathTypes verifier to mathematically check the claimed result or conclusion. This is the positive property; higher is better.

“Claim-blind” is the rate at which the annotator did not use either a binding type or a condition (claimed_*, new_*, or target_* patterns). E.g.: Gemini 3.1 Pro proposed a theorem that just asserts tan(atan(A)) == tan(atan(A)). Note that this always passes, so it’s claim-blind, which is undesirable.

C3

Table 3 presents question-solving accuracy verified by final answer correctness, as is common in the six exams we test. We will update the caption to make this explicitly clear.

C4

We will color the best result in each column and split long paragraphs. We also split the analyses into sub-sections on the effect of the correction loop (section 4.3), registry construction ablations (4.4), verifier soundness (4.5), and generalization beyond mathematics (4.6).

We thank the reviewer and ask that you consider adjusting the score. Thank you for taking the time to evaluate our work.

I thank the authors for their response. I am satisfied with their responses and I have changed my score to 4 since I believe this is a well engineered system that can be useful for verifiable reasoning with small models in settings where you require trust in the reasoning.

AuthorsOpenReview ↗

Author Response to Reviewer szgG

We thank the reviewer for reconsidering our work in light of the rebuttal and recognizing the importance of MathTypes in improving the faithfulness of small language models. We will incorporate all changes outlined in our rebuttal into the final version of the paper.

Reviewer_NE3LOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes a typed, verifiable programming language for representing mathematical reasoning. They propose a method to translate natural language deductions into proofs written in this language, such that a compiler can verify the correctness of each step and provide feedback to the LLM solver. LLMs equipped with this harness outperform baselines on difficult math problems.

Critical strengths
  1. The idea to add auto-formalization into LLM reasoning is novel and interesting. The results show good improvements to performance on medium scale LLMs. The main concern with such an approach would be cost-efficiency, but the paper addresses this with their Pareto frontier analysis.
Critical weaknesses
  1. The paper is somewhat difficult to follow, specifically regarding the distinction between the method as a verifier of LLM reasoning and its use as a test-time inference method. The exposition throughout the paper should make it clearer what the motivation behind each component of the framework is. For example, it would be helpful to have better explanations of what the aim of the annotator, reviewer, auditor, and solver are more explicitly.

  2. Some key concepts in the paper, such as the type registry, need to be explained with more background as the broader community will struggle to understand its significance in the method. In general, greater care should be taken to provide the relevant background for technical concepts.

Questions for authors
  1. How can you be sure that the seed grammar generation process does not leak signal about the types of problems in the test set? I would be concerned about data leakage here.

  2. How dependent is your method on a task bespoke seed grammar, and can it generalize to math domains outside the scope of the data that was used to generate the seed grammar?

2 comments
AuthorsOpenReview ↗

Author Response to Reviewer NE3L

W1

We thank the reviewer for raising this point and will include a new subsection under Section 3 to discuss the “Compile–Verify–Correct” test-time loop separately. The current draft had to be consolidated due to space constraints, and we aim to address this in the final version. Section 3 will separate registry construction (new 3.2: annotator, reviewer, auditor) from the test-time Compile–Verify–Correct loop (new 3.4: solver and verifier), with sections on the language and registry (3.1) and the verifier (new 3.3) introduced first.

W2

The type registry is simply the list of all types, theorems, and escape hatches that are callable. Think of it as all the available objects in a Python library that can be used in code.

  • A type encodes a generic mathematical operation, such as solving a linear equation. The type registry defines a typed signature and the deterministic check for it.
  • A theorem is a named result with a list of preconditions and code to check the claim (e.g., intermediate value theorem)
  • An escape hatch lets the solver write steps that cannot be deterministically checked. E.g., informal_step or translate_to_formal to, say, encode values visible in a figure.

We will add a glossary defining each term to the paper such that the broader community easily understands these concepts.

Q1

All problems used for creating the seed grammar and the type registries are strictly separated from those in the test set, as Line 313-314 notes. We use older problems to induce the grammar and test MathTypes on newer editions of each exam. This is representative of how humans and exam agents work in real life, with training on past editions of the exam and evaluation on the latest held-out version. Table 1 shows the difference relative to the seed grammar, which used only JEE Advanced, CMI, and SAT problems.

ExaminationExam Years used for Seed Grammar CreationExam Years used in the Test Set
JEE Advanced2012–142019–25
CMI2015–192022–26
SATPublic Practice Test 4Not Tested

Table 1. Exam year-based separation used in the seed grammar creation.

All types and theorems in the seed grammar are fundamental mathematical operations such as the Pythagorean theorem or linear equation solving (please see Appendix L for a list of all types and theorems) that do not correlate with any eval set leakage. We agree this detail is not clarified in the Methods section, and we will add a paragraph on the validation-test set separation.

Q2 (Dependence on Seed Grammar)

We thank the reviewer for raising this point. Hand-curating a high-quality seed grammar can greatly help LLM-based induction. The main paper reports an experiment in Section 4.3 (Lines 432-446) on TMUA, where we bootstrapped the LLM induction process from just the informal escape hatches against the hand-curated 17-type and 9-theorem seed. We note that only 7/17 types and 0/9 theorems in the seed were reconstructed. Also, starting without the seed caused two LLM backbones to lose question-solving accuracy, and models frequently accessed “unknown” operations.

We note that since the LLM induction process is probabilistic, providing a high-quality seed is necessary to ground the process.

Q2 (Generality of MathTypes)

We tested our method by creating a hard set on all Physics problems in the OlympiadBench dataset [1]. Opus 5.5 was used as the annotator model to create a PhysicsTypes language with laws and Physics-specific types, and we tested on a held-out set of 102 problems. We took extra care to ensure that problems that share the same Physics setup aren’t present in both validation and test sets.

By offering structured verification for checking the physics setup in the problem (a type proposed by Opus), PhysicsTypes improved problem-solving accuracy by 19.9 points over CoT and remains 12.7 points better than the best method, PS+.

Setting (LLM: Muse Spark 1.3)Accuracy
CoT (official OlympiadBench prompt)19.3% ± 3.7
PoT22.9% ± 2.0
PedCoT21.9% ± 1.5
SymCode25.2% ± 3.0
SelfCheck25.5% ± 3.5
PS+26.5% ± 1.7
PhysicsTypes (Ours)39.2% ± 2.9

Table 2. Evaluation results on 102 hard-filtered Physics problems from OlympiadBench.

We will add this generalization study to the main body of our paper. We hope these added experimental results and clarifications resolve your concerns about our submission. We request that you consider adjusting the score. Thank you for taking the time to evaluate the paper.

[1] He, C., et al. (2024). OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems. ACL 2024, 3828–3850.

AuthorsOpenReview ↗

Rebuttal deadline is coming soon

Dear Reviewer NE3L,

We thank you for the thoughtful review. Our comments above:

  1. Promised to add a separate section on the Compile-Verify-Correct loop
  2. Provided additional definitions for a few key terms, such as the type registry
  3. Clarified that data leakage was strictly controlled across induction and evaluation stages
  4. Pointed to the experiment in the main paper on seed grammar dependence and its importance (Section 4.3, Lines 432-446)
  5. Presented a new generalization experiment on Physics Olympiad problems, showing significant performance gains with our method

We hope these clarifications have addressed all the highlighted weaknesses in the review. This is a kind reminder that the author–reviewer discussion phase will be closing in less than two days. If you find that our responses satisfactorily addressed your concerns, we would greatly appreciate a reconsideration of your rating. We would be happy to answer any questions or provide further clarification.

Best regards, Authors

Comments on the paper

3 comments
AuthorsOpenReview ↗

Summary of Actions Taken To Address Reviewer Comments

We appreciate that reviewers found our approach to adding autoformalization to LLM reasoning novel, interesting (NE3L), “important,” and “timely” (szgG). We are also glad that szgG found our typed language design “intuitive” and grounded in prior work. CfZj found the contribution to be an “interesting alternative” to both informal LLM verification and full proof assistants. We particularly appreciate their observation that the ablations show MathTypes’ verifier feedback drives performance gains. In light of the rebuttals, Reviewer szgG also increased their score to a perfect 4/4.

Improvements to clarity and structure of the paper (NE3L, szgG, CfZj)

  • We promised to restructure Section 3 into the language and registry (3.1), registry construction with proper definitions of annotator, reviewer, and auditor (3.2), the verifier (3.3), and the test-time loop (3.4).
  • Also, we will split the results into Sections 4.3-4.6 and color-highlight the important takeaways.
  • A separate limitations section will be added to document which verification checks, the absence of structured proofs, and directions for future work.

Type theory framing (CfZj)

We will make it explicitly clear in Section 2.2 that “type theory” has no relation to MathTypes’ type registry and that Martin-Löf’s type theory is being cited as background work that led to proof assistants like Lean, Coq, Agda, etc.

What a PASS means (CfZj)

We will replace “sound logic” with “accepted by the step’s verification procedure." Also, we will update Appendix H with the new experimental results that put the type checker’s failure rate at ~1% when deliberately perturbed.

New PRM-in-the-loop baseline (szgG)

Added a new experiment with Gemini 3.1 Pro and Muse Spark 1.3 as PRM, where their loops scored no better than CoT. We will add these results to Table 3.

Generalization beyond math (NE3L)

We showed that the methods defined in MathTypes can be used to deliver gains in other domains like Physics by building PhysicsTypes. We will add this result to hint at the potential for future work based on our work.

Data leakage and dependence on seed grammar (NE3L)

We clarified that no data was leaked between the grammar induction and evaluation stages. We also pointed to the experiment in Section 4.3 on reviewer 3’s concern about seed grammar dependence.

Structured proofs analysis, rejected proposals and reviewer failure modes analysis (szgG)

Added extra analyses on how often MathTypes allows the use of structured proof methods. Also showed that rejected proposals mostly stem from tautological checks, problem-specific, or redundant types/theorems. We will add these results and examples to the appendix.

Readability (CfZj)

We ran a pilot human study which showed that MathTypes is considerably more readable than Lean or SymPy-based methods, while slightly behind natural-language-based Chain-of-Thought traces. We will include an expanded study with more participants in the main paper.

Thank you for the clarification. However, rather than explicitly stating that Martin-Löf’s type theory is unrelated to MathTypes’ type registry, I recommend simply removing the reference to type theory in the logical tradition altogether from Section 2.2.

Since the paper does not engage with type-theoretic hierarchies (such as simple or dependent types) from logic/type theory, citing Martin-Löf’s type theory merely as general background for proof assistants like Lean or Coq, which is already well known, is unnecessary and potentially misleading to readers.

AuthorsOpenReview ↗

Author Response to Reviewer CfZj

We sincerely thank Reviewer CfZj for their continuous engagement with our paper. We agree that MLTT has no connection with our work, and our current paper also never makes such a claim. We will edit Section 2.2 to remove citations to MLTT and the Curry-Howard correspondence, instead leading directly with modern proof assistants. Specifically, we are planning this edit:

Replace "Modern computational mathematics verification…” through “…and Idris” with just "Proof assistants such as Coq, Lean, Agda, and Idris provide deterministic, kernel-checked verification of mathematics."

We hope the reviewer is satisfied with the remaining results we have provided in our rebuttals. Specifically, we would be glad if you could reconsider your score on the reliability of PASS steps (W2). We showed that our language accepts deliberately falsified inputs only ~1% of the time. These results are also updated in our submitted PDF.

Further, we also provided some clarifications regarding the human subject study and the validity of the ranking in the pilot (W3). It would help us if you would reconsider your stance if all of your concerns have been addressed.

To summarize our answers to your questions:

  1. A type is a registered operation with a typed signature and a deterministic check that has little to no connection with Martin-Löf's Type Theory.
  2. A PASS verdict guarantees whether a mathematical operation modeled through a type matches the model's belief state. It has a ~1% failure rate, pooled, when deliberately attacked with a malicious prompt and frontier LLMs. We believe this failure rate can be minimized with human interventions. The paper studies the extent of a fully automated process' capabilities.
  3. We have early evidence that MathTypes is slightly less readable than natural-language Chain-of-Thought but significantly better than Lean and SymPy-based methods. Four varied annotators converged on the same ranking, with binary agreement at 66%.

Thank you for your time and consideration of our paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.