acceptodds
Withdrawn from ICLR 2027

An Investigation of Robustness of LLMs in Mathematical Reasoning: Benchmarking with Mathematically-Equivalent Transformation of Advanced Mathematical Problems

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Frontier large language models (LLMs) now reach near-ceiling accuracy on standard mathematical-reasoning benchmarks and gold-medal-level performance at the International Mathematical Olympiad. As these benchmarks saturate and their items leak into training data, a high score no longer shows whether a model reasons robustly, nor which component of its reasoning fails when it does not. To evaluate reasoning in a way that remains informative for such models and makes its failures diagnosable, we propose a novel evaluation methodology called GAP (Generalisation-and-Perturbation), where the key idea is to automatically generate many mathematically-equivalent variants of an existing mathematics problem at scale, using two disjoint interpretable transformations: 1) surface renames, which probe the binding between identifiers and latent variable roles, and 2) kernel rewrites, which probe whether the high-level proof plan survives a change of mathematical setting. Compared with existing benchmarks, a benchmark created using GAP has two important benefits: 1) Since the new variants generated are unlikely "seen" by an LLM, a GAP benchmark effectively addresses the data leakage problem. 2) A GAP benchmark enables diagnosis of reasoning failures based on the performance of an LLM over a spectrum of variants of mathematics problems designed with different transformations, each corresponding to a meaningful hypothesis about the cause of failure. We instantiate GAP on all of the 1,051 William Lowell Putnam Competition problems from 1938 to 2024, adding 5,255 unseen transformed instances to form PutnamGAP, a 6,306-item competition-level mathematics corpus, resulting in the first public machine-readable dataset built from the full Putnam archive. Using PutnamGAP, we evaluated 18 commercial and open-source models spanning different sizes and providers, and found that accuracy drops across all models and variant families, with the largest degradations under kernel rewrites. This gap does not close with model strength, suggesting that the dominant remaining weakness of even the strongest models is carrying a proof plan into a changed setting, not handling surface changes. Further diagnostic analysis enables more detailed understanding of the causes of failure and generates potentially useful insights for future research on improving the reasoning capacity of LLMs.

Primary area:
datasets and benchmarks
Keywords:
mathematical reasoning, benchmark, robustness, large language models, evaluation, Putnam

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_fDsJOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors compare each math problem with versions that change its variable names or its mathematical setting while trying to keep the same basic solution method. They test whether models that solved the original can still solve the changed versions. Models struggle more when the mathematical setting changes. The idea is exciting, but I am not sure how varied the kernel rewrites really are. A model could be trained on generated variants and learn the solution patterns needed to handle fresh versions of the same problems. The paper does not investigate this, so I am not convinced by the broad claim about resistance to data leakage. I also want clearer comparisons with Putnam-AXIOM and MATH-Perturb, and more detail about how the human audit checked that the rewrites were correct and the same solution method still worked. The explanation of why models fail needs stronger evidence. I currently recommend weak rejection. I would be happy to raise this to weak acceptance if the authors resolve these questions.

Critical strengths
  1. The benchmark compares different changes to the same problems. Each original problem is compared with versions that change only variable names and a version that changes the problem and answer. This helps show which changes a model can handle. The many Putnam competition problems also give the result broader coverage than a small set of examples.

  2. Models struggle more when the mathematical setting changes (the authors call this as kernel rewrites) than when variables are renamed. This pattern appears across the models tested, even when the comparison includes only problems they solved before the change. The authors also give a model the beginning of an expert solution to check whether a hint helps it solve a problem it previously missed.

Critical weaknesses
  1. The paper does not clearly explain what GAP adds. Putnam-AXIOM already changes variable names and constants. MATH-Perturb changes problems and answers, with some variants keeping the solution method and others requiring a new one. The authors should explain what GAP does differently when it creates and checks problems and studies model failures.

  2. The checks of the rewritten problems need more detail. The authors found no substantive errors in a 10% sample checked by hand. How did they check that the problems were correct and the same solution method still worked? The automated checks reuse the same model, which can repeat mistakes. I do not see how this establishes a near-zero chance of accepting an incorrect problem.

  3. The abstract and introduction make stronger claims than the results support. The rewrites change the problem and answer, so "mathematically equivalent" is misleading. Harder calculations may explain some failures despite the difficulty checks. Similar wording also does not show that solutions use the same mathematical steps. The authors should inspect those steps, allowing other valid proofs, and directly test whether hints help renamed problems more than rewritten ones.

Questions for authors
  1. Could you compare GAP with Putnam-AXIOM and MATH-Perturb using worked examples from the same source problems? For MATH-Perturb, please include its simple variants, which keep the solution method. Figure 2 replaces b with b+c while keeping the proof steps. What does GAP add to how variants are generated, checked and used to explain model failures?

  2. What does GAP's leakage-resistance claim cover if a model is fine-tuned on generated variants, as the paper itself suggests? Fresh test questions could still reuse the source problems and solution steps seen during fine-tuning. Could you compare gains on fresh variants of those training problems with gains on source problems kept entirely out of fine-tuning?

  3. Who checked the 10% sample, and how many rewrites did it contain? How did they check that the rewritten problems were correct and that the same solution method still worked? For the automated checks, how was the error rate estimated, and what supports treating mistakes across repeated calls as independent?

  4. Could you examine the mathematical steps in a sample of successful and failed solutions to rewritten problems, allowing other valid proofs? This would help distinguish a failure to use the intended method from a calculation error.

  5. What is the difference between the improvement from hints on renamed problems and on rewritten problems? Please give an uncertainty interval and account for versions that come from the same original problem.

Reviewer_CGHpOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes GAP, a framework for evaluating the robustness of mathematical reasoning through two families of problem transformations. Surface transformations rename mathematical identifiers while preserving the statement, answer, and reference solution. Kernel transformations modify the mathematical setting and answer while attempting to preserve the high-level proof plan. The authors instantiate this framework on 1,051 Putnam problems from 1938–2024. Four surface variants and one kernel variant are generated for each source problem, resulting in 6,306 aligned instances. The paper evaluates 18 commercial and open-source language models. It reports relatively small performance degradation under surface transformations but substantially larger degradation under kernel transformations. Based on trajectory-overlap measurements and solution-prefix interventions, the authors characterize the former as “self-inconsistency” and the latter as “strategy drift.”

Critical strengths
  1. Scale and coverage of the resulting resource. The paper constructs aligned variants for the full Putnam archive and evaluates a broad set of models.
  2. Within-item evaluation design. Comparing multiple transformations of the same source problem is preferable to comparing aggregate performance on unrelated datasets. The deterministic construction of the surface variants is also relatively transparent and reproducible.
Critical weaknesses
  1. Limited novelty and an incomplete treatment of highly relevant prior work The central research question and empirical message overlap substantially with existing work on perturbation-based evaluation, compositional generalization, and the brittleness of mathematical reasoning.

There is substantial overlap with:

  • GSM-Plus: A Comprehensive Benchmark for Evaluating the Robustness of LLMs as Mathematical Problem Solvers
  • GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models
  • MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbations
  • Mathtrap: Exploring the Compositional Deficiency of Large Language Models in Mathematical Reasoning (not mentioned anywhere in the article)
Questions for authors

None

Reviewer_Up49OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

GAP distinguishes surface renamings, which preserve the mathematical problem and answer, from kernel rewrites intended to preserve an abstract proof plan while changing the instance and answer. The contribution applies these transformations to 1,051 original Putnam problems, producing approx. 6x more items in total. The paper evaluates 18 LLMs and uses trajectory comparisons and partial-solution prefixes to investigate their failures. It reports generally larger losses under kernel rewrites and interprets the different behavioral patterns as evidence about symbol-role binding and proof-plan transfer.

The paper approaches a timely problem, namely, how much of success on a familiar mathematical problem is preserved when its presentation or mathematical setting is changed. The useful conceptual move here is to organize evaluation around specified invariances and paired source problems. This could make robustness results more informative than a single accuracy score. The central scientific question is whether these transformations and measurements isolate the capabilities assigned to them. The current paper advances that agenda, but establishes the behavioral distinction more convincingly than the proposed diagnostic explanation.

Critical strengths
  1. A useful evaluation perspective and a potential community resource. The combination of paired source problems, broad archive coverage, and aligned variants could support systematic robustness studies. This resource contribution can be valuable even if the mechanism claims are narrowed.

  2. Good effort to explain observed behavior. Trajectory comparisons, interventions, and controls generate useful hypotheses and add empirical substance beyond aggregate performance scores.

Critical weaknesses
  1. The claims may exceed what the design currently identifies. Different transformation effects do not uniquely identify the failed reasoning component. The overlap measure may lack direct validation as proof-plan preservation. Moreover, significance in only one intervention group does not establish different treatment effects.

  2. Benchmark validity remains an unresolved foundation. The reliability bound’s calibration and assumptions are not fully established. A more controlled design must establish valid transformations and whether the headline results survive, because preserved invariances are foundational to this benchmark.

Questions for authors

Below I put some critical interpretation points that could inform the author response.

  1. As far as I understood, the evidence establishes different behavioral responses to surface and kernel transformations, but does not uniquely attribute kernel failures to proof-plan transfer. Differences in difficulty, task validity, and the information supplied by prefixes remain alternative explanations. The conclusion about a dominant planning weakness is stronger than the evidence supports.

  2. Benchmark validity remains unresolved. The reliability analysis does not establish that the intended invariances hold throughout the benchmark.

  3. The intervention results do not establish a difference between surface and kernel recovery effects. Statistical significance for one group and nonsignificance for another do not demonstrate different effects. While the null-reattempt and prefix-length controls strengthen the recovery findings, the claimed diagnostic separation remains insufficiently supported.

  4. The motivation, two transformation families, and four contributions are clearly identifiable. However, the headline claims are not consistently qualified by the paper’s own definitions and evidence. The introduction distinguishes answer-preserving surface transformations from proof-plan-preserving kernel rewrites, whereas the title and broader framing describe mathematically equivalent transformations. Generating fresh variants supports contamination resistance without establishing that data leakage is resolved. Moreover, distinct behavioral responses do not by themselves establish diagnostic separation of failure mechanisms.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.