- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors compare each math problem with versions that change its variable names or its mathematical setting while trying to keep the same basic solution method. They test whether models that solved the original can still solve the changed versions. Models struggle more when the mathematical setting changes. The idea is exciting, but I am not sure how varied the kernel rewrites really are. A model could be trained on generated variants and learn the solution patterns needed to handle fresh versions of the same problems. The paper does not investigate this, so I am not convinced by the broad claim about resistance to data leakage. I also want clearer comparisons with Putnam-AXIOM and MATH-Perturb, and more detail about how the human audit checked that the rewrites were correct and the same solution method still worked. The explanation of why models fail needs stronger evidence. I currently recommend weak rejection. I would be happy to raise this to weak acceptance if the authors resolve these questions.
Critical strengths
-
The benchmark compares different changes to the same problems. Each original problem is compared with versions that change only variable names and a version that changes the problem and answer. This helps show which changes a model can handle. The many Putnam competition problems also give the result broader coverage than a small set of examples.
-
Models struggle more when the mathematical setting changes (the authors call this as kernel rewrites) than when variables are renamed. This pattern appears across the models tested, even when the comparison includes only problems they solved before the change. The authors also give a model the beginning of an expert solution to check whether a hint helps it solve a problem it previously missed.
Critical weaknesses
-
The paper does not clearly explain what GAP adds. Putnam-AXIOM already changes variable names and constants. MATH-Perturb changes problems and answers, with some variants keeping the solution method and others requiring a new one. The authors should explain what GAP does differently when it creates and checks problems and studies model failures.
-
The checks of the rewritten problems need more detail. The authors found no substantive errors in a 10% sample checked by hand. How did they check that the problems were correct and the same solution method still worked? The automated checks reuse the same model, which can repeat mistakes. I do not see how this establishes a near-zero chance of accepting an incorrect problem.
-
The abstract and introduction make stronger claims than the results support. The rewrites change the problem and answer, so "mathematically equivalent" is misleading. Harder calculations may explain some failures despite the difficulty checks. Similar wording also does not show that solutions use the same mathematical steps. The authors should inspect those steps, allowing other valid proofs, and directly test whether hints help renamed problems more than rewritten ones.
Questions for authors
-
Could you compare GAP with Putnam-AXIOM and MATH-Perturb using worked examples from the same source problems? For MATH-Perturb, please include its simple variants, which keep the solution method. Figure 2 replaces
bwithb+cwhile keeping the proof steps. What does GAP add to how variants are generated, checked and used to explain model failures? -
What does GAP's leakage-resistance claim cover if a model is fine-tuned on generated variants, as the paper itself suggests? Fresh test questions could still reuse the source problems and solution steps seen during fine-tuning. Could you compare gains on fresh variants of those training problems with gains on source problems kept entirely out of fine-tuning?
-
Who checked the 10% sample, and how many rewrites did it contain? How did they check that the rewritten problems were correct and that the same solution method still worked? For the automated checks, how was the error rate estimated, and what supports treating mistakes across repeated calls as independent?
-
Could you examine the mathematical steps in a sample of successful and failed solutions to rewritten problems, allowing other valid proofs? This would help distinguish a failure to use the intended method from a calculation error.
-
What is the difference between the improvement from hints on renamed problems and on rewritten problems? Please give an uncertainty interval and account for versions that come from the same original problem.