- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper investigates whether LLMs preserve the scope of mathematical statements when rewriting them across different mathematical subfields. The authors introduce an evaluation framework that separately measures truth, content, and scope, and report a systematic direction-dependent failure across seven models.
Critical strengths
-
The paper studies an interesting problem that is not well captured by conventional mathematical reasoning benchmarks. Separating scope preservation from statement correctness is a useful perspective.
-
The authors make an effort to validate their evaluation protocol through held-out experiments, human-authored statements, cross-family judges, and controlled perturbations.
Critical weaknesses
-
The central claim is not sufficiently supported. The paper argues that LLMs lack the structural representations needed for translation between mathematical theories. However, the experiments only demonstrate errors in generated statements. These behavioral observations do not establish what the models internally represent. Other explanations, such as task interpretation or prompting effects, remain plausible.
-
The task formulation is questionable. The basic prompt only asks models to rewrite a statement in another mathematical framing, without explicitly requiring preservation of its logical strength. When this requirement is added, hypothesis promotion increases substantially. The paper still concludes that instruction does not resolve the problem, partly because models retain redundant hypotheses in the reverse direction. However, retaining a redundant hypothesis does not necessarily make a mathematical statement incorrect.
-
The evaluation is too narrow for the conclusions drawn. The main findings are based on only three vertical framing pairs, with 45 held-out statements evaluated repeatedly across seven models. The additional human-authored corpus is also small. While the observed pattern is interesting, this coverage is insufficient to support broad conclusions about mathematical representation in LLMs. The paper also does not evaluate mathematics-specialized models.
-
The proposed evaluation does not fully establish the claimed failure mechanism. Although the authors include useful reliability checks, the judge-free metric relies on manually specified patterns with limited recall. More importantly, detecting the presence or absence of a hypothesis is not equivalent to evaluating the full mathematical correctness of a translation. The paper does not sufficiently separate failures of mathematical understanding from failures to follow a particular rewriting convention.
Questions for authors
-
The main seven-model evaluation uses only 45 held-out statements from three framing pairs. How representative are these pairs of mathematical translation more broadly? Would the findings hold across substantially more mathematical domains and theory relationships?
-
The ProofWiki evaluation contains only 25 statements, with 7, 15, and 3 examples across the three pairs. How stable are the reported results at this scale? Similarly, Table 4 draws conclusions from categories containing as few as 4 or 8 examples.
-
The instruction experiment in Table 7 uses only 37 scorable statements from one model with a single sample per item. Why is this sufficient to support the conclusion that instruction cannot remedy the problem? Have the authors evaluated the full semantic fidelity of the rewritten statements under the explicit instruction?
-
The judge-free hypothesis detector has a reported recall of 0.52. How sensitive are the main conclusions to alternative valid formulations of the required assumptions? Could the evaluation be extended beyond fixed textual patterns?