acceptodds
Under review as a conference paper at ICLR 2027

Objects Without Morphisms: What LLMs for Mathematics Do Not Represent

Abstract

Large language models (LLMs) have reached expert-level performance on competition mathematics largely through the volume of search placed around them: candidate solutions are sampled in quantity and retained only when an external criterion accepts them. Such a procedure improves the outcome that survives it while leaving untouched what the model represents. We examine that question where no external criterion exists: translating statements between the dialects of neighbouring subfields, where fidelity turns on the level of generality at which content is asserted. The source leaves that level implicit in its vocabulary, so a faithful translation must recover it from the relation between the theories. We introduce an instrument that codes truth, content and scope in separate blind queues, with a judge-free measure of whether a rewrite states the hypothesis implicit in its source, and establish its sensitivity with a planted-positive control. Across seven models from four families, translating towards the general framing widens the domain of quantification in of rewrites and narrows it in none; translating towards the concrete framing narrows it in and widens it in . The hypothesis that would prevent it is stated in of model rewrites and of human statements. Capability does not govern the asymmetry: it appears in every model tested, and the most capable widens least. It replicates on the half of the benchmark held out by a pre-registered rule, and on statements written by mathematicians. Instructing a model to state every hypothesis it requires raises that rate but not its sensitivity to direction. We argue that these systems have acquired an object-level correspondence between subfield vocabularies without the constraint under which a translation between theories carries hypotheses to hypotheses.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Mathematical reasoning, Large language models, evaluation, benchmarks, LLM-as-a-judge, autoformalization, semantic equivalence, informal mathematics, prompt sensitivity

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_xo9dOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates whether LLMs preserve the scope of mathematical statements when rewriting them across different mathematical subfields. The authors introduce an evaluation framework that separately measures truth, content, and scope, and report a systematic direction-dependent failure across seven models.

Critical strengths
  1. The paper studies an interesting problem that is not well captured by conventional mathematical reasoning benchmarks. Separating scope preservation from statement correctness is a useful perspective.

  2. The authors make an effort to validate their evaluation protocol through held-out experiments, human-authored statements, cross-family judges, and controlled perturbations.

Critical weaknesses
  1. The central claim is not sufficiently supported. The paper argues that LLMs lack the structural representations needed for translation between mathematical theories. However, the experiments only demonstrate errors in generated statements. These behavioral observations do not establish what the models internally represent. Other explanations, such as task interpretation or prompting effects, remain plausible.

  2. The task formulation is questionable. The basic prompt only asks models to rewrite a statement in another mathematical framing, without explicitly requiring preservation of its logical strength. When this requirement is added, hypothesis promotion increases substantially. The paper still concludes that instruction does not resolve the problem, partly because models retain redundant hypotheses in the reverse direction. However, retaining a redundant hypothesis does not necessarily make a mathematical statement incorrect.

  3. The evaluation is too narrow for the conclusions drawn. The main findings are based on only three vertical framing pairs, with 45 held-out statements evaluated repeatedly across seven models. The additional human-authored corpus is also small. While the observed pattern is interesting, this coverage is insufficient to support broad conclusions about mathematical representation in LLMs. The paper also does not evaluate mathematics-specialized models.

  4. The proposed evaluation does not fully establish the claimed failure mechanism. Although the authors include useful reliability checks, the judge-free metric relies on manually specified patterns with limited recall. More importantly, detecting the presence or absence of a hypothesis is not equivalent to evaluating the full mathematical correctness of a translation. The paper does not sufficiently separate failures of mathematical understanding from failures to follow a particular rewriting convention.

Questions for authors
  1. The main seven-model evaluation uses only 45 held-out statements from three framing pairs. How representative are these pairs of mathematical translation more broadly? Would the findings hold across substantially more mathematical domains and theory relationships?

  2. The ProofWiki evaluation contains only 25 statements, with 7, 15, and 3 examples across the three pairs. How stable are the reported results at this scale? Similarly, Table 4 draws conclusions from categories containing as few as 4 or 8 examples.

  3. The instruction experiment in Table 7 uses only 37 scorable statements from one model with a single sample per item. Why is this sufficient to support the conclusion that instruction cannot remedy the problem? Have the authors evaluated the full semantic fidelity of the rewritten statements under the explicit instruction?

  4. The judge-free hypothesis detector has a reported recall of 0.52. How sensitive are the main conclusions to alternative valid formulations of the required assumptions? Could the evaluation be extended beyond fixed textual patterns?

Reviewer_jj8bOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

Modern LLM performance at math is the result of brute force. They generate a lot of answers, but only the ones that pass the checker are kept. This can help with final answers, but it doesn’t show that the models actually understand the math.

The authors want to test a setting with no answer checker. They consider the setting where mathematical statements in neighboring subfields are translated to one another. They found that models would often broaden the claims in the translated statements in a way that could make the statements false. This happened less for more capable models.

The abstract, introduction, methodology, and experiment sections are very difficult to follow and require many re-reading sessions to understand. The paper requires substantial re-writing. The lack of clarity of this paper makes it very difficult for me to assess the value of the contributions and soundness of the methodology.

Critical strengths

The subject of study, assessing whether leading AI models understand math structure rather than learning math by brute force, is timely and important

The consistent pattern of broadening mathematical statements during translation is interesting and is a failure mode worth studying

Critical weaknesses

The level of clarity of this paper is below the standard of ICLR. I suggest that the authors re-write the paper.

The lack of clarity of this paper makes it very difficult for me to assess the value of the contributions and soundness of the methodology.

If I am the only reviewer who had significant trouble understanding the paper, I invite the AC to disregard my review.

Questions for authors

The lack of clarity of this paper makes it very difficult for me to assess the value of the contributions and soundness of the methodology.

Reviewer_iToEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies how LLMs translate a mathematical statement into a neighboring subfield, for example from vector spaces to modules. A statement in one subfield often carries a hidden assumption in its vocabulary. "Vector space" already assumes the scalars form a field, so a correct translation into module theory must state this assumption, and the reverse translation may drop it. The authors evaluate seven models from four families on statement pairs from the MELD benchmark and on 25 pairs from ProofWiki. LLM judges label each translation for truth, for equivalence to the original, and for whether it covers the same objects as the original, more, or fewer. A regex separately checks whether the translation states the hidden assumption.

The main finding is that the error depends on the direction of the translation. When translating to a more abstract subfield, 60.6% of the translations cover more objects than the original, and none cover fewer. When translating to a more specific subfield, 28.3% cover fewer objects and almost none cover more. The pattern appears in all seven models and on held-out items and ProofWiki pairs. A prompt that asks for a translation "no more general and no more specific" makes one model state the assumption more often. The authors conclude that models learn which words correspond across subfields, but not that the assumptions behind those words must carry over.

Critical strengths

The problem is legit and relevant. An error in which objects a statement covers can leave the statement true and fluent, so standard correctness checks do not catch it. The direction-dependent pattern is also clean. A total of 191 translations toward the abstract subfield cover more objects and none cover fewer, and the pattern holds across all seven models and on held-out data. This is the main reason the paper is worth considering.

Critical weaknesses
  1. The judges are not specified or validated. All main results depend on LLM judges, but the paper never states which models these judges are, including the ones behind the reported interjudge agreement. Only the scope judgment is checked against a human, on 60 translations. The truth and content judgments are never checked against humans, and their interjudge agreement in the earlier run is low (Krippendorff's α between 0.467 and 0.538). Without knowing what the judges are and how accurate they are, the reported rates are hard to trust.

  2. What counts as an error is not well defined. The main results use the prompt "Rewrite the source statement in the language of the target framing," which never asks the model to keep the statement exactly as strong as the original. A statement can often be translated in more than one valid way, so a more general translation may just follow a reasonable reading of this prompt. The stricter prompt is tested on only one model, and the paper never reports how often it changes the objects the statement covers, which is the main measure. The benchmark's own reference translations keep the original objects in only 43.3% of the probability-to-measure-theory items. In addition, the paper never defines what it means for a translation to cover more or fewer objects than the original. This requires a mapping between the objects of the two subfields. The mapping is clear for vector spaces and modules, but for set theory and category theory it is a choice, and the paper leaves it to the judge.

  3. The writing also makes the paper hard to follow. Many terms and phrases are undefined or used before they are explained, and they are unfamiliar to the ML community. For example, Section 3.3 states "Widening is the most reliable code in the instrument, and unlike the composite scope ok used in earlier work it does not depend on the flag," but "scope ok" is not defined there. Section 3.4 refers to "register conformity", and Section 4.5 tests whether "pin presence responds to register pressure alone", but neither term is defined. Phrases such as "the cell that carries" in Section 3.3 are also unclear.

Questions for authors

Clarifications:

  1. Which models are used as judges for each table and for each agreement figure (κ = 1.000, κ = 0.900, and the α values in Appendix G)?
  2. How do you map objects between the two subfields when comparing what the original and the translation cover, especially for set theory and category theory? How is this decided when a statement quantifies over more than one kind of object?

Requests that could change my assessment: 3. Please report the truth and content judgments against human labels on a sample. 4. Please run the stricter prompt (condition B) on all seven models and report the same results as in Table 2. 5. Please report how often the MELD reference translations cover more objects than their sources, as a baseline for the models. 6. Please define all the terms the results depend on in the main text, before they are used, such as "scope ok", "flag", "register conformity", and "register pressure", and remove references to earlier versions and runs of the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.