acceptodds
Under review as a conference paper at ICLR 2027

Mathematics Modulo Equivalence: Theorem Classes as Retrieval Supervision

Abstract

The same theorem can be stated in very different mathematical language. Cayley–Hamilton, for instance, can be phrased through the annihilator ideal of an operator or as a linear recurrence along its orbits. Text embedders often fail to retrieve one such formulation from the other, and retrieval datasets rarely record which statements are equivalent, so contrastive training can push equivalent statements apart as negatives. We build Different but Equivalent (DBE), a dataset organized around reviewed theorem classes: 3,000 anchor theorems, 9,385 equivalent formulations in 18,770 wordings, and 62,514 search queries linked to their targets. Class membership tells the contrastive loss which other statements are also correct answers. For formal retrieval, we pair 235,930 Mathlib declarations with natural-language search queries, each written by a language model for its declaration, and link 1,447 DBE anchors to declarations. On Qwen3-Embedding-8B, one LoRA trained on DBE, Lean declaration lookup, and premise retrieval, with scores distilled from relation-specific specialists, reaches 57.15 macro nDCG@10 on 13 MIRB tasks, against 53.10 for the base model and 53.98 for MathLeap-Qwen-8B. Removing DBE from this recipe lowers MELD full MRR by 8.93 points but the MIRB average by only 0.18. Matched controls trained without teacher scores, at 8B and on Qwen3-Embedding-0.6B, come within 0.28 points of the distilled models on broad averages, so most of the broad gain comes from the multi-relation training data rather than from distillation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
mathematical information retrieval, theorem retrieval, text embeddings, formal mathematics, Lean, representation learning, contrastive learning, low-rank adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_LvvROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces DBE, a dataset grouping equivalent formulations of 3,000 theorems to improve mathematical retrieval, alongside natural-language queries for 235,930 Mathlib declarations. It trains a unified retriever for theorem equivalence, declaration lookup, and premise retrieval. Experiments on 8B and 0.6B models show improved retrieval performance, with the main 8B model raising MIRB macro nDCG@10 from 53.10 to 57.15. Matched ablations attribute equivalence and declaration-search gains to DBE, while broader gains mainly come from combining training relations rather than specialist distillation.

Critical strengths
  1. Equivalence-aware supervision addresses a concrete retrieval failure. DBE groups alternative theorem formulations into reviewed classes, allowing training to avoid treating known equivalents as negatives. Its practical value is supported by an 8.93-point improvement in MELD full MRR over the matched control without DBE, making the dataset a substantive contribution to mathematical retrieval.
  2. Matched ablations clarify what drives the improvements. Comparisons with and without DBE and specialist distillation distinguish gains from equivalence supervision, the broader training mixture, and teacher scores. This strengthens the empirical contribution by showing that DBE benefits specific retrieval tasks, while training across multiple relations accounts for most aggregate gains.
Critical weaknesses
  1. Insufficient documentation and validation of the central dataset contribution. Although Sections 3.1–3.5 describe DBE’s structure, the paper does not explain how the initial 3,000 anchor theorems were sourced or selected. The generation and review prompts are also missing, and choices such as producing literature-style and plain-language wordings receive little justification. These omissions are consequential because DBE’s central contribution depends on the correctness of its equivalence labels, which directly determine training positives and exclusions. The construction is insufficiently reproducible.
  2. The 8B and 0.6B studies use different multi-positive losses, training mixtures, distillation procedures, and evaluation protocols (Sections 4–5 and Appendix K). Although the paper documents these differences and provides useful matched controls within each setting, it does not adequately establish why the changes are necessary or whether the findings persist under a common training and evaluation protocol. Furthermore, the 0.6B evaluation suite was repeatedly used to select recipes and hyperparameters, limiting its value as independent evidence of generalization. The experiments establish results for separately developed configurations, while leaving uncertainty about which design choices are essential and how reliably the proposed approach transfers across settings.
Questions for authors
  1. How were the initial 3,000 anchor theorems sourced and selected?
  2. Beyond the reported audits identifying problems in approximately 4% of sampled anchors before correction, was any independent audit performed on the corrected dataset? Can you provide any statistics and quality of the final dataset?
  3. What motivated using different multi-positive losses and distillation procedures for the 8B and 0.6B models? Do any existing controlled comparisons isolate these choices while holding the data and evaluation protocol fixed?
  4. For the main 8B and 0.6B retrievers, which evaluation sets, if any, remained unused during recipe, hyperparameter, and checkpoint selection? Please distinguish those results from development results, since this affects how strongly the reported improvements support generalization.
  5. There are some valuable experiments to be put in the main text. Is there any to enrich the paper within the 9-page limit?
2 comments

1. Anchor construction

During the construction of the dataset we realized: equivalence relationship is hard to mine from existing data at scale. Because to find equivalent pairs in existing datasets, we needs a strong model to have extensive reasoning on a large corpus to extract them. So it's much more efficient to generate and expand along the equivalence axis directly. We then suggested the model starting directions which it should cover; analysis, linear algebra, algebra, and suggested it to start from theories in Undergraduate Texts in Mathematics (UTM) to Graduate Texts in Mathematics (GTM).

2. Audit

We audited the entire corpus with GPT 6 Astra with the following prompt, 7 representations were found as not faithful equivalent statements to the anchor.

"Review DBE anchors [start]–[end] and all their representations. Read each anchor’s invariant and both the literature and natural-language wording of every representation. Determine whether each representation faithfully expresses the same mathematics as its anchor. Check the objects, hypotheses, quantifiers, existence, uniqueness, conclusions and parameter ranges. Explain how the actual conditions, solutions or structures correspond in both directions. Merely establishing that both statements are true is insufficient. Be sensible about standard implicit conventions and equivalent mathematical encodings. Do not silently add assumptions, exclude cases or replace the original objects. For each representation, report its anchor ID, representation ID, verdict and mathematical justification. Distinguish full equivalence from a specialization, generalization, consequence or incorrect statement. Give a counterexample or identify the precise missing implication where possible. Flag genuine problems in the original text and propose a minimal, sensible correction separately."

3. Loss and distillation.

The recipes were developed separately. We now test the two differences on 0.6B, changing one component at a time while keeping the data, initialization, batch order, teachers and evaluation fixed. Scores are nDCG@10 ×100:

Gold lossDBE distillationMacro19MIRB13MELD
Independent-positivePositive-pooled KL44.9349.7944.79
Shared-denominatorPositive-pooled KL43.7848.5547.14
Independent-positiveCandidate-wise KL44.8949.7444.64

The shared-denominator loss improves MELD while lowering the broad averages. KL aggregation has little effect here.

4. LoRA capacity exploration.

We used the 0.6B model to test on different adapter capacity because the appropriate number of trainable parameters for the size of our training data was uncertain. For the 0.6B model, we compared LoRA ranks 32, 64 and 128, keeping the data, batch order, and 604-update training fixed. Scores are nDCG@10 on a 0–100 scale:

RankMacro19MIRB13MELD
3244.3949.3742.97
6444.6349.4443.38
12844.8249.6045.48

These results suggest that adapter capacity was not a major bottleneck within this range, and increasing rank yielded modest aggregate improvements.

AuthorsOpenReview ↗

Valuable experiments enrichment

Valuable experiments enrichment

Thank you for your constructive suggestion.

The controlled equivalent-label experiment is now presented in Section 6.2 (Figure 2, lines 354–377, p. 7) in the revised paper, where we discuss the effect of masking equivalents or using them as positives.

We also added MathNet-Retrieve and Theorem Search results to Section 6.1 (Table 3, lines 324–345, p. 7). MathNet-Retrieve hard-tier Recall@10 improves from 30.27% to 73.81%, and Theorem Search Hit@10 improves from 54.95% to 63.96%. Protocols are documented in Appendix G.2 (lines 929–959, p. 18).

The LoRA rank comparison is added to Appendix K.5 (Table 26, lines 1460–1470, p. 28) and the controlled three-arm loss/distillation comparison to Appendix K.6 (Tables 27–28, lines 1476–1532, pp. 28–29).

Reviewer_xYPKOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper addresses a limitation of existing retrieval datasets for formal mathematics: semantically equivalent mathematical statements can have different syntactic representations, making them difficult to identify using embedding retrieval methods. The authors argue that existing retrieval models struggle to recognize such equivalences. The authors introduce DBE, a dataset containing 3,000 mathematical theorems and 9,385 equivalent formalizations, together with corresponding search queries. The dataset is constructed using LLM-generated equivalent formulations and includes links to Mathlib. The authors use DBE to fine-tune a Qwen3 embedding model, aiming to improve retrieval of semantically equivalent but syntactically different formal statements. They evaluate the resulting model on retrieval tasks and report performance improvements over the pretrained Qwen3 embedding model.

Critical strengths
  • The study addresses the relevant and timely issue of semantically equivalent but syntactically different formal mathematical statements.
  • The research direction and experimental results are promising.
Critical weaknesses
  • Figure 1 is confusing and potentially misleading. The y axis is log scaled but does not explicitly states its scaling. The “equivalent-formulation rank” metric is unclear and not explained in the paragraph that introduces the figure (lines 49-81) or the caption. The x-axis is cropped. Some text in the figure is not readable.
  • In Figure 1 (b) it is unclear what the objective of the retriever is. The authors do not explain why a semantically similar but not equivalent theorem is labeled as an incorrect retrieval result.
  • The reported quality control measures reported in Section 3.2 raise concerns regarding the validity of the dataset. The authors report that the human reviewers only review a small subset and find issues in 4% of the samples. However, the authors trust the LLM generated data with the justification that the reviewers are weaker mathematicians than the LLM (lines 197-204).
  • The dataset lacks documentation, details and quality control. The dataset is fully LLM generated and the authors only report minimal quality control, lacking statistics about how example were filtered, how many were removed, etc.
  • The study is hard to follow, as the authors use abbreviations without introducing them first and use many abbreviations. Sentences within the same paragraph are seemingly disconnected, making it difficult to read and follow a train of thought.
  • The manuscript requires thorough restructuring. The introduction mixes motivation with details about the experimental setup and its results.
  • The study repeats the same information multiple times without adding further details.
  • The presentation of the experimental setup and results (specifically the tables) is difficult to follow and the discussion is not well structured and unclear.
  • The study is weakly embedded in existing literature. In the introduction, the authors claim its primary use if for mathematicians and student to verify existing results and as a search index, but do not motivate its application in automated formalisation models or theorem-provers.
  • the performance improvements seem marginal compared to qwen 3 embedding base model, considering the authors claim that existing retrieval models fail to retrieval equivalent but syntactically different theorems.
Questions for authors
  • From which datasets did you extract the theorems included in DBE?
  • How did you extract and verify equivalent formalisations for the theorems in DBE?
  • What are the “links” between the theorems in DBE and Mathlib?
  • What are the 226 Lean wrappers for these “links” and how do you generate them (line 98-99)?
  • What is the motivation for using a student-teacher setup when training the retrieval model?
  • How can the retrieval model performance for semantically equivalent but syntactically different Lean statements be improved?
  • Are there common differences between different equivalent formalisations that can be categorized, i.e. fundamentally different proofs and minor syntactic changes?
  • How do you ensure equivalent formalisations are exactly equivalent (i.e. not introducing additional constraints) and faithful to the informal statement?
1 comment

Thank you for the detailed review.

We revised Figure 1(a) and Figure 1(b) now shows how DBE groups formulations into a theorem class. We also reorganized the introduction and motivates the design of the dataset. Abbreviations are used much less. And we expanded on declaration search and premise selection in Related Work.

How were the theorems and their formulations obtained?

Mining equivalent statements from a large corpus requires identifying candidate pairs and checking that they express the same theorem. So it's much more efficient to generate and expand along the equivalence axis directly. We then suggested the model starting directions which it should cover; analysis, linear algebra and suggested it to start from theories in Undergraduate Texts in Mathematics (UTM) to Graduate Texts in Mathematics (GTM). We recorded the mathematical argument connecting each one to the anchor. This let us construct equivalence classes directly.

What are the Mathlib links and wrappers?

A link identifies a Mathlib declaration and records how it relates to a particular DBE formulation. For example, the epsilon–delta formulation of continuity links to Metric.continuous_iff. It was first introduced to see if we could bring the abstraction of a theorem class up a level, and test whether this could support a smaller dataset for the sake of retrieval and still get the same improvement. However mathlib lack the exact formalisation. Of the 1,447 links, 787 match a formulation and 660 identify a related result, such as a special case. And some existing mathlib lemmas would need to be wrapped to express the same relevant claim. We tested these links in a matched 0.6B experiment. Adding the linked Lean targets raised little improvements and lowered mean nDCG@10 across 19 task. So we kept the links only in the auxiliary experiments. For the 8B model training we used the separate 230k query–declaration pairs for declaration lookup. (Appendices K.2 and K.7, Tables 19 and 29).

Why use specialist teachers?

Theorem-equivalence retrieval asks for encoding the math relationship instead of textual overlap. Declaration lookup rewards finding the particular Lean declaration a query asks for. We wanted one retriever to retain the strengths of all specialists, so we trained the student on gold labels and the specialists’ candidate scores. In the matched 8B comparison, adding teacher scores raised MELD full MRR from 55.67 to 58.68 and MathlibQR nDCG@10 from 50.25 to 52.75 (Section 6.3, Table 4).

Retrieving equivalent Lean statements

We found that an informal description helped even when the query used Lean notation. We held Qwen3-Embedding-8B fixed and varied how we indexed the same Mathlib declarations. Across 170 Lean-style MathlibQR queries, the target appeared in the top ten for 93 queries (54.7%) with the formal declaration alone, 82 (48.2%) with the informal description alone, and 100 (58.8%) with both.

Types of equivalent formulations in DBE

Yes, there are, and could be classified mainly into five categories:

  1. Logical or definitional restatements of the same claim.
  2. Alternative encodings or parametrizations of the same mathematical object.
  3. Equivalent criteria for a property, including obstruction and optimality conditions.
  4. Alternative formulas, recurrences, or algorithms for the same quantity.
  5. Formulations of the same theorem in different branches of mathematics.

equivalence and faithfulness?

We audited the entire corpus with GPT 6 Astra with the following prompt, only 7 representations were found as not faithful equivalent statements to the anchor.

"Review DBE anchors [start]–[end] and all their representations. Read each anchor’s invariant and both the literature and natural-language wording of every representation. Determine whether each representation faithfully expresses the same mathematics as its anchor. Check the objects, hypotheses, quantifiers, existence, uniqueness, conclusions and parameter ranges. Explain how the actual conditions, solutions or structures correspond in both directions. Merely establishing that both statements are true is insufficient. Be sensible about standard implicit conventions and equivalent mathematical encodings. Do not silently add assumptions, exclude cases or replace the original objects. For each representation, report its anchor ID, representation ID, verdict and mathematical justification. Distinguish full equivalence from a specialization, generalization, consequence or incorrect statement. Give a counterexample or identify the precise missing implication where possible. Flag genuine problems in the original text and propose a minimal, sensible correction separately."

Reviewer_nUAoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper tries to build a math-aware retrieval system. The main issue the authors attempt to solve is that embeddings from models like Qwen3-Embedding-8B often fail to retrieve one formulation of a theorem from another, because in contrastive training, equivalent statements can end up as in-batch negatives. Therefore, the authors built a dataset (DBE) of theorem classes with query–anchor pairs, and in addition they built a Lean dataset. Then they fine-tuned Qwen3-Embedding-8B with LoRA, which improved the MIRB average from 53.10 to 57.15.

Critical strengths
  • Valuable dataset construction. The authors built Different but Equivalent (DBE) around theorem classes, where each theorem is grouped with its equivalent formulations, so equivalent statements are not treated as negatives during training. Their results are supported with empirical experiments that show this helps, improving cross-formulation retrieval.
Critical weaknesses

1- The method is a careful combination of known techniques multi-positive contrastive learning, masking known positives as in LeanHammer, and distillation from specialist models) rather than a new approach. The main contribution is the data, especially DBE, which is a distinctive and potentially useful resource; however, its value is limited by its size and it's fully synthetic

2- the lack of evaluation on independent human-curated benchmarks such as Theorem Search queries or MathNet-Retrieve.

referenced papers: LeanHammer (Zhu et al., ICLR 2026, arXiv:2506.07477), Theorem Search (Alexander et al., 2026, arXiv:2602.05216), MathNet (Alshammari et al., ICLR 2026, arXiv:2604.18584), ARQMath (Zanibbi et al., CLEF 2020), NTCIR-12 MathIR (Zanibbi et al., NTCIR 2016)

Questions for authors
  • Suggestion: How would your fine-tuned model generalize to other recent mathematical retrieval benchmarks such as Theorem Search (Alexander et al., 2026) and MathNet (Alshammari et al., 2026), or to older standard math IR benchmarks such as ARQMath and the NTCIR Math tasks? These benchmarks use human-written queries or human-curated pairs, so they would show whether the gains hold beyond data written by the same model family used to build DBE.

  • Question: Any interesting insights from the dataset construction or results? For example, were some types of similarity more influential for fine-tuning than others? What kind of cases does the fine-tuned model fail at?

2 comments
AuthorsOpenReview ↗

MathNet and Theorem Search results

Thank you for your time and kind suggestions.

1. Generalization to other math retrieval benchmarks.

MathNet-Retrieve (Alshammari et al., 2026): all 15,000 queries, 117,088 documents. Both models use the instruction "Given a mathematical query, retrieve mathematical content that answers the query," and identical corpus and scoring settings; Recall@1 / @5 / @10 (%):

TierQwen3-Embedding-8B (our run)C1C_1
Easy16.23 / 86.53 / 94.0328.98 / 91.97 / 98.44
Medium3.88 / 78.41 / 90.874.46 / 79.11 / 95.15
Hard0.01 / 4.65 / 30.270.24 / 26.50 / 73.81

Theorem Search (Alexander et al., 2026): 111 human-written queries. The public release has only permissively licensed papers (1.34M of the 9.2M theorems), so only 15 of the 111 target theorems exist in the public corpus. But each query identifies its arXiv paper and theorem number. We recovered the missing targets from the cited arXiv paper versions and checked their identities against the PDFs. We also included other extracted theorem statements from those papers as retrieval candidates. Both models search the same augmented corpus of 1,343,104 raw theorem statements using the authors’ query and document instructions.

ModelHit@1(%)Hit@10(%)Hit@20(%)MRR@20
Qwen3-Embedding-8B25.2354.9565.770.366
C1C_127.9363.9671.170.393

(Absolute numbers are not comparable with the original paper, which they search 9.2M theorems through LLM-written slogans; we search 1.34M theorem statements directly as they are written in the papers.)

2. Insights.

We want the embedding model to be trained to encode mathematical structure, rather than surface-level textual overlap. That motivates us to design a dataset to capture the deep structural relationship of mathematical equivalence.

A really interesting observation from the construction of the dataset: equivalence relationship is hard to mine from existing data at scale. Finding equivalent pairs in existing datasets needs a strong model to compare every pair of statements which it read and reason on it. So it's much more efficient to generate and expand along the equivalence axis directly.

Another interesting insight from the results is that fine-tuning separates genuinely related theorems from randomly sampled background statements, so targets rise in rank. For example, on Theorem Search, Hit@10 rises from 55.0% to 64.0%, while the median target–competitor margin remains essentially unchanged. Targets and close competitors gain similar amounts of cosine similarity (+0.046 and +0.040), while random-background similarity falls by 0.109. The improvement is consistent with better localization of the relevant mathematical neighborhood, with ambiguity within that neighborhood persisting.

The results of MathNet-Retrieve and Theorem Search are now added to Section 6.1 (Table 3, lines 324–345, p. 7).

MathNet-Retrieve hard-tier Recall@10 improves from 30.27% to 73.81%, and Theorem Search Hit@10 improves from 54.95% to 63.96%. Protocols are documented in Appendix G.2 (lines 929–959, p. 18).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.