- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper introduces DBE, a dataset grouping equivalent formulations of 3,000 theorems to improve mathematical retrieval, alongside natural-language queries for 235,930 Mathlib declarations. It trains a unified retriever for theorem equivalence, declaration lookup, and premise retrieval. Experiments on 8B and 0.6B models show improved retrieval performance, with the main 8B model raising MIRB macro nDCG@10 from 53.10 to 57.15. Matched ablations attribute equivalence and declaration-search gains to DBE, while broader gains mainly come from combining training relations rather than specialist distillation.
Critical strengths
- Equivalence-aware supervision addresses a concrete retrieval failure. DBE groups alternative theorem formulations into reviewed classes, allowing training to avoid treating known equivalents as negatives. Its practical value is supported by an 8.93-point improvement in MELD full MRR over the matched control without DBE, making the dataset a substantive contribution to mathematical retrieval.
- Matched ablations clarify what drives the improvements. Comparisons with and without DBE and specialist distillation distinguish gains from equivalence supervision, the broader training mixture, and teacher scores. This strengthens the empirical contribution by showing that DBE benefits specific retrieval tasks, while training across multiple relations accounts for most aggregate gains.
Critical weaknesses
- Insufficient documentation and validation of the central dataset contribution. Although Sections 3.1–3.5 describe DBE’s structure, the paper does not explain how the initial 3,000 anchor theorems were sourced or selected. The generation and review prompts are also missing, and choices such as producing literature-style and plain-language wordings receive little justification. These omissions are consequential because DBE’s central contribution depends on the correctness of its equivalence labels, which directly determine training positives and exclusions. The construction is insufficiently reproducible.
- The 8B and 0.6B studies use different multi-positive losses, training mixtures, distillation procedures, and evaluation protocols (Sections 4–5 and Appendix K). Although the paper documents these differences and provides useful matched controls within each setting, it does not adequately establish why the changes are necessary or whether the findings persist under a common training and evaluation protocol. Furthermore, the 0.6B evaluation suite was repeatedly used to select recipes and hyperparameters, limiting its value as independent evidence of generalization. The experiments establish results for separately developed configurations, while leaving uncertainty about which design choices are essential and how reliably the proposed approach transfers across settings.
Questions for authors
- How were the initial 3,000 anchor theorems sourced and selected?
- Beyond the reported audits identifying problems in approximately 4% of sampled anchors before correction, was any independent audit performed on the corrected dataset? Can you provide any statistics and quality of the final dataset?
- What motivated using different multi-positive losses and distillation procedures for the 8B and 0.6B models? Do any existing controlled comparisons isolate these choices while holding the data and evaluation protocol fixed?
- For the main 8B and 0.6B retrievers, which evaluation sets, if any, remained unused during recipe, hyperparameter, and checkpoint selection? Please distinguish those results from development results, since this affects how strongly the reported improvements support generalization.
- There are some valuable experiments to be put in the main text. Is there any to enrich the paper within the 9-page limit?
2 comments
Valuable experiments enrichment
Valuable experiments enrichment
Thank you for your constructive suggestion.
The controlled equivalent-label experiment is now presented in Section 6.2 (Figure 2, lines 354–377, p. 7) in the revised paper, where we discuss the effect of masking equivalents or using them as positives.
We also added MathNet-Retrieve and Theorem Search results to Section 6.1 (Table 3, lines 324–345, p. 7). MathNet-Retrieve hard-tier Recall@10 improves from 30.27% to 73.81%, and Theorem Search Hit@10 improves from 54.95% to 63.96%. Protocols are documented in Appendix G.2 (lines 929–959, p. 18).
The LoRA rank comparison is added to Appendix K.5 (Table 26, lines 1460–1470, p. 28) and the controlled three-arm loss/distillation comparison to Appendix K.6 (Tables 27–28, lines 1476–1532, pp. 28–29).
1. Anchor construction
During the construction of the dataset we realized: equivalence relationship is hard to mine from existing data at scale. Because to find equivalent pairs in existing datasets, we needs a strong model to have extensive reasoning on a large corpus to extract them. So it's much more efficient to generate and expand along the equivalence axis directly. We then suggested the model starting directions which it should cover; analysis, linear algebra, algebra, and suggested it to start from theories in Undergraduate Texts in Mathematics (UTM) to Graduate Texts in Mathematics (GTM).
2. Audit
We audited the entire corpus with GPT 6 Astra with the following prompt, 7 representations were found as not faithful equivalent statements to the anchor.
"Review DBE anchors [start]–[end] and all their representations. Read each anchor’s invariant and both the literature and natural-language wording of every representation. Determine whether each representation faithfully expresses the same mathematics as its anchor. Check the objects, hypotheses, quantifiers, existence, uniqueness, conclusions and parameter ranges. Explain how the actual conditions, solutions or structures correspond in both directions. Merely establishing that both statements are true is insufficient. Be sensible about standard implicit conventions and equivalent mathematical encodings. Do not silently add assumptions, exclude cases or replace the original objects. For each representation, report its anchor ID, representation ID, verdict and mathematical justification. Distinguish full equivalence from a specialization, generalization, consequence or incorrect statement. Give a counterexample or identify the precise missing implication where possible. Flag genuine problems in the original text and propose a minimal, sensible correction separately."
3. Loss and distillation.
The recipes were developed separately. We now test the two differences on 0.6B, changing one component at a time while keeping the data, initialization, batch order, teachers and evaluation fixed. Scores are nDCG@10 ×100:
The shared-denominator loss improves MELD while lowering the broad averages. KL aggregation has little effect here.
4. LoRA capacity exploration.
We used the 0.6B model to test on different adapter capacity because the appropriate number of trainable parameters for the size of our training data was uncertain. For the 0.6B model, we compared LoRA ranks 32, 64 and 128, keeping the data, batch order, and 604-update training fixed. Scores are nDCG@10 on a 0–100 scale:
These results suggest that adapter capacity was not a major bottleneck within this range, and increasing rank yielded modest aggregate improvements.