- Rating
- 2(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces TOME-Math, a large-scale mathematical proof-supervision dataset constructed from structured textbooks. The authors investigate its effectiveness through fine-tuning, structure-guided retrieval, curriculum learning, and cross-subject evaluations. While the work demonstrates substantial engineering effort and encouraging results, I have significant concerns about its scientific motivation, methodological justification, and whether the experiments adequately support its central claims.
Critical strengths
-
Substantial engineering effort: The construction of a dataset from over 2,000 textbooks across 29 mathematical subjects represents considerable engineering effort. The pipeline covers multiple processing stages and preserves potentially useful structural information.
-
Broad experimental coverage.: The paper evaluates multiple mathematical reasoning benchmarks and investigates curriculum learning, retrieval, and source diversity. Some controlled comparisons, particularly those involving curriculum ordering, provide useful empirical observations.
Critical weaknesses
-
Insufficient motivation and unclear scientific contribution. The paper does not clearly establish why losing textbook structure constitutes an important limitation for mathematical reasoning. The distinction between building a larger proof dataset and introducing a scientifically meaningful supervision paradigm remains insufficiently justified.
-
Insufficient methodological justification. Around Lines 184–209, multiple models are introduced without sufficient task-specific justification or validation of their capabilities. The Gap QA construction process also lacks clear criteria for identifying meaningful proof gaps and sufficient independent evidence of the correctness of generated proofs.
-
Insufficient evidence for the central claim. Although Section 5 provides useful empirical observations, the experiments do not sufficiently disentangle the effects of textbook structure from data content, diversity, and supervision volume. In particular, the retrieval comparisons conflate structural constraints with candidate selection, while the Gap QA ablation is not size-matched. Consequently, the unique contribution of structure-preserving supervision remains insufficiently established.
-
Evaluation reliability and protocol inconsistencies. Appendix D.1 states that MiniF2F and ProofNet use their official task definitions, checkers, and native metrics, whereas Appendix E.3 states that these benchmarks are evaluated as natural-language proof tasks using a Luna-based proxy judge rather than official formal checkers. These are materially different evaluation protocols, and the inconsistency directly affects the interpretation and comparability of the main benchmark results.
Questions for authors
-
Evaluation consistency: Appendix D.1 states that MiniF2F and ProofNet use official evaluators, while Appendix E.3 describes a Luna-based proxy judge. Which evaluation protocol was actually used to produce the main results?
-
Gap QA reliability: How do the authors distinguish genuine reasoning gaps from mathematically acceptable omissions? What evidence supports the correctness of the generated gap propositions and their proofs?
-
Model selection: How do the model assignments described in Section 3.1 correspond to the actual generation distributions reported in Section 3.2? What quantitative evidence supports these choices?
-
Attribution of improvements: Given the unmatched training-token budgets across baselines and the reduced data volume when removing Gap QA, what evidence supports attributing the improvements specifically to structured supervision rather than differences in data quantity or content?
Thank you to the authors for their detailed responses and active participation in the discussion. The rebuttal has addressed some of my concerns and helped clarify several aspects of the work. While some reservations remain, I acknowledge the authors' efforts and the additional evidence provided. Accordingly, I have raised my score from 1 to 2.