acceptodds
Desk rejected at ICLR 2027

Scaling Mathematical Proof Supervision from Structured Textbooks

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Mathematical supervision for language models is typically flattened into isolated problem–solution pairs, discarding the curricular organization and evidential structure of the original sources. We introduce TOME-Math (Textbook-Oriented Mathematical Evidence), a large-scale corpus that converts 2,118 textbooks spanning the core areas of university-level mathematics subjects into approximately 380K structured proof-supervision records while preserving chapter order, proposition–proof links, dependency references, and source provenance. Leveraging these structural signals, we explore curriculum learning across multiple mathematical subjects and characterize scaling patterns with supervision volume and source breadth. At full scale, fine-tuning on TOME-Math raises the unweighted mean score across our held-out evaluations and established public benchmarks by 16.7 percentage points over the base model, while exceeding all three public instruction-data baselines evaluated in aggregate. Our findings highlight structured textbooks as both a scalable source of mathematical supervision and an empirical framework for studying the impact of data organization on mathematical reasoning.

Primary area:
datasets and benchmarks
Keywords:
math textbook, math datasets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_Da1tOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces TOME-Math, a large-scale mathematical proof-supervision dataset constructed from structured textbooks. The authors investigate its effectiveness through fine-tuning, structure-guided retrieval, curriculum learning, and cross-subject evaluations. While the work demonstrates substantial engineering effort and encouraging results, I have significant concerns about its scientific motivation, methodological justification, and whether the experiments adequately support its central claims.

Critical strengths
  1. Substantial engineering effort: The construction of a dataset from over 2,000 textbooks across 29 mathematical subjects represents considerable engineering effort. The pipeline covers multiple processing stages and preserves potentially useful structural information.

  2. Broad experimental coverage.: The paper evaluates multiple mathematical reasoning benchmarks and investigates curriculum learning, retrieval, and source diversity. Some controlled comparisons, particularly those involving curriculum ordering, provide useful empirical observations.

Critical weaknesses
  1. Insufficient motivation and unclear scientific contribution. The paper does not clearly establish why losing textbook structure constitutes an important limitation for mathematical reasoning. The distinction between building a larger proof dataset and introducing a scientifically meaningful supervision paradigm remains insufficiently justified.

  2. Insufficient methodological justification. Around Lines 184–209, multiple models are introduced without sufficient task-specific justification or validation of their capabilities. The Gap QA construction process also lacks clear criteria for identifying meaningful proof gaps and sufficient independent evidence of the correctness of generated proofs.

  3. Insufficient evidence for the central claim. Although Section 5 provides useful empirical observations, the experiments do not sufficiently disentangle the effects of textbook structure from data content, diversity, and supervision volume. In particular, the retrieval comparisons conflate structural constraints with candidate selection, while the Gap QA ablation is not size-matched. Consequently, the unique contribution of structure-preserving supervision remains insufficiently established.

  4. Evaluation reliability and protocol inconsistencies. Appendix D.1 states that MiniF2F and ProofNet use their official task definitions, checkers, and native metrics, whereas Appendix E.3 states that these benchmarks are evaluated as natural-language proof tasks using a Luna-based proxy judge rather than official formal checkers. These are materially different evaluation protocols, and the inconsistency directly affects the interpretation and comparability of the main benchmark results.

Questions for authors
  1. Evaluation consistency: Appendix D.1 states that MiniF2F and ProofNet use official evaluators, while Appendix E.3 describes a Luna-based proxy judge. Which evaluation protocol was actually used to produce the main results?

  2. Gap QA reliability: How do the authors distinguish genuine reasoning gaps from mathematically acceptable omissions? What evidence supports the correctness of the generated gap propositions and their proofs?

  3. Model selection: How do the model assignments described in Section 3.1 correspond to the actual generation distributions reported in Section 3.2? What quantitative evidence supports these choices?

  4. Attribution of improvements: Given the unmatched training-token budgets across baselines and the reduced data volume when removing Gap QA, what evidence supports attributing the improvements specifically to structured supervision rather than differences in data quantity or content?

1 comment

Thank you to the authors for their detailed responses and active participation in the discussion. The rebuttal has addressed some of my concerns and helped clarify several aspects of the work. While some reservations remain, I acknowledge the authors' efforts and the additional evidence provided. Accordingly, I have raised my score from 1 to 2.

Reviewer_XgwHOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TOME-Math, a proof-supervision corpus of 380,292 QA records (205,861 Concept QA + 174,431 Gap QA) constructed from 2,118 university-level mathematics textbooks via a structure-preserving pipeline, retaining chapter order, proposition–proof links, dependency references, and source provenance. Validation proceeds in three parts: (1) full-corpus SFT on Qwen3-30B-A3B-Base raises the six-benchmark average (TOME-LONGFORM/OBJECTIVE, MiniF2F, ProofNet, MATH-500, AIME-26) from 29.17% to 45.83%, outperforming OpenMathInstruct-1 (1.82M examples), MetaMathQA, and MathInstruct in aggregate with roughly half the training tokens; (2) controlled structure studies show training-free structure-guided retrieval improves pass@1 from 13.25% to 17.38% under a matched context budget, forward textbook curricula (FORWARD–GAP) win in all 18 subject–scale cells, and broader source coverage at a fixed token budget yields +3.2 points on average; (3) the natural-language proof evaluation uses an eight-review pessimistic verifier with a 600-response human calibration study.

Critical strengths
  1. Fills a genuine gap: the first large-scale proof-supervision corpus that preserves structure.** Unlike flat problem–solution corpora (MATH, MetaMath, OpenMathInstruct), TOME-Math uniquely combines 2,118-textbook scale with chapter order, a proposition dependency graph, and provenance (Table 1 makes this contrast clear). The Gap QA stream — making intermediate proof obligations that textbooks leave implicit into explicit supervision — is a clever and novel supervision form.
  2. Controlled experimental design well above the norm. Curriculum schedules within each cell use identical QA IDs, token counts (±33 tokens), and hyperparameters, with each pairwise comparison isolating exactly one factor; source breadth is token-matched. The FORWARD–GAP advantage over SHUFFLE holds in all 18 cells and widens with scale — strong evidence for the paper's causal claim about data organization.
  3. The curriculum finding is consistent across subjects and scales. FORWARD–GAP wins in all 18 cells, and its margin over SHUFFLE widens with scale (4.34 → 7.72 points). This cross-subject, cross-scale uniformity is more persuasive than any single-point gain.
Critical weaknesses
  1. Internal contradiction about formal-benchmark scoring. Appendix D.1 claims MiniF2F/ProofNet use the "official ... checker and native metric," while E.3 says they use the "Luna proxy judge rather than their official formal checkers." If the proxy judge was used, the numbers are incomparable to published results, and TOME-Math's lead may partly reflect judge affinity for TOME-style long-form proofs — weakening the "best on five of six" headline.
  2. Single base model and unexplained curriculum-effect reversal. All experiments use Qwen3-30B-A3B-Base, so generality is unknown. Worse, Section 5.2 finds chapter direction primary and gap-first secondary, while Section 5.4 finds the opposite; the paper notes but does not explain this reversal, and all curriculum cells are single runs without seeds.
  3. Training-data quality is questionable for nearly half the corpus.** Appendix B.3 reports groundedness/evidence-alignment pass rates below 50% and a "no verbatim evidence" flag covering 48.4% of records, yet no human error-rate estimate on the final training set and no quality-tier-stratified training ablation are provided.
Questions for authors
  1. Please resolve the D.1 vs. E.3 contradiction on MiniF2F/ProofNet scoring. If the Luna proxy judge was used, can you provide numbers under the official formal checkers? If those change, does the "best on five of six" conclusion in Table 2 survive?
  2. Can you replicate the core findings on at least one additional base model (different family or scale)? Even replicating just Table 2 or a single curriculum conclusion would be valuable.
  3. Appendix B.3 reports that the "no verbatim evidence" human-review flag covers 48.4% of the corpus and that groundedness/evidence-alignment pass rates are below 50%. What is the human-checked mathematical error rate on a random sample of the final training set, and does a quality-tier-stratified training ablation show whether low-tier data helps or hurts?
Reviewer_xv6ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces TOME-Math, approximately 380K natural-language mathematical QA records constructed from 2,118 textbooks. It extracts proposition–proof pairs as Concept QA and uses LLMs to turn omitted reasoning steps into standalone Gap QA. SFT with textbook-based curricula improves performance on book-held-out exercises and public mathematical benchmarks. Controlled experiments also investigate training order, structure-guided retrieval, and source breadth.

Critical strengths
  1. A substantial textbook-derived resource. The dataset preserves chapter positions, proposition references, and source provenance, supporting both proof supervision and studies of data organization.
  2. Controlled curriculum experiments. Comparisons using identical examples and token budgets provide evidence that textbook progression and Gap-first ordering can improve training outcomes.
Critical weaknesses
  1. Gap QA quality is insufficiently characterized. The paper provides source-conditioned auditing and corpus-level quality analyses, but does not separately establish the mathematical correctness of generated Gap QA. The detailed human calibration concerns evaluation responses rather than synthetic training records.
  2. The benefit of standalone Gap QA remains unclear. Removing Gap QA also reduces training volume and removes the additional reasoning content. This does not establish whether separate Gap tasks are more effective than incorporating the same reasoning directly into Concept QA proofs.
Questions for authors
  1. Is retained Gap QA auditted by human? Please report whether the generated statements preserve necessary assumptions, their proofs are valid, and they actually bridge the identified gaps. How reliable is the automatic judge on these records?
  2. Appendix D.1 and D.5 state that MiniF2F and ProofNet use their official evaluators, whereas Appendix E.3 states that they are evaluated as natural-language proof tasks using the Luna judge rather than formal checkers. Which protocol produced the reported results?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.