- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
To address the scarcity of research-level mathematical training data, incomplete problem context, and the difficulty of verifying solutions, this paper introduces ResearchMath-14K, a data construction and training framework. The framework uses multiple agents to extract problems from mathematical literature and supplement missing context, then combines teacher-generated reasoning trajectories with behavioral filtering to construct a large-scale research-level mathematical training corpus. Experiments across multiple mathematical benchmarks and three model families show that the framework yields larger average gains on graduate- and research-level mathematical tasks than the existing training datasets evaluated, and further improves model performance when combined with existing training data.
Critical strengths
- The paper addresses the important bottleneck of scarce research-level mathematical training data, providing a valuable resource for improving advanced mathematical reasoning in open models.
- The experiments span multiple model families and mathematical benchmarks at different difficulty levels, complemented by mixed-data training, human evaluation, and analyses of reasoning trajectory behavior and diversity.
- The paper presents a reasonably complete line of argument covering data construction, quality assessment, training effectiveness, and potential mechanisms, with implementation details and research limitations documented in the appendix.
Critical weaknesses
- The caption of Figure 4 reports 98,778,540 problem pairs, corresponding to the 14,056 problems remaining after deduplication. However, the text in Appendix G.1 states that the figure shows the similarity distribution of the original 20,835 problems. Please check whether this discrepancy is a typo or an issue with the description, and clarify which dataset was used.
- What criteria define a “research-level mathematical problem” in the Introduction? Please provide explicit criteria for determining whether a problem qualifies as research-level.
- How are “open,” “unknown,” and “partially solved” distinguished in Section 3.1? Please clarify the definitions of these categories and conduct a small-scale validation study to assess the accuracy of the assigned labels.
- What is the practical meaning of the difficulty Elo scores in Section 3.3 and Figure 2? Does a difference of approximately 400 Elo points merely indicate that the judge model more frequently considers one dataset harder, or does it correspond to a measurable difference in actual problem-solving difficulty? How do these scores relate to solving success rates, required mathematical knowledge, and proof complexity? Further clarification would be helpful.
- What specifically constitutes “complementary supervision” in Section 6 and Figure 3? Which aspects of the supervision are complementary—for example, mathematical knowledge, reasoning strategies, or problem distributions? Please elaborate.
Questions for authors
Please refer to Weaknesses.