acceptodds
Under review as a conference paper at ICLR 2027

ResearchMath-14K: Scaling Research-Level Mathematics Via Agents

Abstract

The frontier of mathematics is defined by problems whose solutions are not yet known. However, whether language models can meaningfully engage with such problems without human intervention remains unclear. A major obstacle is the lack of large-scale research-level math datasets. To this end, we introduce ResearchMath-14k, a set of problems curated from academic sources via a multi-agent pipeline. ResearchMath-14k spans 11 mathematical domains and ranks above existing math datasets on knowledge, novelty, and procedural difficulty. To our knowledge, it is the largest research-level mathematical problem set available for training. We additionally generate K teacher trajectories and use prompting and behavioral filtering. Notably, however, generating correct trajectories is nontrivial at this level, and two LLM judges label only and of sampled ResearchMath training trajectories as correct. Nevertheless, across three model families, full-parameter training on ResearchMath improves performance on graduate- and research-level mathematics benchmarks by points over the starting checkpoints. In comparison, training on existing datasets such as DASD and Nemotron-SFT-Math-v4 changes performance by and points, respectively. Notably, mixing DASD with ResearchMath yields higher scores than token-matched DASD alone on benchmarks covering olympiad short-form (), graduate- and research-level short-form (), graduate- and research-level symbolic (), and proof evaluation (). Further analysis suggests that research-level mathematical content and greater reasoning diversity may help explain these gains. These broad gains indicate that ResearchMath provides complementary supervision to contemporary datasets. We make publicly available for future works on research-level mathematical reasoning.

Primary area:
datasets and benchmarks
Keywords:
Research-Level Math, AI For Math, Datasets

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 5 reviews, above 27% of the others.
Reviewer_QfP3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

To address the scarcity of research-level mathematical training data, incomplete problem context, and the difficulty of verifying solutions, this paper introduces ResearchMath-14K, a data construction and training framework. The framework uses multiple agents to extract problems from mathematical literature and supplement missing context, then combines teacher-generated reasoning trajectories with behavioral filtering to construct a large-scale research-level mathematical training corpus. Experiments across multiple mathematical benchmarks and three model families show that the framework yields larger average gains on graduate- and research-level mathematical tasks than the existing training datasets evaluated, and further improves model performance when combined with existing training data.

Critical strengths
  1. The paper addresses the important bottleneck of scarce research-level mathematical training data, providing a valuable resource for improving advanced mathematical reasoning in open models.
  2. The experiments span multiple model families and mathematical benchmarks at different difficulty levels, complemented by mixed-data training, human evaluation, and analyses of reasoning trajectory behavior and diversity.
  3. The paper presents a reasonably complete line of argument covering data construction, quality assessment, training effectiveness, and potential mechanisms, with implementation details and research limitations documented in the appendix.
Critical weaknesses
  1. The caption of Figure 4 reports 98,778,540 problem pairs, corresponding to the 14,056 problems remaining after deduplication. However, the text in Appendix G.1 states that the figure shows the similarity distribution of the original 20,835 problems. Please check whether this discrepancy is a typo or an issue with the description, and clarify which dataset was used.
  2. What criteria define a “research-level mathematical problem” in the Introduction? Please provide explicit criteria for determining whether a problem qualifies as research-level.
  3. How are “open,” “unknown,” and “partially solved” distinguished in Section 3.1? Please clarify the definitions of these categories and conduct a small-scale validation study to assess the accuracy of the assigned labels.
  4. What is the practical meaning of the difficulty Elo scores in Section 3.3 and Figure 2? Does a difference of approximately 400 Elo points merely indicate that the judge model more frequently considers one dataset harder, or does it correspond to a measurable difference in actual problem-solving difficulty? How do these scores relate to solving success rates, required mathematical knowledge, and proof complexity? Further clarification would be helpful.
  5. What specifically constitutes “complementary supervision” in Section 6 and Figure 3? Which aspects of the supervision are complementary—for example, mathematical knowledge, reasoning strategies, or problem distributions? Please elaborate.
Questions for authors

Please refer to Weaknesses.

Reviewer_vCvYOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces ResearchMath-14K, a dataset of 14,056 research-level mathematical problems spanning 11 domains. The authors use a two-stage agent pipeline to extract questions from mathematical literature and open-problem collections, reconstruct the necessary context, and rewrite the questions into standalone problems. They additionally generate 220K teacher reasoning trajectories and construct a 77K training subset through length and behavioral filtering, without filtering for mathematical correctness. Despite the low correctness rates estimated by two LLM judges, training on these trajectories improves performance on graduate- and research-level mathematics benchmarks across three student model families. Mixing ResearchMath with DASD also outperforms token-matched DASD-only training across several evaluation categories. The paper investigates research-level mathematical content and reasoning diversity as possible explanations for these benefits.

Critical strengths
  1. The paper addresses a meaningful gap in mathematical training data. Existing mathematical post-training resources largely emphasize competition-style or answer-verifiable problems, while research questions are often scattered across specialized literature and depend heavily on surrounding definitions and assumptions. This work tackles the practical challenge of making those questions usable for training. The resulting collection of 14,056 problems across 11 mathematical domains is a substantial resource, and the effort to reconstruct context makes the contribution more useful than simply aggregating excerpts from papers.
  2. The central empirical finding is interesting and worth investigating. The paper shows that research-level reasoning attempts can provide useful supervision even when most sampled trajectories are not judged mathematically correct. This raises an important question about what models learn from difficult mathematical content beyond complete, verified solutions. Although the experiments do not fully isolate the source of the gains, the observed usefulness of this training data is itself a valuable finding. I appreciate that the authors report the low judged correctness rates directly rather than presenting the generated trajectories as reliable solutions.
  3. The mixed-data experiments provide compelling evidence of complementary training value. In particular, the comparison between DASD-only scaling and a token-matched mixture of DASD and ResearchMath is informative. The mixture improves performance across all three student models and all four evaluation categories, including short-answer, symbolic-answer, and proof tasks. These results make a stronger case for the dataset’s practical usefulness than improvements over the starting checkpoints alone: they suggest that ResearchMath contributes training content that is not fully supplied by simply adding more DASD data.
  4. The evaluation examines both the benefits and the boundaries of the dataset. The main experiments cover three model families and distinguish olympiad-level performance from graduate- and research-level performance. This helps reveal where the data is most useful, rather than relying only on an aggregate score. The results show a meaningful pattern: ResearchMath provides stronger gains on the higher-level benchmarks, while DASD remains stronger on some competition tasks. The additional proof evaluation broadens the evidence beyond short-answer accuracy.
  5. The paper is transparent about important limitations and appropriately cautious in its main conclusion. The authors acknowledge that the trajectories contain mathematical errors, that the human study primarily evaluates statement self-containedness, and that the decontamination procedure does not rule out semantic overlap. They also explicitly distinguish improved downstream performance from solving the underlying open problems. This transparency makes the contribution easier to assess and helps frame the work as a useful training resource and empirical investigation.
Critical weaknesses
  1. The experiments do not sufficiently separate the contribution of mathematical knowledge in the questions from that of the generated reasoning trajectories. The training objective includes the question text as well as the reasoning and final answer, so part of the improvement could come from exposure to research-level definitions, concepts, and problem formulations. Furthermore, a trajectory judged incorrect overall can still contain useful knowledge and correct intermediate steps. The comparisons in Section 7 suggest possible explanations, but do not establish whether the generated attempts provide substantial benefits beyond the mathematical content itself. This limits the interpretation of why the data is effective, although it does not negate the reported downstream gains.

  2. The human evaluation provides limited evidence about the dataset’s broader quality. The study contains 138 reviews covering 137 distinct problems, with 99 reviews contributed by a single expert. Experts also select problems they feel qualified to assess, resulting in uneven coverage and potential selection bias. More importantly, the rubric mainly evaluates self-containedness rather than fidelity to the original mathematical statement or the accuracy of the problem’s resolution status. Thus, the positive ratings support the readability and completeness of the reviewed examples, but provide less assurance about these other central aspects of dataset quality.

Questions for authors
  1. Could you provide a controlled comparison between training on the research-level questions or their mathematical background alone and training on the questions together with the generated trajectories? A complementary experiment masking question tokens from the loss could also help clarify their contribution. If feasible, separating useful correct intermediate content from incorrect reasoning steps would further strengthen the interpretation of the results.

  2. Could you supplement the human evaluation with a domain-stratified random sample and direct comparison against the source documents? In particular, it would be useful to assess whether the rewrites preserve the original assumptions, quantifiers, and objectives, and whether the assigned resolution-status labels are accurate. More balanced reviewer coverage and overlapping assessments on a subset would also strengthen confidence in the quality estimates.

Reviewer_ygbnOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper builds a dataset of 14K research-level math problems, taken from open-problem lists and papers using two LLM agents. It also generates 220K reasoning traces (only about 4% are judged correct). Fine-tuning three small models on this data gives small gains on graduate- and research-level benchmarks.

Critical strengths
  1. Useful open resource. Research-level math data for training is rare, and most existing sets are small and for evaluation only. A 14K open dataset with source links, status labels and a clear pipeline can help many groups.
Critical weaknesses

Results are still not finished - "TBD" on table 18

Questions for authors

see the weakness

Reviewer_3pHuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces a novel dataset containing 14K mathematical questions selected from 1123 academic sources, which is notably different from prior Olympiad and other mathematical datasets. The selected questions first go through GPT5.5 to provide initial questions that are shorter in length, then go through Opus 4.7 to verify open status and refine to a self-contained problem and a category label. These questions are then fed to GPT-OSS-120b and Qwen3-30-A3B to generate 220K trajectories for each question (multiple attempts per question). Out of these 220K, 150K are filtered by LLM judges to remove fabricated citations and refused attempts. Only around 5% of the generated trajectories are correct, as identified by the judge. The authors hypothesize that these incorrect reasoning traces provide reasonable supervision to help solve open problems. They train 3 models using full parameter SFT with their curated dataset and show modest improvements in held-out datasets like AIME.

Critical strengths

The dataset itself is a new collection of open research problems, which is something different that available solved olympiad questions. This pushes the frontier in terms of available research questions. This is an important gap identified in this paper.

The empirical finding is interesting, that incorrect/unverified reasoning traces seem to improve the performance of smaller models in solving other challenging math datasets.

Critical weaknesses

Considering this dataset is positioned as adding new frontier research level open questions, I was expecting to see results with some API-access limited models like Fable or 5.6 Sol. There is no performance floor reported, we have no idea what is the current performance of these gated models.

The pipeline relies entirely on LLM generated verification, trajectories, problem selection. A strong dataset should have majority human supervision than LLM supervision. The improvements due to ResearchMath on the held-out datasets can partly be attributed to knowledge transfer from the large model to the small model, which is standard distillation. I would expect to see similar improvements with any dataset, even if the supervision itself is incorrect.

I appreciate the questions curated by the dataset. However, the teacher-generated reasoning traces will be outdated very quickly as model quality improves. As smaller models get better, fixed supervision from older models will harm reasoning than help. The questions themselves are helpful, but I doubt if the reasoning traces will help push frontier math. The exercise in the paper is extremely useful in showing that incorrect reasoning helps distillation, but I don't think it increases the value of this dataset. A fewer human written solutions would help improve the value of this dataset significantly. DASD also uses LLM generated traces, but all questions have a verifiable solution, making the reasoning correct.

Finally, full-parameter SFT vs LORA SFT has not been properly compared. Full parameter SFT is known to cause catastrophic forgetting. A controlled comparison of LoRA vs Full SFT would help show what causes better improvements. The improvements over DASD are also very modest. While the gain is real, I wonder if this will show up as smaller models start to get better.

Questions for authors
  1. Do you think this dataset will help improve more recent models like qwen 3.8 27B, or larger models like 400B+?
  2. The teacher models currently rely on their internal cut-off knowledge to synthesize solutions. Do you think providing access to the internet will help provide much better reasoning traces?
  3. Can you provide a matched comparison of LORA vs Full parameter SFT?
Reviewer_zasmOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses two agents, an Extractor and a Refiner, to collect mathematical problems from academic sources such as arXiv, recover the necessary context, and produce self-contained problem statements. The resulting ResearchMath-14K dataset contains approximately 14,000 problems. The authors then generate and filter teacher reasoning trajectories for SFT, showing improvements on advanced mathematics benchmarks. Notably, the training corpus remains useful even though most sampled trajectories are judged incorrect by LLM evaluators.

Critical strengths
  • The constructed training data provides measurable improvements in mathematical performance, particularly on advanced mathematics benchmarks.
  • The experimental comparisons are rigorous, evaluating the performance of different models with approximately matched numbers of training tokens.
  • The observation that a corpus containing predominantly judged-incorrect reasoning trajectories can still benefit training is interesting and merits further investigation.
Critical weaknesses
  • The paper’s contribution lacks novelty, and the provided data does not even consist of correct solutions. Although it helps improve models’ mathematical abilities, it does not address the fundamental problem of open-source models’ limited capabilities in research-level mathematical reasoning.
  • The authors find that incorrect reasoning trajectories can also improve model capabilities, but do not sufficiently explain the reasons for these gains through systematic experiments. It remains unclear whether the improvements come from correct intermediate reasoning steps, from training the models to become domain experts, or from the usefulness of incorrect attempts themselves.
Questions for authors

None.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.