- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes a multi-agent framework, Code2Math, used in taking math problems and making them more challenging. The framework is based on 3 separate agents, one for proposing a new problem, one for verifying that the proposed problem is logical, and one for rating the difficulty of the proposed problem. The resulting dataset of evolved problems is shown to be harder for models to solve, requiring more reasoning tokens. A human study also supports the claims about difficulty. Lastly, 2500 generated problems are used as extra training data in Qwen3-8B and Qwen3-14B to improve scores on AIME24/25/26.
Critical strengths
The paper is well written and easy to understand. The authors thoroughly analyzed their method using humans, a variety of models, analysis on models' answers, and more. The paper's premise of synthesizing useful training data automatically is of great interest to the community.
Critical weaknesses
As far as I understand, the reason for generating harder problems is to eventually used them to train/finetune models. The paper mainly focuses on the problem generation, and only dedicates one section to the finetuning part. I think the finetuning should have played a bigger role, with potentially longer training and more results. In general, the paper explores question complexity in depth, but I'm not convinced that the best way to finetune a model is to augment its data with very complex problems (wouldn't a uniform range of difficulty also make sense?).
Concretely said:
- It's hard for me to see where this work stands in relation to prior work, given that lines 125-126 mention effective prior work that also generates math problems but doesn't contrast them with this work.
- For Table 2: I think a better baseline would be to ask to create 2500 additional problems without the pipeline, then train on that. What about just having the evolution agent generate problems? How would that fare? It would also be nice to error bars if possible.
- I think the evaluation is somewhat weak: 1) no other data augmentation methods are shown, and, the evaluation set is AIME24/25/26, which somewhat overlaps with the initial seed problem set. Additionally evaluating on Omni-MATH [1] or a similar benchmark would be very helpful.
Minor: Table 1 - the table is too small, with the font being very hard to read even when zoomed in. A re-organization of the table, even at the cost of removing some information, would be helpful to legibility. In Figure 3, is the y axis (density) equal to the number of problems? Figure 4's text is somewhat small and hard to read
[1] Gao, Bofei, et al. "Omni-math: A universal olympiad level mathematic benchmark for large language models." International Conference on Learning Representations. Vol. 2025. 2025.
Questions for authors
- How does Math2Code's augmentation compare to previous methods that generate evolutions/pertubations of data? How does using other methods vs Math2Code compare when finetuning a model on the extra data?
- How would a baseline of just asking a language model to generate more data (in an agentic setting, where the model can take a few steps, use code) fare against Math2Code?
- Would it be possible to show evaluations on math datasets that are disjoint from AIME24/25/26?
9 comments
Response 2
(Continued from the previous comment)
W2/Q2: Evolution-Agent-Only Baseline and the Role of Verification
We thank the reviewer for this valuable suggestion. We agree that an Evolution-Agent-only baseline would be useful for further isolating the contribution of our verification components.
While we have not conducted this exact ablation, our existing results in Table 3 and Figure 4 already provide empirical evidence for the importance of verification.
Specifically, Figure 4 analyzes the distribution of generation failures across 100 seed problems for each evolution model. Based on these results, we summarize the first-attempt rejection rates and the frequency of different verification failures below.
Table: Generation failure statistics derived from Figure 4 (% of 100 seed problems).
| Evolution Model | First-Attempt Rejection | At Least One Solvability Failure | At Least One Difficulty Failure |
|---|---|---|---|
| DeepSeek-Chat | 67% | 58% | 41% |
| DeepSeek-Reasoner | 70% | 58% | 40% |
| Gemini-3-Pro-Thinking | 42% | 24% | 34% |
Note: The first column measures the proportion of problems whose first generation attempt fails the overall verification process. The other two columns measure whether at least one corresponding verification failure occurs during the entire evolution process; they are not first-attempt failure rates and are not mutually exclusive.
These results reveal that 42–70% of initial generations fail to satisfy our verification criteria, even when using capable evolution models. Moreover, both solvability and difficulty failures occur frequently, indicating that these two verification mechanisms address distinct challenges in mathematical problem evolution.
Table 3 further supports this observation. For example, DeepSeek-Chat encounters an average of 4.11 failed attempts before successful evolution, including 2.65 solvability failures and 1.46 difficulty failures. Other models exhibit similar patterns, although the dominant failure type varies across models.
These findings provide empirical motivation for our dual-verification design. Without the verification components, candidates that fail mathematical validity or difficulty requirements would not be systematically identified and filtered. Therefore, verification plays an important role in controlling the quality of the generated problems.
We acknowledge that these analyses do not fully replace a controlled Evolution-Agent-only downstream SFT experiment. Unfortunately, due to the limited computational resources available to us and the short rebuttal period, we are unable to complete this additional experiment. Nevertheless, we hope that the existing failure analysis, together with our newly added comparisons against Few-shot Generation and DéjàQ-inspired Symbolic Mutation, provides meaningful evidence supporting the motivation and effectiveness of the overall Code2Math framework.
W3/Q3: Evaluation Benchmarks and Data Contamination
We thank the reviewer for these concerns. We clarify the dataset distinction, provide additional contamination analysis, and explain our benchmark selection.
1. Clarification of the Two Datasets.
As described in Section 3.1, Code2Math uses two separate datasets:
- 100 seed problems are used exclusively to evaluate problem evolution quality, including solvability, difficulty improvement, and exploration efficiency.
- Approximately 2,500 training problems are independently collected to construct the Original and Original + Evolved datasets for downstream SFT.
Although some evaluation seeds originate from AIME24/25, they are not used for downstream SFT. Thus, their overlap with AIME does not imply contamination of the downstream training experiments.
2. Additional Data Contamination Audit.
During the rebuttal period, we conducted a multi-stage contamination audit comparing 2,560 original training problems against all 90 AIME24/25/26 evaluation problems (230,400 pairs).
The audit included exact matching (raw text, normalized mathematical tokens, and alphanumeric text), similarity retrieval (TF-IDF, character n-grams, and token shingles), number-masked matching, and format-independent retrieval. High-similarity candidates were further inspected for mathematical equivalence, including problem objects, constraints, and objectives.
The results are summarized below.
Table: Data contamination audit against AIME24/25/26.
| Audit Stage | Coverage | Findings |
|---|---|---|
| Exact matching (3 methods) | 230,400 problem pairs | 0 matches |
| Multi-view similarity retrieval | 1,167 candidate pairs | High-similarity candidates identified |
| Detailed candidate inspection | 94 selected high-similarity pairs | 0 confirmed direct duplicates or clear trivial variants |
| AIME24 / AIME25 / AIME26 | 38 / 33 / 23 inspected pairs | 0 confirmed duplicates in each year |
(Continued in the next comment below...)
Response 3
(Continued from the previous comment)
No direct duplicates or clear trivial variants were identified. We also preserved the audit procedures, candidate-level rationales, source-file hashes, and artifacts for reproducibility. While this provides evidence against direct problem-level contamination, it does not establish complete semantic non-overlap or cover evolved problems and model pretraining data.
3. Sufficiency of AIME Evaluation and Omni-MATH.
Our evaluation covers AIME24/25/26 with Avg@10, providing repeated measurements across three competition years. The models also exhibit non-saturated performance, making these benchmarks suitable for assessing downstream improvements.
We appreciate the suggestion to include Omni-MATH. However, its substantially larger size and difficulty would require approximately 49× the evaluation workload under our Avg@10 protocol, exceeding our available rebuttal-period computational budget.
While broader benchmark coverage would strengthen generalization claims, we believe the three-year AIME evaluation, together with the additional contamination audit, provides meaningful evidence of Code2Math's downstream training effectiveness.
W4: Problem Difficulty and Downstream Training
We thank the reviewer for this thoughtful comment. We would like to clarify that the primary objective of Code2Math is not to investigate the optimal difficulty distribution or training strategy for mathematical reasoning models. Rather, our work studies whether code agents can autonomously explore mathematical structures and construct valid, increasingly challenging problems.
We believe that the ability to discover and formulate harder mathematical problems is itself an important capability of mathematical reasoning agents, independently of their downstream training utility. This is why our paper places substantial emphasis on evaluating problem validity, difficulty improvement, and the exploration process. Downstream SFT experiments serve as additional evidence that the evolved problems can provide useful training signals.
Regarding the reviewer's suggestion of using a broader or more uniform difficulty distribution, we agree that training exclusively on highly difficult problems may not be optimal. However, our experiments use Original + Evolved rather than Evolved Only. This setting preserves the original problems while incorporating more challenging evolved versions, thereby expanding the difficulty range of the training data and providing a coarse-grained difficulty spectrum.
Importantly, we do not claim that our approach produces an optimal difficulty distribution, nor that harder problems are universally better for training. Investigating optimal data selection and curriculum learning strategies is an interesting but distinct research direction.
Additional evidence from extended training. To further investigate downstream training utility, we extended our Qwen3-14B experiments to 12 epochs. Under equal training-example exposure (Original at epoch 12 versus Original + Evolved at epoch 6), Code2Math achieves an average accuracy of 57.78%, compared with 56.00% for Original. Furthermore, the best observed average accuracy reaches 59.56% for Original + Evolved (epoch 9), compared with 56.00% for Original (epoch 12).
These additional results suggest that evolved problems provide useful training signals even when controlling for the number of training examples processed, and that their benefits are not restricted to the original six-epoch evaluation.
Overall, our central contribution is to investigate autonomous mathematical problem evolution through executable exploration, rather than to establish an optimal mathematical reasoning training strategy. The downstream training results complement this contribution by demonstrating that the generated problems are not only more challenging but also practically useful for improving mathematical reasoning performance.
[1] Röpke et al., 2026. DéjàQ: Open-Ended Evolution of Diverse, Learnable and Verifiable Problems. https://arxiv.org/abs/2601.01931
Response 1
We sincerely thank the reviewer for the positive assessment of our work and the thoughtful, constructive feedback. We have carefully considered each of the reviewer's concerns and conducted extensive additional experiments and analyses, including comparisons with existing problem generation methods, an investigation of the role of verification, extended downstream training experiments, and a systematic data contamination audit. We hope these additional results and clarifications will help address the reviewer's concerns and further strengthen the evidence supporting our contributions. We also appreciate the suggestions regarding presentation and readability, and will improve the formatting and legibility of Table 1, Figure 3, and Figure 4 in the revised version.
Note on Reproduced Experiments. Due to an unexpected issue that rendered our original experimental environment unrecoverable, we had to reconstruct the environment and rerun several experiments during the rebuttal period. Consequently, some numerical results in our responses may differ slightly from those reported in the original submission. We have carefully checked the newly obtained results and ensured that comparisons within each additional experiment use consistent experimental settings.
W1: Novelty and Comparison with Prior WorkWe thank the reviewer for raising this important concern. We would like to clarify that the key distinction between Code2Math and prior mathematical problem generation methods lies in how new mathematical problems are discovered and constructed.
Rather than relying primarily on direct LLM generation or predefined mutation strategies, Code2Math enables agents to actively explore mathematical structures through executable computation, including symbolic manipulation, combinatorial search, numerical experimentation, and counterexample discovery. The generated problems are further filtered through dedicated solvability and difficulty verification. Our central contribution is therefore to investigate code-driven mathematical exploration as a mechanism for constructing valid and challenging problems.
To further establish the effectiveness of our approach, we conducted additional experiments with two baselines:
We reran the downstream SFT experiments using Qwen3-14B-Base under identical training configurations and report results at the same checkpoint (epoch 8). Since these are new training runs, the absolute results differ slightly from those reported in the original submission.
Table: Downstream SFT performance (Avg@10, %) on AIME benchmarks.
Code2Math consistently outperforms both Few-shot Generation and DéjàQ-inspired Symbolic Mutation across all three AIME benchmarks. This advantage is maintained throughout all eight training epochs, including when comparing the best observed checkpoint of each method. Interestingly, while the two baselines occasionally outperform Original at intermediate epochs, their best observed results offer little or no improvement over Original. In contrast, Code2Math achieves a clear and consistent improvement, suggesting that simply generating more problems, even through structural mutation, does not necessarily provide substantial training benefits. These findings support the effectiveness of Code2Math in generating more valuable mathematical reasoning data through executable exploration and dual verification.
(Continued in the next comment below...)