- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to generate better training data based on the student model's own reasoning mistakes. The authors point out that existing data synthesis methods usually rely on task descriptions, failed problems, or capability labels, which may not tell us exactly where the model makes mistakes. To address this, the authors propose Diagnosis-Driven Synthesis (DDS), which has two main parts. First, Trace-centric Diagnosis uses a teacher model to analyze the student's incorrect reasoning traces and identify specific errors. Then, Diagnostic Crossover randomly selects two diagnoses and combines them to generate a new problem that requires the model to address both weaknesses. The generated problems are verified and then used for SFT. This process can also be repeated as the student model gets updated. The experiments use a fixed budget of 12K training samples. DDS improves the average performance of Qwen2.5-3B-Instruct and Llama-3.2-1B-Instruct by 7.54 and 8.32 points across six math benchmarks, respectively. The authors also provide experiments on crossover ablation, iterative re-diagnosis, and multimodal tasks.
Critical strengths
- Instead of only looking at which problems the model fails on, the authors try to find out exactly where the reasoning goes wrong and use this information to guide data synthesis. I think this is a reasonable way to generate more targeted training data.
- Instead of generating problems for each weakness separately, the authors try to combine two diagnoses into one problem that requires multiple reasoning skills. The comparison between DDS-AUG and DDS-EVOLVE also shows some benefits, especially on the compositional holdout, where DDS-EVOLVE improves performance by 4.7 points.
- The experiments are comprehensive. The authors test DDS on two student models and six math benchmarks, with clear improvements. They also provide several useful analyses, including crossover ablation, iterative re-diagnosis, teacher model robustness, different patch budgets, run-to-run variance, and multimodal experiments.
Critical weaknesses
- My first concern is that the experiments mainly focus on relatively small and weak student models, such as Llama-3.2-1B-Instruct and Qwen2.5-3B-Instruct. Although the authors also test a 7B VLM, I think it would be important to see results on stronger text-only reasoning models, such as Qwen3-4B or some 7B/8B models. Since DDS relies on identifying the student's reasoning mistakes, the types of errors may become quite different as the student gets stronger. So I wonder whether DDS can still bring similar improvements to stronger models.
- I am also curious about why DDS always combines exactly two diagnoses. The authors mention in the limitations that combining more diagnoses may lead to more complex problems, but could also increase the generation cost and the risk of ill-posed problems. However, there are no experiments on different crossover depths. For example, what happens if we combine three or four diagnoses instead of two? I think it would be helpful to see how the number of diagnoses affects the final performance, data quality, and generation cost. It is still unclear whether combining two diagnoses is a good choice.
- Another concern is about the overall data synthesis cost. DDS involves several stages, including diagnosis, problem generation, verification, and solution generation, which seem quite expensive. I noticed that Appendix D.2 already reports the number of model calls and some token statistics, which is helpful. However, the paper does not directly compare DDS with other baselines under the same end-to-end compute budget. Also, diagnostic crossover requires noticeably more tokens for solution generation. So I feel that using the same number of training samples does not necessarily mean the comparison is fair in terms of compute. I would like to see more analysis of the cost per accepted training sample and whether DDS still performs better under the same overall compute budget.
Questions for authors
The authors show that re-running diagnosis and data synthesis after updating the student gives better results than simply reusing the old patches. However, this also means that each round may require many additional teacher calls, problem generation, and verification steps. Have the authors considered more efficient iterative patching strategies, such as only re-diagnosing some of the failed problems or reusing part of the previous patches? I wonder if similar improvements could be achieved with much lower synthesis costs.
3 comments
Response to Author
Thank you for your response and for running the additional experiments with Qwen3-4B. The results are helpful, but I still have concerns about the gains from DDS on a stronger student model.
Comparing the baseline scores in your table with those in the original Qwen3 paper, my guess is that Qwen3-4B was evaluated in non-thinking mode.
I’d be interested to know whether DDS still yields meaningful gains over both the base model and EvalTree when thinking mode is enabled for all three models under the same evaluation settings.
Since open-source reasoning models are typically used with thinking enabled for reasoning tasks, this comparison would help clarify how much DDS adds when the student is already using its reasoning capabilities at inference time.
Response to Reviewer BjxK
Thank you for your follow-up and for raising your score. We agree that evaluating DDS in thinking mode is an important extension. A full comparison would require regenerating training data from the student's thinking-mode failures and retraining the models under the same setting. We will consider this in future work.
Response to Reviewer BjxK
We thank the reviewer for the careful reading. We address each point below.
W1. Stronger student
DDS also improves a stronger text-only student. We ran our pipeline on Qwen3-4B, using Qwen3.6-27B for diagnosis, problem generation, and solution writing. We compare with EvalTree because, like DDS, it builds training data from the student's own failures. Random and MICOTA do not generate new problems, and TreeSynth generates problems independent of the student. Both methods use 3K training samples.
Avg. is the mean over MATH-500, OlympiadBench, and AIME25. DDS leads EvalTree by 6.75 points on the student's own failures and by 3.54 on the math average. We will add this experiment to the revision.
W2. Number of diagnoses per problem
Our results show that training on problems built from two diagnoses gives higher accuracy than training on problems built from one, while the effect of a third is inconclusive. We trained Qwen2.5-3B for each number of diagnoses k and evaluated the average accuracy on MATH-500, OlympiadBench, and AIME25.
We also observe that more diagnoses lead to more ill-posed problems, which the verifier filters out (2.4%, 6.3%, and 10.8% for k = 1, 2, and 3). This lowers generation efficiency.
In our paper, we combine two diagnoses, which improves over a single diagnosis (Figure 4). However, extending to three is not straightforward and needs further study and more computation, as noted in our Limitations section. We will add these results to the appendix.
W3. Synthesis cost under the same compute
We have computed the end-to-end costs as suggested by the reviewer, and we observe that DDS costs less than EvalTree while still performing better. We measured the cost in the Qwen3-4B pipeline from W1, where every teacher call is logged. Both methods first run the student on the seed problems and grade its answers. This part is identical, so we leave it out. We also skip the verifier in both pipelines, since Table 10 shows that DDS keeps its lead over EvalTree without it.
End-to-end compute. EvalTree uses 1.3x more teacher tokens than DDS, and DDS still outperforms it on the student's own failures and on all three math benchmarks in W1.
Cost per accepted sample. We count the cost of every generated problem, including those removed by the filters, and divide it by the 3K accepted training samples. DDS needs 50.1K tokens per accepted sample and EvalTree needs 65.1K.
Equal sample counts. Crossover solutions are indeed longer. However, in the paper's main setting (Qwen2.5-3B, 12K samples per method), EvalTree's training set still has more tokens than ours (103.4M vs. 98.9M), so using the same number of samples does not give DDS more training compute. We also trained DDS on only 5K samples, 40% of EvalTree's training tokens, and it still outperforms EvalTree with 12K samples.
We will add the 5K result to the appendix.
Q. Cheaper iterative patching
This is an interesting and important question. In our pipeline, most of the cost comes from solution generation, so generating fewer new samples per round would save the most. The reviewer's suggestion does this by reusing earlier diagnoses and patches. To make it work, one needs to decide which earlier patches to reuse, which newly failed problems to diagnose, and which new problems to write solutions for. These choices are an interesting direction for future work, with the general goal of keeping the gains while making each round cheaper. We will discuss these directions in the revision.
We hope these results address the reviewer's concerns. Should the reviewer have any further questions, we would be happy to discuss and address them.