acceptodds
Under review as a conference paper at ICLR 2027

Refining Problem Specifications via Trace-Level Diagnosis for Reasoning Data Synthesis

Abstract

Synthetic data has become central to LLM post-training, but existing synthesis methods differ in what they tell the generator to synthesize. We call this conditioning signal the specification. Prior methods often use task descriptions or capability labels derived from problems the student answered incorrectly. These signals identify broad weak regions, but they do not explain where the student's reasoning failed. We propose Diagnosis-Driven Synthesis (DDS), a framework that uses the student's incorrect solution trace as a more specific synthesis signal. DDS first performs trace-centric diagnosis: a teacher inspects each failed solution and produces an actionable directive describing where the reasoning broke and what a new problem should require. DDS then introduces diagnostic crossover, which pairs two such diagnoses and asks a generator to create a single coherent problem that requires both weaknesses to be repaired jointly. A verifier filters generated candidates for well-posedness and solution correctness before fine-tuning. On math reasoning, DDS recovers and Pass@1 points on the student's failed problems over the base student on Qwen2.5-3B-Instruct and Llama-3.2-1B-Instruct, and improves the held-out math average by and points. At the same patch budget, DDS exceeds the strongest baseline by and points on the held-out math average. Iterative re-diagnosis further lifts the math average to . Beyond math, DDS reaches on a seven-benchmark multimodal average, over the strongest baseline.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic data generation, reasoning data synthesis, LLM post-training, trace-level diagnosis, failure-driven synthesis

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_BjxKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to generate better training data based on the student model's own reasoning mistakes. The authors point out that existing data synthesis methods usually rely on task descriptions, failed problems, or capability labels, which may not tell us exactly where the model makes mistakes. To address this, the authors propose Diagnosis-Driven Synthesis (DDS), which has two main parts. First, Trace-centric Diagnosis uses a teacher model to analyze the student's incorrect reasoning traces and identify specific errors. Then, Diagnostic Crossover randomly selects two diagnoses and combines them to generate a new problem that requires the model to address both weaknesses. The generated problems are verified and then used for SFT. This process can also be repeated as the student model gets updated. The experiments use a fixed budget of 12K training samples. DDS improves the average performance of Qwen2.5-3B-Instruct and Llama-3.2-1B-Instruct by 7.54 and 8.32 points across six math benchmarks, respectively. The authors also provide experiments on crossover ablation, iterative re-diagnosis, and multimodal tasks.

Critical strengths
  • Instead of only looking at which problems the model fails on, the authors try to find out exactly where the reasoning goes wrong and use this information to guide data synthesis. I think this is a reasonable way to generate more targeted training data.
  • Instead of generating problems for each weakness separately, the authors try to combine two diagnoses into one problem that requires multiple reasoning skills. The comparison between DDS-AUG and DDS-EVOLVE also shows some benefits, especially on the compositional holdout, where DDS-EVOLVE improves performance by 4.7 points.
  • The experiments are comprehensive. The authors test DDS on two student models and six math benchmarks, with clear improvements. They also provide several useful analyses, including crossover ablation, iterative re-diagnosis, teacher model robustness, different patch budgets, run-to-run variance, and multimodal experiments.
Critical weaknesses
  • My first concern is that the experiments mainly focus on relatively small and weak student models, such as Llama-3.2-1B-Instruct and Qwen2.5-3B-Instruct. Although the authors also test a 7B VLM, I think it would be important to see results on stronger text-only reasoning models, such as Qwen3-4B or some 7B/8B models. Since DDS relies on identifying the student's reasoning mistakes, the types of errors may become quite different as the student gets stronger. So I wonder whether DDS can still bring similar improvements to stronger models.
  • I am also curious about why DDS always combines exactly two diagnoses. The authors mention in the limitations that combining more diagnoses may lead to more complex problems, but could also increase the generation cost and the risk of ill-posed problems. However, there are no experiments on different crossover depths. For example, what happens if we combine three or four diagnoses instead of two? I think it would be helpful to see how the number of diagnoses affects the final performance, data quality, and generation cost. It is still unclear whether combining two diagnoses is a good choice.
  • Another concern is about the overall data synthesis cost. DDS involves several stages, including diagnosis, problem generation, verification, and solution generation, which seem quite expensive. I noticed that Appendix D.2 already reports the number of model calls and some token statistics, which is helpful. However, the paper does not directly compare DDS with other baselines under the same end-to-end compute budget. Also, diagnostic crossover requires noticeably more tokens for solution generation. So I feel that using the same number of training samples does not necessarily mean the comparison is fair in terms of compute. I would like to see more analysis of the cost per accepted training sample and whether DDS still performs better under the same overall compute budget.
Questions for authors

The authors show that re-running diagnosis and data synthesis after updating the student gives better results than simply reusing the old patches. However, this also means that each round may require many additional teacher calls, problem generation, and verification steps. Have the authors considered more efficient iterative patching strategies, such as only re-diagnosing some of the failed problems or reusing part of the previous patches? I wonder if similar improvements could be achieved with much lower synthesis costs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer BjxK

We thank the reviewer for the careful reading. We address each point below.

W1. Stronger student

DDS also improves a stronger text-only student. We ran our pipeline on Qwen3-4B, using Qwen3.6-27B for diagnosis, problem generation, and solution writing. We compare with EvalTree because, like DDS, it builds training data from the student's own failures. Random and MICOTA do not generate new problems, and TreeSynth generates problems independent of the student. Both methods use 3K training samples.

Qwen3-4BOwn seed failuresMATH-500OlympiadBenchAIME25Avg.
Base22.7583.2549.2017.5049.98
EvalTree25.2080.0552.2528.3353.54
DDS31.9585.2553.5032.5057.08

Avg. is the mean over MATH-500, OlympiadBench, and AIME25. DDS leads EvalTree by 6.75 points on the student's own failures and by 3.54 on the math average. We will add this experiment to the revision.

W2. Number of diagnoses per problem

Our results show that training on problems built from two diagnoses gives higher accuracy than training on problems built from one, while the effect of a third is inconclusive. We trained Qwen2.5-3B for each number of diagnoses k and evaluated the average accuracy on MATH-500, OlympiadBench, and AIME25.

# of diagnosesk = 1k = 2k = 3
Avg33.7934.7134.45

We also observe that more diagnoses lead to more ill-posed problems, which the verifier filters out (2.4%, 6.3%, and 10.8% for k = 1, 2, and 3). This lowers generation efficiency.

In our paper, we combine two diagnoses, which improves over a single diagnosis (Figure 4). However, extending to three is not straightforward and needs further study and more computation, as noted in our Limitations section. We will add these results to the appendix.

W3. Synthesis cost under the same compute

We have computed the end-to-end costs as suggested by the reviewer, and we observe that DDS costs less than EvalTree while still performing better. We measured the cost in the Qwen3-4B pipeline from W1, where every teacher call is logged. Both methods first run the student on the seed problems and grade its answers. This part is identical, so we leave it out. We also skip the verifier in both pipelines, since Table 10 shows that DDS keeps its lead over EvalTree without it.

Teacher tokensDDSEvalTree
Diagnosis / capability annotation11.6M46.3M
Capability tree descriptionsn/a23.2M
Problem generation51.1M37.8M
Solution generation87.4M88.0M
End-to-end total150.2M195.3M
Per accepted training sample50.1K65.1K

End-to-end compute. EvalTree uses 1.3x more teacher tokens than DDS, and DDS still outperforms it on the student's own failures and on all three math benchmarks in W1.

Cost per accepted sample. We count the cost of every generated problem, including those removed by the filters, and divide it by the 3K accepted training samples. DDS needs 50.1K tokens per accepted sample and EvalTree needs 65.1K.

Equal sample counts. Crossover solutions are indeed longer. However, in the paper's main setting (Qwen2.5-3B, 12K samples per method), EvalTree's training set still has more tokens than ours (103.4M vs. 98.9M), so using the same number of samples does not give DDS more training compute. We also trained DDS on only 5K samples, 40% of EvalTree's training tokens, and it still outperforms EvalTree with 12K samples.

Qwen2.5-3BSeed failuresHeld-out math avg.
EvalTree, 12K (paper)20.542.20
DDS, 5K24.243.62

We will add the 5K result to the appendix.

Q. Cheaper iterative patching

This is an interesting and important question. In our pipeline, most of the cost comes from solution generation, so generating fewer new samples per round would save the most. The reviewer's suggestion does this by reusing earlier diagnoses and patches. To make it work, one needs to decide which earlier patches to reuse, which newly failed problems to diagnose, and which new problems to write solutions for. These choices are an interesting direction for future work, with the general goal of keeping the gains while making each round cheaper. We will discuss these directions in the revision.

We hope these results address the reviewer's concerns. Should the reviewer have any further questions, we would be happy to discuss and address them.

Reviewer_BjxKOpenReview ↗

Response to Author

Thank you for your response and for running the additional experiments with Qwen3-4B. The results are helpful, but I still have concerns about the gains from DDS on a stronger student model.

Comparing the baseline scores in your table with those in the original Qwen3 paper, my guess is that Qwen3-4B was evaluated in non-thinking mode.

I’d be interested to know whether DDS still yields meaningful gains over both the base model and EvalTree when thinking mode is enabled for all three models under the same evaluation settings.

Since open-source reasoning models are typically used with thinking enabled for reasoning tasks, this comparison would help clarify how much DDS adds when the student is already using its reasoning capabilities at inference time.

AuthorsOpenReview ↗

Response to Reviewer BjxK

Thank you for your follow-up and for raising your score. We agree that evaluating DDS in thinking mode is an important extension. A full comparison would require regenerating training data from the student's thinking-mode failures and retraining the models under the same setting. We will consider this in future work.

Reviewer_FpnBOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Diagnosis-Driven Synthesis (DDS) for generating training data from a student model’s errors. A teacher analyzes each failed solution trace and writes a specification describing where the reasoning failed and what a new problem should require. DDS can also combine two diagnoses to generate one problem that requires both weaknesses to be addressed. A verifier filters the generated problems before they are used for fine-tuning. Experiments on math reasoning and multimodal benchmarks show improvements over the base models and existing baselines.

Critical strengths
  1. The motivation is well developed. Synthesizing high-quality data is an important topic, and the paper identifies limitations of existing synthetic data methods.

  2. The paper is clearly written and includes extensive experiments. The reported improvements are substantial, and the case studies in the appendix provide concrete examples.

Critical weaknesses
  1. The paper does not formalize the proposed concept of a specification or explain it in sufficient detail. Software engineering uses a related concept of specification. Could the authors draw on principles from software engineering to define specification more clearly?

  2. I would like to know the time or token cost of the proposed method. Consider Figure 2. If generating two problems, A' and B', costs the same as generating one problem, A+B, does DDS still have an advantage when the token budget is matched?

  3. In line 63, the paper defines F=(q,rs,r∗)F = {(q, r_s, r*)}, while the text also uses (q,r∗,a∗)(q, r*, a*). Is (q,rs,r∗)(q, r_s, r*) a typo? Should the last element of the tuple be (q,r,a(answer))(q, r, a (answer))? Please check whether this notation is consistent throughout the paper.

Questions for authors

See Weaknesses.

2 comments
AuthorsOpenReview ↗

Response to Reviewer FpnB

We thank the reviewer for the careful reading. We address each point below.

W1. Definition of a specification

In Section 3.1, we define the specification as the information given to the generator before it writes a new problem. Formally, σ=((qi,rs,i,di),(qj,rs,j,dj))\sigma = ((q_i, r_{s,i}, d_i), (q_j, r_{s,j}, d_j)). It contains two problems the student failed (qi,qjq_i, q_j), the student's incorrect solutions (rs,i,rs,jr_{s,i}, r_{s,j}), and the teacher's diagnoses of these solutions (di,djd_i, d_j).

From a software engineering perspective, σ\sigma is like a requirements document for the new problem, and the generator is like the developer who writes the problem to meet it. In our view, the requirements are the reasoning steps that the new problem should make the student practice. These steps come from the two diagnoses. Each diagnosis points to where the student's reasoning went wrong, and the new problem should require the student to get both steps right. The failed problems and incorrect solutions give the context for these requirements. We will revise Section 3.1 to describe the specification more clearly.

W2. Token cost, and A+B versus A′ and B′

We would like to clarify that writing a solution for a crossover problem takes about 1.3 times as many tokens as for a single-failure problem, not twice as many (Appendix D.2). To compare performance at a matched token budget, we trained DDS on a 9.7K subset of its 12K samples, which has the same number of training tokens as the 12K DDS-AUG set.

Qwen2.5-3B, same training tokensSeed failuresHeld-out math avg.
DDS-AUG, 12K26.244.57
DDS, 9.7K subset27.445.10

The subset still recovers more of the student's failures and scores higher on the held-out math average than DDS-AUG trained on the full 12K set. We will add this comparison to Appendix D.2.

W3. Notation

Thank you for checking the notation. Both tuples are correct, as they describe two different sets. A seed example (q,r⋆,a⋆)(q, r^\star, a^\star) contains a problem, its reference solution, and its reference answer, which matches the (q,r,a)(q, r, a) form the reviewer suggests. A failure in F\mathcal{F}, (q,rs,r⋆)(q, r_s, r^\star), contains the problem, the student's incorrect solution, and the reference solution, which the teacher compares with the student's solution when writing the diagnosis. We checked the paper, and the notation is consistent throughout. We will add a short notation table to the appendix and refer to it in Section 3.1.

We hope these responses address the reviewer's concerns. Should the reviewer have any further questions, we would be happy to discuss and address them.

Thank you. I maintain my positive score.

Reviewer_8i7POpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Diagnosis-Driven Synthesis (DDS), a framework for generating targeted synthetic training data from a student model’s own failures. A teacher model analyzes the student’s incorrect solution traces and isolates the reasoning steps where the student fails. DDS then uses diagnostic crossover, where two diagnosed failures are combined into a single new task, so the model learns to handle these weaknesses together rather than in isolation. Generated problems are verified for correctness before fine-tuning. The method is evaluated on math tasks with Llama and Qwen students and extended to vision-language post-training.

Critical strengths

DDS substantially and consistently improves performance across diverse math and multimodal benchmarks. It outperforms all baselines at an equal data budget, and the gains are robust across seeds and evaluation domains.

Critical weaknesses

The authors do motivate the use of two failures in one task, but I’m not fully convinced by the motivation yet. Intuitively, I would have expected a concept to be easier to learn when the other concepts in the task are already clear. Maybe the authors could elaborate on this or show concrete examples demonstrating the benefit, if possible.

Questions for authors
  • Q1: The Random baseline performs surprisingly well. If I understand correctly, the seed set includes 5,000 problems from OpenMathReasoning, and Random samples its 12K training problems from the same dataset. Did you exclude seed problems, in particular the student’s seed failures, from Random’s training sample? If not, how many of the seed failures evaluated in Fig. 3 also appear in Random’s training data, and how does Random’s seed-failure recovery change when those overlapping problems are removed?
  • Q2: Do you expect that increasing the number of failed concepts per task could boost performance further, or do you expect a ceiling or sweet spot?

I apologize for the brevity, and in case I missed something, this is an emergency review.

2 comments
AuthorsOpenReview ↗

Response to Reviewer 8i7P

We thank the reviewer for the careful reading. We address each point below.

W. One failure per problem vs. two

We would like to clarify that our paper includes comparisons that motivate this choice (Figure 4). With the same 12K samples, we compare training on problems built from one failure (DDS-AUG) with training on problems built from two failures (DDS). On held-out problems that each test one diagnosed skill, the two are nearly tied. However, on problems that need two skills together, DDS leads by 4.7 points. Many math problems need several skills at once, which is consistent with DDS also leading on the student's seed failures and on the held-out math average.

For example, Case 31 in Appendix H (Tables 18 and 19) combines two failures. The student simplified a dice-probability fraction incorrectly and took four quarter-circles of radius 1/2 to have area π instead of π/4. In the crossover problem, the player wins only if six dice show distinct numbers and six beads dropped on a unit board all land in its gray region, outside four white quarter-circles of radius 1/3. The answer, (5/324)(1 − π/9)^6, needs both the exact dice probability and the gray-area fraction. Thus, a student who has fixed only one of the two skills cannot reach it. This is the worked example summarized in Figure 2, and we will refer to it in Section 4.4.

Q1. Overlap between Random and the seed failures

Thank you for the careful reading and for catching this. Indeed, there is overlap. Random samples 12K problems from OpenMathReasoning as a standard fine-tuning baseline, and OpenMathReasoning is also one of our four seed sources, so 328 of the 500 Qwen2.5-3B seed failures in Figure 3 also appear in Random's data.

We removed the overlapping problems and retrained Random on the deduplicated 12K data.

Qwen2.5-3BSeed failuresHeld-out math avg.
Random22.7539.03
Random, deduplicated20.8239.42
DDS28.4845.02

Without the overlap, Random's seed-failure recovery drops from 22.75 to 20.82, while its held-out math average stays about the same (39.42 vs. 39.03). DDS still leads on both. We will update Random with the deduplicated version.

Q2. More failed concepts per task

This is an important question. Our results show that training on problems built from two diagnoses gives higher accuracy than training on problems built from one, while the effect of a third is inconclusive. We trained Qwen2.5-3B on 2K samples for each number of diagnoses and evaluated the average accuracy on MATH-500, OlympiadBench, and AIME25.

# of diagnoses123
Avg.33.7934.7134.45

We also observe that more diagnoses lead to more ill-posed problems, which the verifier filters out (2.4%, 6.3%, and 10.8% for one, two, and three diagnoses). This lowers generation efficiency.

In our paper, we combine two diagnoses, which improves over a single diagnosis (Figure 4). Extending to three or more is not straightforward and needs further study and more computation, so we leave the question of a ceiling or sweet spot to future work.

We hope these responses address the reviewer's concerns. Should the reviewer have any further questions, we would be happy to discuss and address them.

I thank the authors for the detailed response and the additional experiments. The deduplicated Random baseline resolves my concern about overlap, and I appreciate the transparent answer. Overall, I maintain my positive score.

Reviewer_zaFAOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Diagnosis-Driven Synthesis (DDS), which extracts repair requirements from a student's incorrect solution traces and combines two diagnoses into new problems for supervised fine-tuning after verification. Under a fixed problem budget, DDS improves recovery on failed seed problems and the held-out math average for two student models. A single-diagnosis baseline, iterative training, and vision-language experiments further examine its components and applicability.

Critical strengths

● The paper places the question of which new examples would help the current student at the center of data construction and uses specific reasoning failures to constrain generation targets. This connects the value of synthetic data to observable model improvement and addresses the difficulty of distinguishing failure causes through broad problem coverage or capability labels. The motivation has clear practical relevance, and its objectives can be examined through changes in the trained model's behavior. ● DDS turns error analysis into requirements for new training problems, linking observed failures and repair targets to data generation and learning while retaining standard supervised fine-tuning. Its value lies in making diagnosis directly inform training-data decisions and providing relatively clear units of intervention, beyond producing a report of model weaknesses. This organization supports reuse and subsequent controlled research. ● Beyond reporting the main performance gains, the paper examines whether they depend on a particular configuration through shared teacher settings, fewer added problems, removal of verification, and replacement of the reference-solution model. The retained advantages across these checks strengthen the case that the data construction approach has training value and support efficiency measured by the number of added examples. These findings matter for evaluating the overall recipe.

Critical weaknesses

● The central account is that constructing data from the current student's specific errors enables targeted capability repair. However, the experiments do not adequately separate the roles of the original error context, the semantic content of the teacher's diagnosis, and generally useful mathematical supervision. A plausible explanation of one error also does not automatically identify a stable, student-specific knowledge deficit. Overall score improvements therefore do not establish that matching the diagnosis to the student is essential. ● The main interpretation of compositional capability repair lacks evidence that distinguishes alternative causes. Crossover training changes supervision density, reasoning complexity, and alignment with the test distribution, while the central compositional evaluation shares the training generation framework. Greater familiarity with joint problem types, stronger constituent skills, and improved coordination could all produce the observed gains. Without testing unseen combination relationships at comparable levels of constituent competence, the paper cannot readily interpret the practical benefits of joint exercises as transferable compositional reasoning ability. ● The paper interprets renewed diagnosis as a response to changes in the student's failure distribution, but the experiment also replaces the training examples and may change their difficulty and distribution. The results do not determine whether the useful factor is new diagnostic information or a fresh set of relevant exercises. Because continued adaptation is important to the paper's account of long-term value, showing that the complete process can produce another gain does not establish feedback diagnosis as an indispensable mechanism.

Questions for authors

● The specification in Section 3.1 includes the problem, incorrect trace, and diagnosis. Have the authors fixed the problem and trace while removing the diagnosis or replacing it with a coarsened or shuffled version? If the strong generator can analyze the trace directly, a separate teacher diagnosis may not be necessary. ● If repeated samples on the same seed problem contain different errors, are the teacher's diagnoses stable, or do they predict errors on later related problems? Can diagnoses from the current student be compared with those from another student, especially when they make different errors on the same problem? ● Some DDS problems in Appendix I express the geometric constraints in text or coordinates. If the vision-language model is trained on the same textual problems without the corresponding images, are similar gains obtained? How does the improved model behave when the images are removed or replaced? ● When two diagnoses come from different images, how does the generator create a visual input that supports both repair requirements? Can a complete record be shown, including the two parent images and incorrect responses, the diagnoses, the new image and problem, the reference solution, and the verification result?

4 comments
AuthorsOpenReview ↗

Response to Reviewer zaFA (1/4)

We thank the reviewer for the detailed reading. We address each point below.

W1. Role of the diagnosis and failure context

We would like to clarify that the paper already separates the student's errors from generally useful mathematical supervision, and we added a new control in this rebuttal that separates the diagnosis from the original error context. Below, we go through each factor.

Generally useful mathematical supervision. TreeSynth and Random provide general math supervision without any student errors or diagnoses. DDS outperforms both on seed-failure recovery and on the held-out math average.

Semantic content of the diagnosis. In Q1, we keep the original error context and remove only the diagnosis. Adding the diagnosis back improves recovery of the student's own failures.

Stable, student-specific deficit. DDS does not assume that each error reflects a stable weakness. A single error still shows a reasoning step the student got wrong, and the same step may cause errors on other problems. Therefore, DDS targets each observed error without first checking whether it is stable.

Q1. Removing the diagnosis

We ran the control the reviewer suggested, which removes the diagnosis and keeps the problem and the incorrect trace. Adding the diagnosis improves recovery of the student's own failures, and the held-out math average also rises slightly. Starting from the same failed problems and incorrect student solutions, we generated 2K training problems with and without the diagnosis and fine-tuned Qwen2.5-3B on each set.

Qwen2.5-3BSeed failuresAvg. (MATH-500, OlympiadBench, AIME25)
Without diagnosis9.9526.47
With diagnosis (DDS)11.6026.50

Thus, although the generator can analyze the trace directly, the diagnosis still helps the new problems target the student's errors.

W2. What drives the compositional gain

Because the compositional holdout comes from the same generators as our training data, we also evaluate DDS on six held-out math benchmarks that our generators did not produce, and it still leads there. DDS scores an average of 45.02, compared with 44.57 for DDS-AUG (Figure 4a).

We would like to clarify that we do not aim for transfer to entirely unseen skill combinations. Instead, our goal is to patch the mistakes a student actually makes and to check that these patches also help on general math exams.

To patch two mistakes at once, we put two diagnoses into one training problem (supervision density), so the student has to use both skills in the same solution (improved coordination). This matches Figure 4b, where DDS leads DDS-AUG by 4.7 points on problems that need both skills but stays close to it on problems that need only one.

Simply making problems harder does not produce this gain (reasoning complexity). For this rebuttal, we made the single-failure problems harder, with 58% longer questions and 38% longer teacher solutions, and trained on 2K problems from each version. The harder version scored 19.45 on seed failures and 33.33 on the average over MATH-500, OlympiadBench, and AIME25, compared with 19.85 and 34.95 for the original version.

We will revise Section 4.4 to describe the gain as better performance on the tested joint-skill problems, rather than as transferable compositional reasoning.

W3. New diagnoses or fresh exercises

We tested this directly, and a fresh set of relevant exercises alone does not reproduce the second-round gain. We started from the same iteration-1 student and trained it on 12K fresh problems from OpenMathReasoning with the same training recipe as DDS2. DDS2 instead re-diagnoses the updated student (feedback diagnosis) and trains on new problems built from these diagnoses.

Qwen2.5-3BSeed failuresHeld-out math avg.
Iteration-1 student (Figure 4a)28.545.02
+ 12K fresh problems, no re-diagnosis28.6844.52
+ re-diagnosed patches, DDS2 (Table 2)33.148.1

With fresh problems, the student barely changed. With re-diagnosis, DDS2 improved seed-failure recovery by 4.6 points and the held-out math average by 3.1 points. We will add this control to Section 4.5.

AuthorsOpenReview ↗

Response to Reviewer zaFA (2/4)

Q2. Stability of diagnoses

As noted in W1, DDS does not need each error to reflect a stable weakness, but we ran this test at the reviewer's request. The teacher's diagnoses stay the same for the same error and change when the error changes, which is exactly how we designed DDS to work. This holds both within one student and across students. We sampled four solutions from each student on 200 seed problems that both Qwen2.5-3B and Llama-3.2-1B failed, diagnosed each incorrect solution with the GPT-5 teacher, and asked a GPT-5 judge to compare them.

GPT-5 judge checkSame studentAcross two students
Do two incorrect solutions to the same problem have different errors? (no diagnoses shown)60%84%
Given one solution and both diagnoses, does the judge pick the one written for it?98.7%99.5%
Does the same solution, diagnosed twice, get the same error?91.7% (both students)
Is the diagnosed error actually in the solution?99.3% (both students)

In other words, two incorrect solutions to the same problem often contain different mistakes, whether they come from the same student (60%) or from two students (84%). When that happens, each diagnosis fits its own solution and not the other one, so it points to that specific mistake rather than making a general comment about the problem. When the same solution is diagnosed twice, the teacher points to the same mistake, so its diagnoses are not random. Finally, the judge found that the diagnosed mistakes are actually in the solutions.

We have not tested whether diagnoses predict later errors, because DDS uses each observed error as a target for new problems rather than predicting future failures.

Q3.

That is an interesting ablation. Prior work [1,2] has shown that training vision-language models exclusively on textual reasoning problems can improve performance on multimodal benchmarks. However, these gains do not necessarily imply improved visual perception as stronger reasoning can help a model make better use of its existing visual capabilities. [3,4] highlight limitations of this transfer, showing that text-only training can leave visual perception and image–text alignment bottlenecks unresolved, while additional training through the visual modality can yield further improvements. Thus, text-only training can provide substantial benefits, but comparable gains to image–text training are not guaranteed.

[1] Liu, Qianchu, et al. "X-reasoner: Towards generalizable reasoning across modalities and domains." arXiv preprint arXiv:2505.03981 (2025).

[2] Bousselham, Walid, Hilde Kuehne, and Cordelia Schmid. "Vold: Reasoning transfer from llms to vision-language models via on-policy distillation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

[3] Peng, Yingzhe, et al. "Lmm-r1: Empowering 3b lmms with strong reasoning abilities through two-stage rule-based rl." arXiv preprint arXiv:2503.07536 (2025).

[4] Yamabe, Shojiro, et al. "Text-printed image: bridging the image-text modality gap for text-centric training of large vision-language models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2026.

AuthorsOpenReview ↗

Response to Reviewer zaFA (3/4)

Q4.

We would like to clarify that our VLM implementation performs crossover between visual-extraction and logical-error diagnoses associated with the same image. It does not perform cross-image diagnosis crossover. This is an adaptation of the language-only setting, where diagnosis crossover combines feedback from two problems. When diagnoses concern different images, constructing a new visual input that faithfully supports both repair requirements is nontrivial. Our VLM implementation retains the original image and uses the two complementary diagnoses to generate a new question grounded in its contents. Motivated by the distinction between visual grounding and reasoning discussed in [3,4], the generated question targets both accurate extraction of visual information and correct reasoning over that information.

Consequently, the relevant record contains one retained image, the original incorrect response, the visual and logical diagnoses, the new question, its reference solution, and its verification result. There are no separate parent images or newly synthesized image. The revised example below illustrates this structure:

Retained image https://huggingface.co/datasets/MMEVAL/mmevalpro/viewer/default/test?row=15

Job-cost table used in the original problem

Original problem and incorrect response

The original question asks for January’s work in process inventory, finished goods inventory, and cost of goods sold. According to the supplied diagnosis, the incorrect response reports work in process of $13,997 and cost of goods sold of $8,088, then selects option D despite the mismatch.

Diagnoses used for crossover

{
  "diagnosis": {
    "visual_extraction_errors": [
      {
        "mistake": "The response treats $3,250 for AA5 and $3,409 for AA3 as January costs.",
        "correction": "Both amounts appear in the February column. The January column contains $2,500 for AA2 and $4,838 for AA4."
      },
      {
        "mistake": "The response identifies February as AA4's completion month.",
        "correction": "AA4 is completed in January and sold in February. The completion and sale columns must be read separately."
      }
    ],
    "logical_errors": [
      {
        "mistake": "The response incorrectly classifies completed-but-unsold jobs as work in process.",
        "correction": "At the reporting date, unfinished jobs belong in work in process, while completed-but-unsold jobs belong in finished goods."
      },
      {
        "mistake": "The response includes AA4 and AA5 in January cost of goods sold.",
        "correction": "Job costs enter cost of goods sold when the corresponding jobs are sold. AA4 is sold in February and AA5 in March, so neither contributes to January cost of goods sold."
      }
    ]
  }
}

Both diagnoses concern the same job-cost image. The follow-up question combines their repair requirements by requiring correct extraction of monthly costs and dates, followed by correct inventory classification.

AuthorsOpenReview ↗

Response to Reviewer zaFA (4/4)

Revised follow-up problem and reference solution

{
  "rationale": "Targets both visual-extraction and logical errors while retaining the original image. Changing the reporting period to February requires the model to distinguish monthly cost columns, accumulate costs incurred through the reporting date, and classify jobs using their completion and sale dates.",
  "new_question": "Using the same job-cost table, determine work in process inventory and finished goods inventory at the end of February, and cost of goods sold during February. Identify the jobs contributing to each amount. Explain how costs incurred in different months are accumulated and why March costs are excluded.",
  "answer": "Work in Process = $3,409: AA3 has incurred $3,409 by February but is not completed until April; its March cost of $2,319 is excluded. Finished Goods = $6,950: AA2 has accumulated $2,500 + $1,200 = $3,700 and is completed in February but remains unsold; AA5 costs $3,250, is completed in February, and is sold in March. Cost of Goods Sold = $4,838: AA4 is completed in January and sold in February, so its $4,838 cost is recognized in February."
}

Manual verification result

{
  "verification": {
    "type": "Manual consistency check",
    "result": "Passed",
    "classification_check": "Each job is classified according to its completion and sale dates, using only costs incurred through February.",
    "allocation_check": "$3,409 + $6,950 + $4,838 = $15,197.",
    "incurred_cost_check": "$7,338 in January + $7,859 in February = $15,197.",
    "conclusion": "The allocated amounts reconcile with cumulative costs incurred through February, and each cost is counted once."
  }
}

The revised example illustrates how visual and logical repair requirements can be combined in a new question while preserving the original image.

We hope these responses address the reviewer's concerns. Should the reviewer have any further questions, we would be happy to discuss and address them.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Final Remarks by Authors

We thank the reviewers for their constructive feedback, which helped us strengthen the paper. The main updates are as follows:

To address the reviewer zaFA's question of whether the diagnosis itself helps, our paper compares DDS-AUG, which uses one trace diagnosis per problem, with EvalTree, which uses capability labels instead of trace-level diagnoses, at the same 12K budget. Even without crossover, DDS-AUG does better on the student's own failures (26.2 vs. 20.5) and on held-out math (44.57 vs. 42.20; Figure 4a), which supports the value of trace-level diagnosis. In the rebuttal, we also ran the control the reviewer zaFA suggested, using the same failed problems and incorrect solutions with and without the diagnosis. We observed that the diagnosis still improves the performance.

On whether crossover helps (8i7P), our paper shows that DDS outperforms DDS-AUG at the same 12K training sample budget, and the gap is largest (4.7 points) on problems that need two diagnosed skills together (Figure 4b), along with a gain on held-out math (45.02 vs. 44.57), which substantiates the benefits of crossover. Moreover, to address a concern from the reviewer FpnB, we also matched training tokens: DDS still does better than DDS-AUG (27.4 vs. 26.2 on seed failures, 45.10 vs. 44.57 on held-out math). To address zaFA's concern about reasoning complexity, we also made the DDS-AUG problems harder, with longer questions and solutions, and found that this does not give the same gain.

We also checked whether updating the diagnoses and their crossover helps in iterative training (zaFA). In the second round, training with re-diagnosed patches reaches 33.1 on seed failures and 48.1 on held-out math, while training with 12K fresh problems reaches 28.68 and 44.52. This shows the benefit of generating new patches based on the updated student's failures, rather than simply adding more training problems.

For evaluation on a stronger student and analysis on the cost of data synthesis (BjxK), we ran DDS on Qwen3-4B. DDS still does better than EvalTree on the student's own failures (31.95 vs. 25.20) and on held-out math (57.08 vs. 53.54), while also using fewer teacher tokens to build the data.

We also addressed the remaining points on crossover depth, diagnosis consistency, the Random baseline with deduplication, and the specification, notation, and VLM details. We have updated the revised version, with changes marked in blue.

In summary, existing data synthesis methods mostly look at which problems a student fails, but they do not examine where the student's solution goes wrong. Therefore, we propose to leverage the diagnosed reasoning trace to refine the specification for generating better training data. Moreover, when we combine two diagnosed errors into each new problem, the improvement grows further. Lastly, we also see these gains across model families and domains.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.