- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how to generate useful training data for RLVR. The authors point out that whether a problem can provide useful learning signals depends not only on the problem itself, but also on the target policy and the rollout budget. They find that around 88% of the problems in a 78K math dataset fall into what they call the "zero-advantage trap", where the model either solves all rollouts or fails all of them, and thus receives no group-relative advantage signal. To address this issue, the authors propose MagicBeanStalk (MBS), a recursive data synthesis framework that generates new training problems from hard seed problems. The main idea is to first decompose a hard problem into simpler subproblems, and then generate harder variants of these subproblems and recursively combine them into new parent problems. In this way, MBS tries to generate new problems that preserve the original reasoning blueprint while having an appropriate difficulty for the target policy. Experiments on math benchmarks show that MBS improves gpt-oss-120b and DeepSeek v4 Flash by 8.6 and 5.4 points, respectively. The authors also show some improvements in the finance domain.
Critical strengths
- The authors show that simply having more training data does not necessarily help RLVR, since many problems may not provide useful learning signals for the current policy. They also show how the difficulty of a problem changes with different target policies and rollout budgets, which gives a good motivation for their proposed method.
- Instead of directly generating variations of the original problem, MBS first decomposes it into simpler subproblems and then recursively combines newly generated subproblems into harder problems. I think this is an interesting way to preserve the reasoning skills required by the original problem while generating new training data. The effectivity test is also a nice design, since it checks whether the decomposed subproblems can actually help the student solve the original problem.
- The experimental results are promising. MBS achieves 69.8% on gpt-oss-120b, which is 2.8 points higher than the strongest filtered-source baseline. The same synthesized data also improves DeepSeek v4 Flash by 5.4 points. I also find the results on MathArena Apex interesting, as the improvements are still quite clear with longer inference budgets (24K and 32K). The finance experiments also show that the method has some potential to work beyond math reasoning tasks.
Critical weaknesses
- My main concern is about the baselines. Although the authors compare MBS with Direct Seed Synthesis, Goldilocks Filtering, and Golden Goose, I think it would be important to compare with other existing RLVR data synthesis frameworks that have similar motivations. For example, SwS (NeurIPS 2025) also starts from problems that the model struggles with, extracts the related knowledge concepts, and combines them to generate new training problems. It also filters the generated problems based on the target model's success rate. So I feel SwS is quite related to this work, and it would be interesting to see how MBS compares with it. Without such comparisons, I think it is still a bit unclear how much benefit comes specifically from the proposed decomposition and recursive recomposition, compared to other existing weakness-driven data synthesis approaches.
- I am also a bit concerned about the computational cost of MBS. The whole pipeline seems quite expensive, as it involves multiple teacher models, recursive problem decomposition, effectivity tests with student models, generating harder leaf problems, and then combining them into new parent problems. In particular, I imagine the bottom-up recomposition step could take quite a lot of time and compute, since the teacher needs to generate new parent problems based on the synthesized children while preserving the original reasoning structure. I noticed that Appendix C includes a comparison with direct synthesis under roughly matched construction token budgets, which is helpful. But I would still like to see more analysis of the overall cost of MBS, such as the total token usage, wall-clock time, the cost of different stages, and the average cost of generating one qualified problem. I think this would help us better understand how efficient the method actually is.
Questions for authors
One minor suggestion is that I think the paper could benefit from a more detailed case study. Currently, the authors mainly use framework figures and statistical results to explain MBS. It would be great to see a concrete example showing how an original hard problem is decomposed into simpler subproblems, how the new leaf problems are generated, and how they are combined into a new parent problem. I think such an example would make it much easier to understand the whole synthesis process, especially how the original reasoning blueprint is preserved during the recursive generation.
4 comments
Concrete case study. This is a good point. We will add a worked example to the supplementary material. It will show how one hard problem is decomposed into simpler subproblems, how the new leaf problems are generated, how they are combined into a new parent problem, and how the reasoning blueprint is preserved.
We hope this answers your questions. If anything is unclear, please let us know and we are happy to clarify. Please also consider updating your score if you feel it is appropriate.
Response to Author
Thank you for the detailed response and the additional experiments. The Goldilocks analysis and the comparison with direct seed synthesis are helpful, but they do not fully address my main concern about how MBS compares with existing data synthesis methods.
I understand that rerunning the full SwS pipeline and training gpt-oss-120B may not be feasible during the rebuttal period. Could you instead run MBS on one of the Qwen2.5 models for which SwS has released checkpoints, such as the 32B, 7B, or 3B model? Starting from the same base model and evaluating the resulting MBS and SwS checkpoints under the same conditions would provide a more direct comparison of downstream performance than the Goldilocks analysis alone.
On computational cost, the stage-by-stage breakdown is useful. My question, though, was about the trade-off between synthesis cost and downstream performance across methods. It would be helpful to see the total data construction cost for MBS and SwS, measured on the same basis, alongside the gains each method delivers over the same base model. If MBS yields only a modest additional gain at a substantially higher synthesis cost, that would temper my assessment of its practical value. Reporting cost and performance side by side would make that trade-off much easier to judge.
Thank you for the follow-up. We answer the two points below.
1. Running MBS on Qwen2.5 and comparing with the SwS checkpoints. We looked into this, however:
-
We could not find the released SwS checkpoints you refer to. The SwS repository releases the code and a demo set (500 samples per model and category), the HF account that hosts the demo set lists no models, and the dataset card says:
"The full dataset and model are currently under review by Microsoft and will be released once approved."
If we missed them, please point us to them. Without the checkpoints, the comparison means re-running SwS end to end (including the synthesis of their full dataset). Just for Qwen2.5-7B, SwS reports 2075 A100 GPU hours for weakness identification, 8292.6 for the problem synthesis stages and 9543 for the augmented RL training, which adds up to 19,910.6 (Table 2, NeurIPS version). It also means reproducing the SwS results ourselves first, with no released checkpoint or full dataset to validate the reproduction against.
-
A comparison on Qwen2.5 would measure a different setting. Our math data was synthesized and screened for gpt-oss-120b. One of the key findings of our paper is that the value of RLVR data is relative to the problem, the target policy and the rollout budget, and is not a property of the problems alone. Thus generated problems should be calibrated to the target policy and the training rollout budget (Section 2). The analysis in our first response shows how much this matters: problems that SwS kept at 25–75% accuracy for Qwen2.5-32B give zero observed group-relative advantage for gpt-oss-120b in 77.6% of the tested cases. So running MBS on Qwen2.5 is a new synthesis for a new policy, and its cost and gains would be those for Qwen2.5.
2. Cost and performance side by side. Putting both on the same basis, with gains over the same base model, depends on the comparison in point 1. The same-basis comparison we show in the paper is with direct seed synthesis, which uses the same teachers and the same target-policy filtering (Appendix C, Table 7, and Section 4.1). We also want to highlight that even if the SwS dataset and checkpoints were released and we ran the comparison on Qwen2.5, the result would be for Qwen2.5. As our characterization shows, the usefulness of the data depends considerably on the policy (changing the target policy reclassifies 32.3% of problems, Section 2), so a trade-off measured on Qwen2.5 need not hold for gpt-oss-120b, the target policy of our math experiments.
Finally, we would like to recall the contributions of our paper:
- A characterization of the value of RLVR data as relative to the problem, the target policy and the rollout budget. Your review notes this as a strength: "They also show how the difficulty of a problem changes with different target policies and rollout budgets, which gives a good motivation for their proposed method."
- A reinterpretation of hard problems: instead of discarding or filtering them, we use them as a specification of what data to synthesize.
- A data synthesis algorithm (MBS) that follows from this analysis. As Reviewer vApL puts it, this "creates a strong conceptual link between the paper's initial diagnosis and its algorithmic design."
- Beyond the specific implementation of MBS, in the words of Reviewer vApL, "the framework offers a valuable perspective for future work on RLVR data selection, curriculum construction, and policy-adaptive data synthesis."
- Results in two completely different domains and with 3 target policies. MBS improves gpt-oss-120b on all 9 math benchmarks (the 8 of Table 1 and Apex-Shortlist), and the gains remain at inference lengths beyond the training cap (Table 2). It also improves DeepSeek v4 Flash in math (Table 1) and Qwen3-4B on 2 finance benchmarks (Table 3).
We would appreciate it if the reviewer could weigh the paper on these contributions as a whole, and not only on a comparison with a baseline that was evaluated on Qwen2.5 models and whose full dataset and checkpoints have not been released. With what is released, the direct comparison available is on the data itself, which is the Goldilocks analysis of our first response. It already shows that the released problems we tested give zero observed group-relative advantage for our policy (gpt-oss-120b in 77.6% of the cases, so it does not escape the "zero-advantage trap" for our target policy. Furthermore, it reinforces our finding that the value of the data depends on the target policy.
Thank you very much for recognizing the analysis that motivates the method and the synthesis algorithm, for finding the experimental results promising, and for noting that the finance experiment shows the potential to use the algorithm beyond math.
We address your concerns below:
Related baseline and the contribution of decomposition.
We agree that SwS (cited in Section 5 as Liang et al., 2025) is closely related to our work. A key difference is that MBS keeps the structure of the solution and its reasoning blueprint. Specifically, we decompose the hard problem into a tree of subproblems, test that they help a student solve the parent, and recompose new problems that follow the same structure. We provide two analysis to address your concern with respect to this below.
1. Goldilocks analysis on SwS released data (new rebuttal experiment). Following your suggestion, we ran a new experiment with the data SwS released. For our target policy, 77.6% of the tested SwS problems give zero observed group-relative advantage (approximate 95% interval 72.0–82.3%): 112/250 are all-correct, 82/250 are all-incorrect, and 56/250 fall in the Goldilocks band.
Setup: 250 problems from the Qwen2.5-32B file of the public SwS demo release, selected at random before evaluation from the ones with recorded accuracy (model_acc) between 25%-75% (the band SwS own difficulty filter keeps). We use gpt-oss-120b with 8 rollouts, 16K limit.
Note: We acknowledge the Goldilocks analysis is not necessarily the same that downstream performance. It is a proxy. We also note that SwS released data was calibrated to their own target policy and not to gpt-oss-120b. Unfortunately, only a demo of the SwS data (500 samples for each model and category) is public, and it was generated for a different and smaller policy (Qwen2.5-32B). Regenerating the data for gpt-oss-120b means re-running the full SwS pipeline on the open data, and then training the policy, which takes a significant computational cost and is not possible within the rebuttal period. Instead, we show the goldilocks analysis as our best attempt to address your concern and below show an apple-to-apple analysis on the benefit of decomposition and recomposition using seed synthesis.
2. How much of the benefit comes from the decomposition and recomposition. To answer directly this question, we compare MBS with direct seed synthesis, which uses the same teachers and the same target-policy filtering but generates the new problems directly from the hard seed, without the tree and effectivity tests.
Construction cost. Following your recommendation, we report the total token usage, the cost of the different stages, and the average cost of generating one qualified problem. In the audit, the bottom-up recomposition step is the cheapest construction stage: 6.8% of the construction tokens. Notably, the largest share of the tokens (40.6%) is spent on the effectivity test. This stage is motivated by the controlled in-context experiment in Appendix B.3, and it checks that the subproblems help a student solve the parent.
The table below reports the average input plus output tokens per seed for each stage, measured on a 10 seed audit:
Note: the costs start from the verified teacher solution of the seed, which direct synthesis also needs. The trees in the audit have at most one level of composition which accounts for most of the qualified problems (Table 10).
For reference on efficiency: on the same 64 seeds, direct synthesis receives more construction tokens than MBS (16.0M versus 14.3M) and gives a candidate that preserves the blueprint, is novel, and is in the Goldilocks band for 13 seeds, versus 28 for MBS (Appendix C, Table 7).