acceptodds
Under review as a conference paper at ICLR 2027

MagicBeanStalk: Escaping the RLVR Zero Advantage Trap with Recursive Data Synthesis

Abstract

Data-centric scaling of RLVR has largely focused on collecting or generating verifiable training problems. However, corpus size alone does not determine whether the dataset provides a useful learning signal for a target policy. To guide the synthesis of effective RLVR data, we analyze the RLVR objective and experimentally characterize dataset effectiveness as a function of its problems, the target policy, and the training rollout budget. Under this characterization, we show that a large open-source math corpus pooled from widely used RLVR datasets falls almost entirely into a zero-advantage trap: for a fixed target policy and rollout budget, roughly 88% of its problems are solved in every rollout or failed in every rollout. Filtering concentrates what remains, but it can only select, discarding the hard problems. Effective synthetic problems should therefore require the reasoning operations and compositions required to solve those hard problems. Yet their difficulty must be calibrated, so that some target-policy rollouts succeed, some fail, and at least one complete correct solution fits within the training budget. Prompting a teacher to vary hard problems directly does not achieve this. Guided by these empirical design principles, we introduce MagicBeanStalk, a top-down decomposition and bottom-up recursive data-synthesis algorithm that grows seed problems into families of substantively new, policy- and budget-calibrated training problems while preserving the seeds' reasoning blueprints. Across 8 math reasoning benchmarks, continual RLVR on MagicBeanStalk problems improves gpt-oss-120b from 61.2 to 69.8 (+8.6) and DeepSeek v4 Flash from 72.4 to 77.9 (+5.4) on a 16K-token budget. Training on the full 78K source corpus improves gpt-oss-120b by only +1.1 and its best filtered subset by +5.8. In finance, we synthesize over 14K verifiable training problems from only 340 solutions and improve Qwen3 by +6.9 and +2.4 points on SecQue and FinanceBench, respectively.

Primary area:
foundation or frontier models, including LLMs
Keywords:
synthetic data, RLVR, reasoning models, LLM, data synthesis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_y8HmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies how to generate useful training data for RLVR. The authors point out that whether a problem can provide useful learning signals depends not only on the problem itself, but also on the target policy and the rollout budget. They find that around 88% of the problems in a 78K math dataset fall into what they call the "zero-advantage trap", where the model either solves all rollouts or fails all of them, and thus receives no group-relative advantage signal. To address this issue, the authors propose MagicBeanStalk (MBS), a recursive data synthesis framework that generates new training problems from hard seed problems. The main idea is to first decompose a hard problem into simpler subproblems, and then generate harder variants of these subproblems and recursively combine them into new parent problems. In this way, MBS tries to generate new problems that preserve the original reasoning blueprint while having an appropriate difficulty for the target policy. Experiments on math benchmarks show that MBS improves gpt-oss-120b and DeepSeek v4 Flash by 8.6 and 5.4 points, respectively. The authors also show some improvements in the finance domain.

Critical strengths
  • The authors show that simply having more training data does not necessarily help RLVR, since many problems may not provide useful learning signals for the current policy. They also show how the difficulty of a problem changes with different target policies and rollout budgets, which gives a good motivation for their proposed method.
  • Instead of directly generating variations of the original problem, MBS first decomposes it into simpler subproblems and then recursively combines newly generated subproblems into harder problems. I think this is an interesting way to preserve the reasoning skills required by the original problem while generating new training data. The effectivity test is also a nice design, since it checks whether the decomposed subproblems can actually help the student solve the original problem.
  • The experimental results are promising. MBS achieves 69.8% on gpt-oss-120b, which is 2.8 points higher than the strongest filtered-source baseline. The same synthesized data also improves DeepSeek v4 Flash by 5.4 points. I also find the results on MathArena Apex interesting, as the improvements are still quite clear with longer inference budgets (24K and 32K). The finance experiments also show that the method has some potential to work beyond math reasoning tasks.
Critical weaknesses
  • My main concern is about the baselines. Although the authors compare MBS with Direct Seed Synthesis, Goldilocks Filtering, and Golden Goose, I think it would be important to compare with other existing RLVR data synthesis frameworks that have similar motivations. For example, SwS (NeurIPS 2025) also starts from problems that the model struggles with, extracts the related knowledge concepts, and combines them to generate new training problems. It also filters the generated problems based on the target model's success rate. So I feel SwS is quite related to this work, and it would be interesting to see how MBS compares with it. Without such comparisons, I think it is still a bit unclear how much benefit comes specifically from the proposed decomposition and recursive recomposition, compared to other existing weakness-driven data synthesis approaches.
  • I am also a bit concerned about the computational cost of MBS. The whole pipeline seems quite expensive, as it involves multiple teacher models, recursive problem decomposition, effectivity tests with student models, generating harder leaf problems, and then combining them into new parent problems. In particular, I imagine the bottom-up recomposition step could take quite a lot of time and compute, since the teacher needs to generate new parent problems based on the synthesized children while preserving the original reasoning structure. I noticed that Appendix C includes a comparison with direct synthesis under roughly matched construction token budgets, which is helpful. But I would still like to see more analysis of the overall cost of MBS, such as the total token usage, wall-clock time, the cost of different stages, and the average cost of generating one qualified problem. I think this would help us better understand how efficient the method actually is.
Questions for authors

One minor suggestion is that I think the paper could benefit from a more detailed case study. Currently, the authors mainly use framework figures and statistical results to explain MBS. It would be great to see a concrete example showing how an original hard problem is decomposed into simpler subproblems, how the new leaf problems are generated, and how they are combined into a new parent problem. I think such an example would make it much easier to understand the whole synthesis process, especially how the original reasoning blueprint is preserved during the recursive generation.

4 comments

Thank you very much for recognizing the analysis that motivates the method and the synthesis algorithm, for finding the experimental results promising, and for noting that the finance experiment shows the potential to use the algorithm beyond math.

We address your concerns below:


Related baseline and the contribution of decomposition.

We agree that SwS (cited in Section 5 as Liang et al., 2025) is closely related to our work. A key difference is that MBS keeps the structure of the solution and its reasoning blueprint. Specifically, we decompose the hard problem into a tree of subproblems, test that they help a student solve the parent, and recompose new problems that follow the same structure. We provide two analysis to address your concern with respect to this below.

1. Goldilocks analysis on SwS released data (new rebuttal experiment). Following your suggestion, we ran a new experiment with the data SwS released. For our target policy, 77.6% of the tested SwS problems give zero observed group-relative advantage (approximate 95% interval 72.0–82.3%): 112/250 are all-correct, 82/250 are all-incorrect, and 56/250 fall in the Goldilocks band.

Setup: 250 problems from the Qwen2.5-32B file of the public SwS demo release, selected at random before evaluation from the ones with recorded accuracy (model_acc) between 25%-75% (the band SwS own difficulty filter keeps). We use gpt-oss-120b with 8 rollouts, 16K limit.

Note: We acknowledge the Goldilocks analysis is not necessarily the same that downstream performance. It is a proxy. We also note that SwS released data was calibrated to their own target policy and not to gpt-oss-120b. Unfortunately, only a demo of the SwS data (500 samples for each model and category) is public, and it was generated for a different and smaller policy (Qwen2.5-32B). Regenerating the data for gpt-oss-120b means re-running the full SwS pipeline on the open data, and then training the policy, which takes a significant computational cost and is not possible within the rebuttal period. Instead, we show the goldilocks analysis as our best attempt to address your concern and below show an apple-to-apple analysis on the benefit of decomposition and recomposition using seed synthesis.

2. How much of the benefit comes from the decomposition and recomposition. To answer directly this question, we compare MBS with direct seed synthesis, which uses the same teachers and the same target-policy filtering but generates the new problems directly from the hard seed, without the tree and effectivity tests.

  • With the same seed population and the same amount of qualified data, MBS reaches 69.2 versus 63.9 (+5.3 points) on the 4 core benchmarks (Section 4.1).
  • On the same subset of 64 seeds, MBS produces a candidate that preserves the blueprint, is novel, and is in the Goldilocks band for 28/64 seeds versus 13/64, even though direct synthesis receives more construction tokens (16.0M versus 14.3M, Appendix C-Table 7).

Construction cost. Following your recommendation, we report the total token usage, the cost of the different stages, and the average cost of generating one qualified problem. In the audit, the bottom-up recomposition step is the cheapest construction stage: 6.8% of the construction tokens. Notably, the largest share of the tokens (40.6%) is spent on the effectivity test. This stage is motivated by the controlled in-context experiment in Appendix B.3, and it checks that the subproblems help a student solve the parent.

The table below reports the average input plus output tokens per seed for each stage, measured on a 10 seed audit:

Stage, including its verification/answer checksTokens per seedShare
Decomposition58K26.2%
Effectivity tests91K40.6%
Leaf generation59K26.5%
Parent recomposition15K6.8%
  • Total construction tokens per seed: about 223K
  • Cost per qualified problem: the audit retains 5 qualified problems, which gives ~446K construction tokens per qualified problem
  • Wall-clock time: Wall-clock time depends heavily on implementation details, cluster congestion, and the number of instances that can run in parallel. In our scenario, on a shared cluster, we estimate about 0.4 minutes per seed with only 20 instances in parallel.

Note: the costs start from the verified teacher solution of the seed, which direct synthesis also needs. The trees in the audit have at most one level of composition which accounts for most of the qualified problems (Table 10).

For reference on efficiency: on the same 64 seeds, direct synthesis receives more construction tokens than MBS (16.0M versus 14.3M) and gives a candidate that preserves the blueprint, is novel, and is in the Goldilocks band for 13 seeds, versus 28 for MBS (Appendix C, Table 7).

Concrete case study. This is a good point. We will add a worked example to the supplementary material. It will show how one hard problem is decomposed into simpler subproblems, how the new leaf problems are generated, how they are combined into a new parent problem, and how the reasoning blueprint is preserved.


We hope this answers your questions. If anything is unclear, please let us know and we are happy to clarify. Please also consider updating your score if you feel it is appropriate.

Reviewer_y8HmOpenReview ↗

Response to Author

Thank you for the detailed response and the additional experiments. The Goldilocks analysis and the comparison with direct seed synthesis are helpful, but they do not fully address my main concern about how MBS compares with existing data synthesis methods.

I understand that rerunning the full SwS pipeline and training gpt-oss-120B may not be feasible during the rebuttal period. Could you instead run MBS on one of the Qwen2.5 models for which SwS has released checkpoints, such as the 32B, 7B, or 3B model? Starting from the same base model and evaluating the resulting MBS and SwS checkpoints under the same conditions would provide a more direct comparison of downstream performance than the Goldilocks analysis alone.

On computational cost, the stage-by-stage breakdown is useful. My question, though, was about the trade-off between synthesis cost and downstream performance across methods. It would be helpful to see the total data construction cost for MBS and SwS, measured on the same basis, alongside the gains each method delivers over the same base model. If MBS yields only a modest additional gain at a substantially higher synthesis cost, that would temper my assessment of its practical value. Reporting cost and performance side by side would make that trade-off much easier to judge.

Thank you for the follow-up. We answer the two points below.


1. Running MBS on Qwen2.5 and comparing with the SwS checkpoints. We looked into this, however:

  • We could not find the released SwS checkpoints you refer to. The SwS repository releases the code and a demo set (500 samples per model and category), the HF account that hosts the demo set lists no models, and the dataset card says:

    "The full dataset and model are currently under review by Microsoft and will be released once approved."

    If we missed them, please point us to them. Without the checkpoints, the comparison means re-running SwS end to end (including the synthesis of their full dataset). Just for Qwen2.5-7B, SwS reports 2075 A100 GPU hours for weakness identification, 8292.6 for the problem synthesis stages and 9543 for the augmented RL training, which adds up to 19,910.6 (Table 2, NeurIPS version). It also means reproducing the SwS results ourselves first, with no released checkpoint or full dataset to validate the reproduction against.

  • A comparison on Qwen2.5 would measure a different setting. Our math data was synthesized and screened for gpt-oss-120b. One of the key findings of our paper is that the value of RLVR data is relative to the problem, the target policy and the rollout budget, and is not a property of the problems alone. Thus generated problems should be calibrated to the target policy and the training rollout budget (Section 2). The analysis in our first response shows how much this matters: problems that SwS kept at 25–75% accuracy for Qwen2.5-32B give zero observed group-relative advantage for gpt-oss-120b in 77.6% of the tested cases. So running MBS on Qwen2.5 is a new synthesis for a new policy, and its cost and gains would be those for Qwen2.5.

2. Cost and performance side by side. Putting both on the same basis, with gains over the same base model, depends on the comparison in point 1. The same-basis comparison we show in the paper is with direct seed synthesis, which uses the same teachers and the same target-policy filtering (Appendix C, Table 7, and Section 4.1). We also want to highlight that even if the SwS dataset and checkpoints were released and we ran the comparison on Qwen2.5, the result would be for Qwen2.5. As our characterization shows, the usefulness of the data depends considerably on the policy (changing the target policy reclassifies 32.3% of problems, Section 2), so a trade-off measured on Qwen2.5 need not hold for gpt-oss-120b, the target policy of our math experiments.


Finally, we would like to recall the contributions of our paper:

  • A characterization of the value of RLVR data as relative to the problem, the target policy and the rollout budget. Your review notes this as a strength: "They also show how the difficulty of a problem changes with different target policies and rollout budgets, which gives a good motivation for their proposed method."
  • A reinterpretation of hard problems: instead of discarding or filtering them, we use them as a specification of what data to synthesize.
  • A data synthesis algorithm (MBS) that follows from this analysis. As Reviewer vApL puts it, this "creates a strong conceptual link between the paper's initial diagnosis and its algorithmic design."
  • Beyond the specific implementation of MBS, in the words of Reviewer vApL, "the framework offers a valuable perspective for future work on RLVR data selection, curriculum construction, and policy-adaptive data synthesis."
  • Results in two completely different domains and with 3 target policies. MBS improves gpt-oss-120b on all 9 math benchmarks (the 8 of Table 1 and Apex-Shortlist), and the gains remain at inference lengths beyond the training cap (Table 2). It also improves DeepSeek v4 Flash in math (Table 1) and Qwen3-4B on 2 finance benchmarks (Table 3).

We would appreciate it if the reviewer could weigh the paper on these contributions as a whole, and not only on a comparison with a baseline that was evaluated on Qwen2.5 models and whose full dataset and checkpoints have not been released. With what is released, the direct comparison available is on the data itself, which is the Goldilocks analysis of our first response. It already shows that the released problems we tested give zero observed group-relative advantage for our policy (gpt-oss-120b in 77.6% of the cases, so it does not escape the "zero-advantage trap" for our target policy. Furthermore, it reinforces our finding that the value of the data depends on the target policy.

Reviewer_1yVtOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MagicBeanStalk (MBS), a data synthesis method for more effective RLVR training. The authors aim to address the zero advantage trap, a common problem in RLVR, where a model generates a group of rollouts for a specific data sample and only generates either fully solved groups or fully unsolved groups. The authors argue that the problems of the unsolved groups are useful, as they reveal the missing capabilities of the target policy. MBS employs a pool of teacher models, which can solve the zero-advantage hard problems and decompose the problem into a tree structure recursively. These teacher models are asked to generate variants of the subproblems and then create new root problems by combining these variants. Only new problems where the target policy succeeds in some but not all rollouts are used for RLVR training. The experiments show that the target model reaches higher performance on both math and finance problems.

Critical strengths

The method is overall sound, and tree-based decomposition and construction are well designed. MBS produces problems that are both new and preserve the seed's reasoning.

Critical weaknesses

It is possible that the target policy is learning to finish its answer within the budget, rather than learning to reason better. In Table 2, the gain of MBS over the base model becomes smaller as the inference budget grows. It is +8.6 at 8K and +8.3 at 16K, but only +1.0 at 24K and +1.5 at 32K. If training improves reasoning ability, the gain should remain at larger budgets. The authors should provide the response length and truncation rate before and after training. What is the truncation rate of your rollouts. What happens if the model is truncated mid-reasoning? Is the sample dropped and scored as 0?

Having evaluated gpt-oss-120b on AIME as well, I am aware that the model literally aces these tests, scoring above 90% across them (AIME 24,25,26) with reasoning set to high and on AVG@8. How many eval passes do you do for Aime, and what budget is the evaluation? How high are the truncation rates in the evals? I believe oss120b is already quite strong, and it will be hard to push math even further on this model.

Questions for authors
  • "Section ??" showed in Appendix.
  • Reference error: Figure 2 a should be Figure 3 at line 211.
  • Tables 5 and 6 duplicate Tables 8 and 9.
3 comments

Thank you for the careful review, and for finding the method sound and the tree-based decomposition well designed.

We ran the length and truncation analysis you asked for and answer each question below.


1. “It is possible that the target policy is learning to finish its answer within the budget, rather than learning to reason better.”

To answer directly, our results support both improved mathematical reasoning performance and improved response efficiency. Models trained on MBS data answer hard problems more accurately at matched budgets, and they do it with shorter responses and fewer truncations. We believe this is very much the desired outcome: improved reasoning and improved efficiency go together.

We support these two findings separately below.

Reasoning performance. In addition to the average gains at different inference budgets recognized by the reviewer, Table 2 also shows gains on Apex-Shortlist, including beyond the 16K training budget. Apex-Shortlist is a well known harder math benchmark and the gains are substantial. Notably, the biggest gains in this scenario are at 24K which differs from the 16K training budget. We believe this is a strong result showing improved reasoning performance beyond the training budget. These scores average 16 responses per problem to account for randomness.

Inference budgetBase (%)MBS (%)Gain (points)
16K1.512.5+11.0
24K6.820.1+13.3
32K16.227.1+10.9

At 32K, MBS also exceeds the strongest reported training baseline by 6.5 points. The smaller (although positive) 8-benchmark average gap does not negate these gains on harder mathematics. An additive gap need not remain constant as both policies receive more inference compute. Importantly, MBS-trained models are the only ones with positive improvements at the 32K budget in the 8-benchmark average.

Response efficiency (new rebuttal experiment). Following your recommendation, we ran a fresh evaluation on the saved checkpoints analyzing mean, median and number of tokens as well as % of truncation. Together with Tables 1 and 2, this experiment shows MBS-trained models reach higher accuracy with shorter responses and fewer truncations. The arrow in the tables represents the comparison between based and MBS trained model, shown as base → MBS:

Benchmark (cap)Mean tokensMedian tokensTruncated (%)
AIME 2024 (16K)10,000 → 9,5339,683 → 8,61228.8 → 24.2
AIME 2025 (16K)10,900 → 10,24512,590 → 10,61138.8 → 28.5
Apex-Shortlist (32K)30,809 → 25,34932,768 → 25,96075.9 → 29.3

Setup: all 60 AIME and 47 Apex problems, high reasoning, 16 responses per problem. Tokens count reasoning + answer. Truncated means the response reached the cap.

2. "How many eval passes do you do for Aime, and what budget is the evaluation? How high are the truncation rates in the evals?"

16 samples per problem, high reasoning, and a 16,384-token cap on the full response (reasoning plus answer). We average correctness over the samples to estimate pass@1. At this cap the base is truncated on 28.8% (AIME 2024) and 38.8% (AIME 2025) of responses, and MBS on 24.2% and 28.5% (table above).

We believe the cap may explain the difference from the >90% you observed. Across the 8 benchmarks the base goes from 61.3 at 16K to 75.9 at 32K (Table 2). We train at 16K for compute reasons (Section 4), so Table 1 reports the training cap and Table 2 the larger ones.

I believe oss120b is already quite strong, and it will be hard to push math even further on this model.

We agree it is strong on AIME, this is why we also report Apex-Shortlist. There the base reaches only 16.2% even at 32K, and MBS raises it to 27.1% (Table 2). In addition to AIME and Apex, we evaluate on 6 other math benchmarks. Beyond math, we also evaluate MBS on two finance benchmarks.

3. "What is the truncation rate of your rollouts. What happens if the model is truncated mid-reasoning? Is the sample dropped and scored as 0?"

Responses are not discarded solely because they reach the cap. We follow the standard procedure, and it is the same for every model. A response is scored 0 if no answer can be parsed, and if an answer is parsed it is scored 1 when correct and 0 otherwise. Truncation rates are in the table above. On the training side, every MBS problem has at least one correct target-policy rollout that ends normally within 16K (in-budget solvability, Section 2).

4. Presentation corrections. Thank you for catching these. We apologize. We will resolve the "Section ??" references, correct the figure cross-reference at line 211, and remove the duplicated tables in the camera ready.


We hope this answers your questions. If anything is unclear, or there is another analysis you would like to see, we are happy to run it. Otherwise, we would be grateful if you would consider updating your score.

Thank you for the response. The new length analysis makes my concern more concrete rather than resolving it. The gains mix two things, solving more problems and finishing within the token cap. Currently, your evaluation does not separate them. Thus, we can not tell whether in any of the setups the model actually improves reasoning or just learns to answer within the token cap of the evaluation.

This is especially concerning since the authors claim to improve gpt-oss-120b from 70/63.3 to 80.2/78.8 on AIME 2024 / 2025. The exact same model run at native context length measures 92.9 / 92.9 on AIME 2024 / 2025 in our internal eval. While I do not have numbers for any other evaluation results, I wonder whether this is also the case on the other evaluations? What happens if you run the model at native context length on them? How does MBS compare to the base model there? Are the gains only coming from the defined token caps? Why modulate the reasoning effort over the token cap and not use the model reasoning effort parameter? This makes the efficiency analysis is even more concerning since the model is run at high reasoning effort. This also questions whether the model is actually more efficient. E.g. if deployed at medium effort, it might be even more token efficient. Furthermore, it might now also terminate within the token cap of the authors.

As it stands, the evidence shows the MBS model finishes within budget more often, not that it reasons better. I keep my score until a comparison without the truncation confound is available.

Thank you for the follow-up. We answer each question below.


1. "What happens if you run the model at native context length?" This is a different setting from the one our paper studies. We synthesize and train at a 16K rollout budget, and every math training condition, including the baselines, uses this same budget. Extrapolation to 8x the training budget at inference (native 128K) is a completely different problem and open question. We augment on this below:

  • Our data is calibrated to the training budget. One of the key findings of our paper is that difficulty is a property of the problem, the target policy and the rollout budget, so generated problems should be calibrated to the target policy and the training rollout budget (Section 2). Our math data was synthesized for gpt-oss-120b at 16K.
  • The matched test at 128K is to synthesize data for 128K and train at 128K. MBS could be used in this setting the same way. We train at 16K for compute reasons (Section 4). This is the reason we synthesize data at 16K. RL training of gpt-oss-120b with 128K rollouts requires massive compute and infra.
  • Our results are stated at a response budget. The abstract reports the gains "on a 16K-token budget" and Table 1 is "at 16K output tokens". The paper also reports that the base improves with the budget, from 61.2 at 16K to 78.9 at 64K on the 8 benchmarks (Appendix B.2). We do not claim anywhere we extrapolate 8x or are better at 8x the training budget as this is a different problem.

Beyond the training budget, the paper evaluates up to 32K, 2x the training budget, and the MBS gains remain (Table 2). Notably, this is not always the same for baselines that were given the same conditions for synthesis and training.

2. "Are the gains only coming from the defined token caps?" No. Every training condition and baselines have the same 16K cap, the same RL implementation and hyperparameters, and the same evaluation. Only the training data changes. If the gains came from the cap, every condition would show them. Training on the full source corpus under the same cap gains only +1.1, against +8.6 for MBS (Table 1). Beyond the training cap, the in-budget source subset and seed synthesis fall to or below the base on the 8-benchmark average , and MBS stays above it (Table 2).

3. "Why modulate the reasoning effort over the token cap and not use the model reasoning effort parameter?" We do not change the reasoning effort. It is same for training and in every evaluation, for the base and for all trained models (including baselines data synthesis methods). The token cap is the RL rollout budget (Section 4). It is a hard limit on the compute per rollout, which the effort parameter does not give.

4. "The MBS model finishes within budget more often, not that it reasons better." A response is scored only on a correct final answer, so finishing within the budget gives nothing if the answer is wrong. What we show is more problems solved correctly, with fewer tokens, under the same budget. We believe reasoning better and reasoning efficiently go together, even more in practice, where inference compute is a limiting factor. And even if the gains are read only as efficiency, the baselines trained under the same cap do not show it: on the 8-benchmark average their gains disappear beyond the training cap. We believe the right comparison to judge this would be vs the other data synthesis baselines.

5. Beyond AIME 2024/2025. AIME 2024/2025 are 2 of the 8 math benchmarks. The paper's results cover 8 math benchmarks, Apex-Shortlist, a second target policy (DeepSeek v4 Flash), and 2 finance benchmarks with Qwen3-4B ( SecQue, FinanceBench). On the Apex results (Table 2), Reviewer y8Hm notes that "the improvements are still quite clear with longer inference budgets (24K and 32K)."

Further, our contribution shows 1) a characterization of the value of RLVR data as relative to the problem, the target policy and the rollout budget. Reviewer y8Hm lists this as a strength: "They also show how the difficulty of a problem changes with different target policies and rollout budgets, which gives a good motivation for their proposed method." 2) a reinterpretation of hard problems: instead of discarding or filtering them, we use them as a specification of what data to synthesize. 3) A data synthesis algorithm (MBS) that follows from this analysis. As Reviewer vApL puts it, this "creates a strong conceptual link between the paper's initial diagnosis and its algorithmic design.", and Reviewer y8Hm describes it as "an interesting way to preserve the reasoning skills required by the original problem while generating new training data." 4) Beyond the specific implementation of MBS, in the words of Reviewer vApL, "the framework offers a valuable perspective for future work on RLVR data selection, curriculum construction, and policy-adaptive data synthesis."

Reviewer_vApLOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates why simply scaling up reinforcement learning with verifiable rewards (RLVR) datasets may not necessarily improve reasoning models. The authors find that, for a fixed target policy and response-length budget, most problems in an open-source corpus of approximately 78,000 math problems produce rollout groups that are either entirely correct or entirely incorrect, and thus provide no group-relative advantage signal. Based on this observation, the authors propose three principles for data synthesis. First, hard problems and their verified solutions can serve as capability specifications, indicating which reasoning operations and compositions the synthesized data should target. Second, synthetic problems should preserve the reasoning blueprint of their seed problems while remaining substantively novel in their problem settings. Third, the difficulty of synthetic problems should be calibrated to the target policy and training response budget. Based on these principles, the authors introduce MagicBeanStalk. The method first decomposes verified hard seed problems top-down into tree structures consisting of effective subproblems. It then generates harder, novel leaf problems and recursively composes them bottom-up to construct new problems that preserve the original reasoning structure while differing substantively in content. Finally, it retains only those problems that produce both successful and unsuccessful rollouts under the target policy and for which at least one correct rollout terminates normally within the training budget.

Critical strengths

The paper’s most important conceptual contribution is its characterization of RLVR data utility as a property jointly determined by the problem, the target policy, and the response budget. Under binary rewards and group-relative advantage estimation, a problem provides no within-group advantage signal when all sampled responses receive the same reward—that is, when they are either all correct or all incorrect. The paper shows that this is not a marginal phenomenon. With gpt-oss-120b, a 32K-token response limit, and eight rollouts per problem, approximately 88% of a 78K-problem open-source math corpus is empirically classified as either all-success or all-failure. Only 12.2% falls within the observed Goldilocks band, where mixed rollout outcomes yield a nonzero group-relative advantage signal.

The controlled experiments further show that these categories are not fixed properties of the problems themselves. Holding the problem set and response budget constant, changing the target policy reclassifies 32.3% of the problems across the all-failure, Goldilocks, and all-success regions. Likewise, holding the policy fixed while increasing the response budget from 16K to 64K reduces the fraction classified as hard from 33.1% to 10.6%. These results turn the intuitive claim that data utility depends on the policy and budget into a concrete, measurable finding, and provide a clear design rationale for policy- and budget-aware data selection and synthesis.

Equally important, the paper offers a useful reinterpretation of hard problems. Rather than treating them only as unusable training examples, it views their verified solutions as capability specifications that identify the reasoning operations—and compositions of operations—that new training data should target. This perspective naturally motivates the data-generation problem addressed by MagicBeanStalk and creates a strong conceptual link between the paper’s initial diagnosis and its algorithmic design. This connection is a major reason I lean toward acceptance. Even apart from the specific implementation of MagicBeanStalk, the framework offers a valuable perspective for future work on RLVR data selection, curriculum construction, and policy-adaptive data synthesis.

Critical weaknesses
  1. The stability of the difficulty labels may be insufficient, as they are based on empirical estimates derived from only eight rollouts. The paper classifies problems into "difficult," "Goldilocks," or "mastered" categories based on eight rollouts of the target policy. The paper’s own results indicate that when resampling—while keeping the base policy constant—9.5% of the problems initially labeled as "0/8" (i.e., zero correct responses) yielded at least one correct answer. This suggests that the classification results may be subject to noise from limited sampling, as well as potential regression-to-the-mean effects. Given that the reported 88% "zero-success" rate, the seed transition rates, and the final synthetic data filtering process all rely on this classification scheme, the authors should provide further evidence—such as repeated measurements, uncertainty estimates, or sensitivity analyses using a larger number of rollouts—to verify that the primary conclusions are robust against classification noise.
  2. The reported downstream performance gains lack sufficient statistical and computational controls. The paper does not report results from multiple independent RL training runs, nor does it provide confidence intervals or significance tests. Furthermore, key experimental parameters—such as training set size, number of optimization steps, total number of sampled tokens, and overall computational cost—are neither fully matched nor disclosed. Additionally, the selection of checkpoints relies on a held-out synthetic subset; however, the construction of this subset, its degree of isolation from the training data, and its applicability across different methods remain inadequately explained. Consequently, it is difficult to determine whether the advantages of MagicBeanStalk over strong baselines stem from its decomposition and recombination mechanism, or whether they are partially influenced by differences in the volume of qualified data, the extent of exposure during training, or the criteria used for checkpoint selection.
Questions for authors

How was the held-out synthetic subset used for early stopping constructed? Was it disjoint from the training problems, their source seeds, and generated descendants? Were the same validation set and stopping criterion used across all training conditions? Please also clarify whether this subset was generated by MagicBeanStalk or through a method-neutral procedure, as its construction could affect the relative stopping behavior across methods.

What checks were performed to detect overlap between the source or synthetic training problems and the evaluation benchmarks? Please clarify whether exact-match, near-duplicate, formula-structure, or semantic-similarity checks were conducted against the math benchmarks, SecQue, and FinanceBench. If any overlap was detected, how were the corresponding training examples handled? This clarification would materially affect confidence that the reported gains reflect generalization rather than benchmark contamination.

1 comment

Thank you very much for the positive feedback on the paper, recognizing the characterization of RLVR data usefulness as a property of the problem, the target policy and the response budget, and for finding the view of hard problems as capability specifications valuable.

We answer each concern and question below.


1. "The stability of the difficulty labels may be insufficient"

We agree the labels are empirical finite estimates. We actually state this in Section 2 (line 135):

These are empirical finite-sample classifications depending on rollout counts, they do not establish absolute inability/mastery

That said, to answer directly, our conclusions do not rely on the labels being exact:

  • Repeated measurement and regression to the mean. The 9.5% you mention is our control for this effect. Thank you for noticing this experiment! In Table 6 we re-sample the hard seeds with the base policy, and we compare the trained policies with this fresh sample (25.0% and 23.2% versus 9.5%), not with the original 0/8 label. It is also a repeated measurement: 90.5% of the seeds are 0/8 again, and the base policy is correct in only 1.7% of the rollouts.
  • Seed conversion. Both methods start from the same seeds and use the same 8-rollout rule, so label noise affects both equally. MBS converts 74.7% of the seeds (approx 95% interval 72–77%) versus 43.8% (41–47%) for seed synthesis (Table 5).
  • Final filtering. A retained problem has at least one correct and one incorrect target-policy response, so both outcomes are observed. The same rule is used for every filtered condition. Notably, subsets selected like this outperform the full proxy while using an order of magnitude fewer problems (Figure 3b).

2. "The reported downstream performance gains lack sufficient statistical and computational controls"

To answer directly whether the gain comes from the volume of qualified data, the training exposure or the checkpoint selection: the matched control in Section 4.1 holds all three fixed. Every condition starts from the same ckpt, uses the same seed population and the same amount of qualified data, and trains for one epoch. Each condition is reported at its best checkpoint (Appendix C) and Figure 3(a) shows the full curves. MBS reaches 69.2, versus 63.9 for seed synthesis and 65.3 for the source seeds. Further, Figure 3(a) also shows standard deviation over 3 eval passes as a shaded area.

We agree that multiple independent RL training runs would be ideal, but we did not have the compute to repeat RL training at this scale multiple times. That said, we emphasize Table 1 and this control are separate training runs! (Sec. 4.1), and both give the same ordering, with MBS first. The gpt-oss-120b results in Table 1 and the Apex results average 16 samples per problem. All conditions use the same RL implementation and hyperparameters. See Sec 4.2, Appendix C


3. "How was the held-out synthetic subset used for early stopping constructed?"

The held-out subset is a fixed set of 80 synthetic problems verified by multiple LLMs together with non-expert humans, disjoint from the training data, the seeds, and the evaluation benchmarks. None of those problems were used as seeds to further generate synthetic data either. We use the same subset and the same stopping rule throughtout the work across all training conditions as mentioned in the paper.


4. "What checks were performed to detect overlap between the source or synthetic training problems and the evaluation benchmarks?"

Of the checks you list, we ran exact-match and near-duplicate checks for this rebuttal. We audited the complete MBS math training data and the generated MBS finance data against the 8 math benchmarks, Apex-Shortlist, SecQue and FinanceBench, using normalized exact matching and character 5-gram Jaccard at a threshold of 0.6. Neither check found a match.

We also note that in the matched control seed synthesis and source-seed training use the same seed population as MBS, so any overlap at the seed level might equally help them at least as much (seed synthesis often do surface level varations) (see Sec 4.1).


We hope this answers your questions. If anything is unclear, we are happy to clarify. Please let us know!

Comments on the paper

1 comment
Area_Chair_SVPTOpenReview ↗

AC Note: Rebuttal Received, Please Engage in Discussion

Dear authors and reviewers,

Thank you to the authors for their detailed rebuttal and for the time and care they put into addressing the reviewers' comments. I appreciate the additional clarifications and results.

Reviewers: please read the authors' responses carefully and reply to them before the end of the discussion period. Even a short acknowledgment is valuable. It is especially helpful to say whether your concerns have been addressed and, if not, what remains unresolved. If the rebuttal changes your assessment, please update your score and review accordingly, whether up or down. If your score stays the same, a brief explanation of why helps me and the senior area chairs.

Authors: I will follow the discussion closely and take your rebuttal fully into account when forming my recommendation.

Thank you all for your time and effort.

Best regards, Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.