acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Reinforcement Learning: Scaling LLM Training with Self-Generated Data

Abstract

The scaling laws are a fundamental principle of large language models (LLMs), showing that model performance improves consistently as the amount of training data increases. In this paper, we propose Dynamic Reinforcement Learning (DynamicRL), a method that advances the scalability of RL for training LLMs autonomously. Unlike conventional approaches, DynamicRL does not require pre-collected datasets, human-labeled golden answers, or external verifiers for correctness; instead, it trains on self-generated data. DynamicRL operates by sampling data from the model as it evolves and using this self-generated data to optimize the model. Its dynamic characteristic allows the data distribution to continuously adapt to the evolving model, fostering better alignment between the training data and the model's capabilities. Experimental results demonstrate that DynamicRL can continuously improve model performance over more than a thousand training steps and achieve results comparable to models trained on large-scale external datasets.

Primary area:
reinforcement learning
Keywords:
Reinforcement Learning, Large Language Model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_DVcUOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DynamicRL, a fully unsupervised RL post-training method for LLM via self-play. This method aims to address the challenge of scaling up training data without the expensive human-crafted tasks and human-labeled solutions. At each training step, the tasks (questions) are generated by the model dynamically, while the proxy ground truth is estimated via majority voting. The reward design consist of two parts: one measures the question quality via agreement rate while the other measures the answer quality via the equivalence with proxy ground truth. To avoid mode collapse, two diversity terms that encourage diverse questions and answers are added to the question reward term. A hyperparameter is introduced to control the progression of question difficulty and prevent reward hacking. The reported experiments show that DynamicRL achieves similar performance compared to training on large datasets with golden answers.

Critical strengths
  1. The motivation of this paper is highly relevant to the future development of RL post training.
  2. The presentation is mostly clear and easy to follow.
Critical weaknesses
  1. Lack of novelty and limited contributions: Most elements in this paper are either standard or have been considered in the literature. Estimating the gold answer via majority voting have been widely adopted (e.g. [1][2][3]). The question reward term RqR_q is the same as the "uncertainty reward" in R-Zero [3]. The main contribution appears to be the remedy for mode collapse which does not seem justified either (see weakness #2). Beyond the question rewards, the setting is almost identical to [2] (which the authors do not cite). The limited initial data considered in [2] can be viewed as a carefully designed initial prompt distribution p(z)p(z) in this work (see minor weakness #1). Also see Question #2 regarding reward hacking.
  2. Unjustified diversity reward term RdqR_{dq}: The proposed question similarity ρq\rho_q does not seem appropriate as it does not capture semantic similarity at all. For any given question, simply prepending a few tokens or rephrasing it would lead to very low similarity score despite having almost identical meaning. The authors claim that embedding-based similarity metric achieves similar performance without providing any evidence. This reward term is supposed to be the main mechanism to alleviate mode collapse, but it is clearly theoretically unsound.
  3. Issues with baselines and experiments: A comparison with the closely related work [2] is missing. This severly undermines the evaluation of whether the question rewards are more effective compared to the filters considered in [2]. More details on self-play baselines should also be given. In particular, this work uses a different objective than [2][3][4] with an unnormalized advantage and no policy ratio clipping. The choice of optimization function should be made explicit and ablated.

Minor Weaknesses:

  1. Trivial initial distribution p(z)p(z): There are good reasons to consider non-deterministic pp. One obvious example is to generate data on tasks in different domain and/or tasks of varying difficulties. Designing the distribution p(z)p(z) would be an important aspect to consider in this line of work (e.g. [2]). Using p(z):=δz0p(z):=\delta_{z_0} necessarily places major limitation on the resulting data distribution and the evidence for the applicability of this method.
  2. Inconsistent definition of objective functions: Section 2.1 introduces the objectives (1)(2), whereas the actual objective (13) being optimized contains extra terms which are not mentioned in Section 2.1. This can lead to confusion, especially given the technically incorrect sentence "DynamicRL jointly optimizes Js(θ)J_s(\theta) and Jq(θ)J_q(\theta)" at Line 124.
  3. Unaddressed dependency of Ls(θ)\mathcal{L}_s(\theta) on initial prompt zz: The definition of Ls(θ)\mathcal{L}_s(\theta) in Eq (13) depends on the initial prompt zz, which the authors do not make explicit. In the paper, experiments are done with p(z)p(z) being supported on one single prompt z0z_0 so there is no actual difference. However, this definition is not sound when pp is non-deterministic, unless (for example) an expectation over p(z)p(z) is added.
  4. Repeated notation: The notation p(z)p(z) was used in Eq (2) as the probability distribution over initial prompts. In Eq (7), p(q)p(q) is defined as the mean solution reward. This leads to confusion even though the two pp's take symbolically different arguments and should be changed.
  5. Typo: τz\tau_z at Line 330 should be τe\tau_e
  6. Eq (14) can benefit from changing the parameter θ\theta in the expectation subscript to θold\theta_\text{old} to be more technically sound (other than simply saying that the distributions are held fixed).
Questions for authors
  1. The results reported in Fig 1 (in particular the semi-dynamic ones without golden answer) have different number of training steps, with fewer iterations for smaller datasets. What is the stopping criterion, and how do the authors ensure the comparison is fair?
  2. The authors claim that semi-dynamic RL without golden answer collapses due to reward hacking. This issue appears to be alleviated by the question difficulty control in the dynamic setting. However, [2] achieves the same goal by only including questions of moderate agreement rate (without using question rewards). Do the authors have experimental results along this line of consideration?
  3. Non-standard choice of objective: In relevant recent work [2][3][4], the advantage is computed with standard deviation normalization, and PPO-style clipping is used for the probability ratio between the new and old policies. Why do the authors adopt unnormalized advantage and exclude the probability ratio clipping?
  4. Generalization to other benchmarks: Beyond mode collapse, training solely on synthetic data might degrade model performance on out-of-distribution tasks. It would be helpful even to include some other reasoning-intensive tasks like coding.
  5. As it is rarely the case that zero training data is available, what is the advantage of this work over the few-shot consideration in [2]? Small question datasets without ground truth are relatively inexpensive to obtain while giving more guided directions in the initial phase of training.
Reviewer_3iFsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DynamicRL, a method for mathematical reasoning self-evolution in which the model generates both questions and solutions by itself. The method jointly optimizes a Questioner and a Solver: the Questioner generates questions, the Solver answers each question multiple times, and a pseudo-gold answer is estimated by majority voting over the answers. The Questioner receives a difficulty reward based on the Solver's answer agreement rate, while the Solver receives a reward based on whether its answer matches the majority answer. To mitigate reward hacking and mode collapse, the method also adds question filtering, question and answer diversity rewards, and a difficulty regulation mechanism. The authors evaluate DynamicRL on multiple mathematical and general reasoning benchmarks. The results show that it outperforms several self-evolution methods that do not use gold answers, but its overall performance remains slightly below the strongest baseline trained with human gold answers.

Critical strengths
  • It directly addresses reward hacking in self-generated data. The paper analyzes problems such as majority voting being possibly wrong, questions possibly lacking a unique answer, and the model being able to manipulate the answer agreement rate by generating sub-questions, and designs filtering and difficulty-control mechanisms to handle them.
  • It jointly considers question generation and solving capability. DynamicRL does not only train the Solver, but also incorporates the Questioner into the optimization process, allowing the training data to adjust with the model's capability and attempting to avoid learning stagnation caused by continuously using fixed-difficulty questions.
  • It designs dedicated components and ablations for mode collapse. The paper designs question diversity and answer diversity rewards separately, and reports that removing these components leads to duplicated questions or converging answers, supporting their role in mitigating mode collapse.
  • It achieves competitive results in the setting without human gold answers. DynamicRL outperforms several self-evolution baselines overall on Qwen2.5Math-1.5B and 7B, and is evaluated on multiple mathematical and general reasoning benchmarks. The authors also explicitly acknowledge that it has not yet surpassed the best method trained with human gold answers.
Critical weaknesses
  • Majority-voting pseudo-labels carry a fundamental reliability risk. If the current model systematically gives the same wrong answer to a question, majority voting will treat the wrong answer as the gold answer, and the Solver reward may reinforce the erroneous reasoning. Difficulty control can reduce this risk, but cannot guarantee that the majority answer is correct.
  • The answer agreement rate is not equivalent to question difficulty or quality. As the paper itself notes, non-unique answers, sub-questions, or formulation issues can all cause answer disagreement; even with filtering, the current signal may still misjudge ambiguous or flawed questions as high-difficulty ones.
  • The diversity rewards may encourage superficial variation rather than valuable new capabilities. The question and answer diversity rewards are based on similarity computation; answer diversity in particular relies on the pseudo-answers obtained from majority voting. Without quality constraints, rewarding different answers or more unique questions may introduce noise, or even conflict with the correctness objective.
  • The method's effective scope remains mainly limited to mathematical reasoning. The reward design relies on tasks where answers can be extracted, compared, and have a unique solution. For tasks with open answers, requiring long-form reasoning, or not verifiable through final answers, the applicability of the current approach is unclear; the results on some math benchmarks are also not comprehensively leading.
Questions for authors
  • Regarding majority-voting pseudo-labels: Have the authors estimated the proportion of cases where the majority-voted answer disagrees with the true answer? When the majority answer is wrong, how do the filtering or training mechanisms avoid treating the wrong answer as a positive reward and reinforcing erroneous reasoning?
  • Regarding the question difficulty signal: How strong is the correlation between the answer agreement rate and independently measured question difficulty and question quality? Can the filtering mechanism distinguish between "genuinely difficult but valid questions" and "ambiguous questions, questions without a unique answer, or questions with multiple sub-questions"?
  • Regarding the diversity rewards: How do the authors confirm that the observed diversity gains reflect effective reasoning paths or question coverage, rather than wording variation, meaningless variants, or the scattering of wrong answers? Has the trade-off between correctness and diversity been evaluated under different similarity metrics and reward weights?
  • Regarding task generalization: Can the method be evaluated on tasks beyond mathematics, where answers are not easily extractable or do not have a unique answer? If majority voting and exact answer matching cannot be used, how should DynamicRL's reward and filtering mechanisms be adjusted?
Reviewer_VKc5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper presents DynamicRL, a self-training framework in which a shared language model generates mathematical questions and learns from majority-vote pseudo-labels. The framework combines an uncertainty-based question reward, question and answer diversity rewards, question filtering, and adaptive reward weighting. Experiments on mathematical reasoning benchmarks report improvements over the base models and several training baselines.

Critical strengths
  1. The paper addresses a relevant problem: improving reasoning through self-generated training data while mitigating unreliable supervision and mode collapse.
  2. The reported performance gains are encouraging, and the component ablations provide evidence that the proposed modifications are useful within the overall framework.
Critical weaknesses
  1. The methodological contribution appears incremental, and the distinction from closely related work is insufficiently established. The central approach—generating questions, deriving supervision from majority-vote answers, and optimizing question generation through answer uncertainty—is already closely represented by R-Zero. In particular, Eq. (8) uses the same uncertainty-reward formulation. The main differences concern shared-policy training, diversity rewards, filtering, and adaptive reward weighting. These modifications may be useful, but the paper does not sufficiently establish a distinct conceptual advance or explain what new insight they provide beyond improving an existing self-play training pipeline. The reported gains alone do not fully resolve this concern about the significance of the contribution.

  2. The comparative evaluation is limited, and the presentation makes the contribution difficult to assess. Table 1 includes Absolute-Zero and R-Zero, but much of the comparison consists of dataset-size and supervision variants within the same Semi-dynamicRL family. This provides useful context but limited coverage of distinct approaches. Moreover, the component-removal ablations do not sufficiently isolate the proposed changes relative to the closest prior framework. The paper also lacks a method overview figure: its main-text figure presents training performance rather than the interaction between question generation, pseudo-label construction, filtering, reward computation, and model updates. Given the number of interacting components, this omission makes the overall design and its novel elements unnecessarily difficult to identify.

Questions for authors
  1. What is the central conceptual contribution beyond R-Zero? Given the overlap in self-generated questions, majority-vote supervision, and the uncertainty reward, please distinguish the inherited mechanisms from the proposed innovations and explain why their combination constitutes a substantial methodological advance.

  2. Can the comparative evaluation more directly establish the value of the proposed changes? In particular, matched-budget comparisons with R-Zero augmented by the proposed mechanisms, together with shared-versus-separate policy and adaptive-versus-fixed reward-weight comparisons, would help identify the actual source of improvement.

  3. How are filtered questions excluded from optimization? Section 2.2 sets R_q and R_s to zero, while Eqs. (13)–(14) retain diversity rewards and batch-centered advantages. Please clarify whether flagged questions are explicitly masked from the question-generation loss and associated batch statistics, or whether they continue to contribute gradients.

  4. Could the authors provide a method overview figure that explicitly identifies the new components? Showing the complete training cycle, parameter sharing, and the interaction between filtering and reward computation would make the methodological differences substantially easier to assess.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.