- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DynamicRL, a fully unsupervised RL post-training method for LLM via self-play. This method aims to address the challenge of scaling up training data without the expensive human-crafted tasks and human-labeled solutions. At each training step, the tasks (questions) are generated by the model dynamically, while the proxy ground truth is estimated via majority voting. The reward design consist of two parts: one measures the question quality via agreement rate while the other measures the answer quality via the equivalence with proxy ground truth. To avoid mode collapse, two diversity terms that encourage diverse questions and answers are added to the question reward term. A hyperparameter is introduced to control the progression of question difficulty and prevent reward hacking. The reported experiments show that DynamicRL achieves similar performance compared to training on large datasets with golden answers.
Critical strengths
- The motivation of this paper is highly relevant to the future development of RL post training.
- The presentation is mostly clear and easy to follow.
Critical weaknesses
- Lack of novelty and limited contributions: Most elements in this paper are either standard or have been considered in the literature. Estimating the gold answer via majority voting have been widely adopted (e.g. [1][2][3]). The question reward term is the same as the "uncertainty reward" in R-Zero [3]. The main contribution appears to be the remedy for mode collapse which does not seem justified either (see weakness #2). Beyond the question rewards, the setting is almost identical to [2] (which the authors do not cite). The limited initial data considered in [2] can be viewed as a carefully designed initial prompt distribution in this work (see minor weakness #1). Also see Question #2 regarding reward hacking.
- Unjustified diversity reward term : The proposed question similarity does not seem appropriate as it does not capture semantic similarity at all. For any given question, simply prepending a few tokens or rephrasing it would lead to very low similarity score despite having almost identical meaning. The authors claim that embedding-based similarity metric achieves similar performance without providing any evidence. This reward term is supposed to be the main mechanism to alleviate mode collapse, but it is clearly theoretically unsound.
- Issues with baselines and experiments: A comparison with the closely related work [2] is missing. This severly undermines the evaluation of whether the question rewards are more effective compared to the filters considered in [2]. More details on self-play baselines should also be given. In particular, this work uses a different objective than [2][3][4] with an unnormalized advantage and no policy ratio clipping. The choice of optimization function should be made explicit and ablated.
Minor Weaknesses:
- Trivial initial distribution : There are good reasons to consider non-deterministic . One obvious example is to generate data on tasks in different domain and/or tasks of varying difficulties. Designing the distribution would be an important aspect to consider in this line of work (e.g. [2]). Using necessarily places major limitation on the resulting data distribution and the evidence for the applicability of this method.
- Inconsistent definition of objective functions: Section 2.1 introduces the objectives (1)(2), whereas the actual objective (13) being optimized contains extra terms which are not mentioned in Section 2.1. This can lead to confusion, especially given the technically incorrect sentence "DynamicRL jointly optimizes and " at Line 124.
- Unaddressed dependency of on initial prompt : The definition of in Eq (13) depends on the initial prompt , which the authors do not make explicit. In the paper, experiments are done with being supported on one single prompt so there is no actual difference. However, this definition is not sound when is non-deterministic, unless (for example) an expectation over is added.
- Repeated notation: The notation was used in Eq (2) as the probability distribution over initial prompts. In Eq (7), is defined as the mean solution reward. This leads to confusion even though the two 's take symbolically different arguments and should be changed.
- Typo: at Line 330 should be
- Eq (14) can benefit from changing the parameter in the expectation subscript to to be more technically sound (other than simply saying that the distributions are held fixed).
Questions for authors
- The results reported in Fig 1 (in particular the semi-dynamic ones without golden answer) have different number of training steps, with fewer iterations for smaller datasets. What is the stopping criterion, and how do the authors ensure the comparison is fair?
- The authors claim that semi-dynamic RL without golden answer collapses due to reward hacking. This issue appears to be alleviated by the question difficulty control in the dynamic setting. However, [2] achieves the same goal by only including questions of moderate agreement rate (without using question rewards). Do the authors have experimental results along this line of consideration?
- Non-standard choice of objective: In relevant recent work [2][3][4], the advantage is computed with standard deviation normalization, and PPO-style clipping is used for the probability ratio between the new and old policies. Why do the authors adopt unnormalized advantage and exclude the probability ratio clipping?
- Generalization to other benchmarks: Beyond mode collapse, training solely on synthetic data might degrade model performance on out-of-distribution tasks. It would be helpful even to include some other reasoning-intensive tasks like coding.
- As it is rarely the case that zero training data is available, what is the advantage of this work over the few-shot consideration in [2]? Small question datasets without ground truth are relatively inexpensive to obtain while giving more guided directions in the initial phase of training.